A 5-task Harbor-format eval suite designed to expose a single structural failure mode in gemini/gemini-3.5-flash: it picks the most-uniform reading of an ambiguous business rule, missing the boundary, exception, or scope condition a senior analyst would catch. I'm calling this exception-blindness.
Aggregate pass@3 = 20.0 % against terminus-2 + gemini/gemini-3.5-flash. Each task is a real industry workflow (multi-source tabular reconciliation against strict business-rule output schemas) with a deterministic single-artifact verifier — no LLM-as-judge, no fractional rewards.
| Tasks | 5 (in samples/) |
| Model under test | terminus-2 + gemini/gemini-3.5-flash |
| Aggregate pass@3 | 1 / 5 = 20.0 % |
| Aggregate pass@1 | 1 / 15 = 6.7 % |
| Sanity gates | 5 / 5 oracle = 1.0; 5 / 5 nop = 0.0 |
I built 5 Harbor tasks targeting a single slice of data-science work: multi-source tabular reconciliation against strict business-rule output schemas. Each of the five tasks is an industry-standard routine.
After doing some initial research on the field, I came up with a list of six different cuts of DS work: multimodal workbook reconciliation (CSV + PDF + chart, etc), inferential statistics (A/B test design, Bonferroni correctness), analytics-engineering / ETL pipeline construction, annual-report table extraction, ML model training, and warehouse-join reconciliation (which I chose). Three main factors made me chose this slice over others:
It represents real and high-volume work. Whatever the job title — data analyst, BI engineer, finance ops, RevOps, etc — day-to-day operations is dominated by joining messy enterprise data and applying business rules against strict output schemas. This is the kind of stuff that takes human professionals hours and days.
It exposes a structural failure mode. Across all 5 tasks, Gemini 3.5 Flash ("Flash") converges on the same pattern: it applies the most-uniform interpretation of an ambiguous business rule, missing the boundary, exception, or scope condition that a senior analyst would catch. I'm calling this failure mode exception-blindness (more on this later).
The verifier can be deterministic and binary. Each task produces a single correct artifact (an .xlsx workbook with named cells and tolerance-bounded numerics) that is fully derivable from the inputs. This means not having to use LLM-as-judge, and no fractional-reward pass@k arithmetic (since the results are completely binary).
I built each task based on recognizable industry workflows that could easily be instructions provided by professionals, rather than writing data and materials from scratch. The five industries are:
- SaaS finance (revenue recognition with paused/held subscriptions)
- International parcel logistics (landed-cost audits with FX edge cases)
- Freight operations (carrier settlement with banker's-rounding)
- Equity comp accounting (RSU vesting true-up close)
- Hospital pharmacy compliance (340B drug rebate reconciliation).
The matrix above shows that the scope is diversified along both dimensions: 5 industries × 5 firing mechanisms, with each task occupying a unique row × column cell — no mechanism is repeated, meaning each task represents a different mechanism and industry which shows the same multi-source tabular reconciliation failure mode. The specific Kaggle datasets, production blog posts, papers, etc I drew on for each task's design are outlined in §3.
In scope:
- Multi-source tabular reconciliation across CSV / XLSX / JSON inputs
- Business-rule application with explicit boundary, exception, and scope conditions
- Strict output-schema conformance (named sheets, named cells, tolerance-bounded numerics)
- Real industry workflows with realistic data shapes (i.e. no toy datasets)
Out of scope:
- Multimodal workbook reconciliation. I initially considered using a visual stack (chart perception, PDF parse, etc), as it is an area with well-documented headroom. After testing multiple tasks, mechanisms, and failure modes, I decided that the exception-blindness failure mode with tabular reconciliation was the cleanest and most interesting find.
- PDF table extraction. Same reason — I initially tested this, but image-as-truth perception is its own failure class and not the focus of this project.
- Inferential statistics / A/B-test correctness. Generally a weaker failure mode because of the thin candidate pool in piloting.
- Analytics engineering / ETL pipeline construction. This is adjacent to the failure mode I chose, and tempting, but the dominant failure mode there is design correctness, not exception-blindness, and the work product (a pipeline) doesn't have a clean single artifact verifier.
- ML model training. Measures research/engineering judgment, not analyst judgment — not what I'm trying to expose.
- T1 — SaaS revenue recognition (monthly close). A revenue-recognition close workbook for a SaaS company with paused and held subscriptions. The analyst recognizes revenue across active, paused, and held subscriptions. Pause rules apply only to paused subs, not held ones. The failure is uniformly applying the pause rule across both statuses.
- T2 — Parcel landed cost / margin audit. A landed-cost reconciliation for an international parcel-shipping business, producing a four-sheet workbook with per-parcel duty, surcharge, and de-minimis-exemption (value below which no duty is owed) columns. The failure is applying FX-conversions (an FX-rates file presented as a distractor), but thresholds are already in USD.
- T3 — Logistics carrier settlement audit. A monthly reconciliation of ~500 carrier invoice lines against shipment manifests, with a per-shipment fuel surcharge, per-code accessorial rounding, and a $0.02 dispute tolerance. The failure is summing the invoice total then rounding, when fuel surcharge should be rounded per-shipment first, then summed.
- T4 — Equity vesting true-up close. A quarterly equity-close workbook covering RSU (Restricted Stock Units) vesting math, termination handling, and forfeiture accounting across three formats: monthly %, basis points per year, or fixed shares per period. The failures are a mis-reading basis points as percent (which zeroes the cohort) and picking the wrong "as of termination date" cumulative-vested branch.
- T5 — Pharmacy 340B rebate workbook. A monthly 340B (federal drug-discount program) rebate reconciliation for a covered entity: claim eligibility, per-drug rebate detail, duplicate-discount exceptions, and a summary metric. The failure is writing the
sum_rebate_usdin prose rather than snake_case.
I ran multiple build-pilot-cull rounds against Flash before committing to the final 5 tasks. The first round started with 18 task candidates across multiple slices of DS and multiple potential failure modes. That round exposed a systemic over-specification anti-pattern (instructions were too specific — they looked more like spec sheets — which defused every trap). The challenge was writing instructions that were comprehensive yet left room for the model's inherent failure (exposing headroom). In the second round I rebuilt 13 candidates with stricter task design and instructions, and used those results to lock in the slice + failure-mode focus. In the third round I added 6 more candidates targeting mechanism diversity within the chosen slice. The final cull selected the best 5 from 7 firing candidates.
I inspected tasks scoring pass@3 = 0/3 for broken task design (verifier or data being wrong) vs. genuine failure mode. Tasks that scored pass@3 = 3/3 means Flash solved it easily and no failure mode was present.
Aggregate pass@3 = 20.0 % (1 of 5 tasks ever pass). Aggregate pass@1 = 6.7 % (1 of 15 trials). Of the 67 verifier assertions across the 5 tasks, Flash gets 58 right at its best (~87 %). Failures are isolated; most are just one asserted value off the right answer, not "wildly wrong."
Reading across the rows: T2 is the only passer — and only on trial 3, after two byte-identical failures at 14/28. T3 is close but it fails: it scores 16/17 on every trial, the same one wrong assertion each time. T4 is the hardest (best 2/6), and T5 is at 5/6 across all three. None of the failures cluster at 0% or near it — Flash is almost always close, and the verifier grades binarily.
The "difficulty curve" on this set means three main layers:
- Deterministic floor. T1, T3, T5 produce byte-identical sub-test counts across all 3 trials. That's the empirical signature of exception-blindness: Flash makes the same mistake the same way every time, which means a failures here are not sampling noise and or a bad verifier — they're a stable misreading of the instruction.
- Single-attempt anomaly. T2's three trials are (14 / 28, 14 / 28, 28 / 28). Two byte-identical failures, then a clean pass. The only stochasticity in the entire eval is T2 attempt 3 — where Flash explicitly recognized that
landed_cost_config.csvexposes USD-denominated thresholds and chose to skip the FX-conversion step (the trap the other trials fell into). This anomaly actually strengthens the isolation of the failure mode in this task. - Minor variance. T4 reads (1 / 6, 2 / 6, 2 / 6). Not a difficulty signal.
With pass@3 binary on 5 tasks the headline number is integer-coarse, but the sub-test view shows the difficulty band with more detail. Every task gets meaningful partial credit, and the failures are concentrated on a small number of high-leverage assertions per task.
Failure modes present:
- FM-1 — strict multi-constraint schemas. Output has to match an exact structure (named sheets, fixed column headers, specific cell formats, etc) or it fails the verifier.
- FM-2 — multi-step cascade. One wrong upstream value flows into many downstream cells.
- FM-3 — compounding arithmetic. Many small calculations stacked together (e.g. rounding or unit errors) compound into a wrong final answer.
FM-1 is the anchor on all 5 tasks — this is why T3 sits at 16/17 and T5 at 5/6: one schema-asserted cell is wrong. FM-2 appears as a stack failure mode on 4 of 5 (T1, T2, T3, T5) — the multi-step coupling which makes one upstream misreading propagate into downstream sheet values. FM-3 is a stack failure on 2 of 5 (T3 rounding scope, T4 vesting math). The consistent failures don't dominate from any one isolated failure mode; they emerge from FM-1 combined with one of FM-2 or FM-3. FM-1 acts as the check (schema constraint) which is a downstream failure of FM-2/FM-3.
The unifying frame across the all 5 tasks is exception-blindness. Flash picks the most-uniform interpretation of an ambiguous business rule, missing the boundary, exception, or scope condition a senior analyst would catch. For each task, this looks like: pause-applied for held subs (T1), FX conversion applied to USD-denominated thresholds (T2 failing trials), fuel-surcharge rounded at invoice scope instead of per-shipment (T3), bps unit-conflation that zeroes the cohort (T4), prose labels instead of the sum_rebate_usd snake_case token (T5). Verbatim trajectory quotes for each are live in §5.
To find tasks that represent the actual work data scientists do in industry, I looked through Kaggle notebooks, recent arxiv benchmarks, production engineering blogs, model docs, and the open-source DS-eval ecosystem. Before diving into task design, I spent a few hours building a synthesized brief covering the DS slice space, failure-mode taxonomy, anti-patterns, and a source ledger; I used that document both as a learning resource and as a context document throughout the project.
Below is the full source list:
Public Kaggle datasets and government data: SaaS Subscription & Churn Analytics, General Ledger, Supply Chain Shipment Pricing Data, DataCo SMART SUPPLY CHAIN, Freight Analysis Framework, Cargo 2000, SEC EDGAR, Synthetic Healthcare Claims, HRSA Office of Pharmacy Affairs.
Production engineering write-ups: modeling subscription revenue, month-end reconciliation best practices, No More $800 Duty-Free Imports, de minimis value explainer, Sennder freight operations, Fuel Surcharges in Trucking, accounting for stock-based comp, What is ASC 718?, Preparing for the 340B Rebate Model, Financial Reconciliation in the 340B Rebate Era, 340B Rebate Model: Building Audit-Ready Operations, How to evaluate AI agents — an introduction to Harbor.
Frontier benchmarks, arxiv papers, model docs: SpreadsheetBench, FLARE, JSONSchemaBench, FinSheet-Bench, How well do LLMs reason over tabular data, really?, Benchmarking Multi-Agent LLM Architectures for Financial Document Processing, RUST-BENCH, Finance Agent Benchmark, WorkstreamBench, DeepAnalyze, DSAEval, Terminal-Bench 3 contribution call, Gemini 3.5 Flash Model Card.
- Tessl, "How to evaluate AI agents — an introduction to Harbor" — Intro to Harbor's task structure. Helped inform decision to keep every task as single-artifact (one
.xlsxoutput) so the verifier is purely structural and numeric. - Terminal-Bench 3 contribution call — Codifies what Terminal-Bench 3 considers a "good task." I used the Terminal-Bench 3 task-implementation rubric as my pre-ship checklist for every task. (Terminal-Bench 3 GitHub)
- SpreadsheetBench (NeurIPS 2024) — real Excel-forum tasks scored against per-cell assertions. Takeaway: agents do meaningful partial work even when they fail the binary check, so multi-assertion deterministic verifiers (not LLM-as-judge) are the best way to get clean signal (this is what I ended up implementing).
- JSONSchemaBench (Jan 2025) — constrained decoding helps syntactic schema compliance but can hurt semantic quality. Takeaway: don't accept format-only schema checks; pin exact values (I did this as well).
- DSAEval — real-world industry DS problems with multimodal-input and multi-query-interaction framing. Covers the multimodal + ETL space which I did not end up pursuing, but comparing my slice to DSAEval did help me in my decision to going narrow on tabular reconciliation rather than broad on DS as a whole.
- Gemini 3.5 Flash model card — published capabilities are benchmark-driven, which meant my eval has a chance of finding headroom that is not in the public suite if I target a narrow slice (multi-rule business logic) rather than industry benchmarks.
Across the production blog posts listed above (dbt Labs on subscription revenue; Numeric on stock-based comp and month-end reconciliation; ShipBob and Easyship on de-minimis fulfillment; the Fivetran Sennder freight case study and the FreightWaves fuel-surcharge primer; Carta on ASC 718; and the 340B Report / Plenful / Proximity / SRM Tech write-ups on the rebate model), the general and consistent takeaway is that real reconciliation work always contains at least one rule that the analyst has to decide on, not just compute. This is exactly where exception-blindness surfaces for Flash. These write-ups were how I confirmed that the boundary/scope ambiguities that I built into tasks (Held subscriptions in T1, per-shipment vs invoice level rounding in T3, bps vs percent unit confusion in T4, snake_case vs prose metric label in T5) demonstrate both real DS work and a real failure mode.
The datasets listed above span SaaS subscription analytics, financial general ledgers, supply-chain shipment pricing, the USDOT Freight Analysis Framework, multi-leg freight tracking, SEC EDGAR filings, synthetic healthcare claims, and HRSA Office of Pharmacy Affairs documentation. The takeaway here is that messy real-world column conventions (mixed case, abbreviations, embedded units in column names, etc) are the setting in which real DS work happens. Toy datasets sand this off and remove the surface where strict-schema failures actually occur. I deliberately preserved this noise (e.g. wac_price_usd vs WAC vs wacPrice) in my generated data.
Going from 5 tasks to 1,000 is mainly three kinds of problems: a sourcing problem (where do tasks come from?), an authoring problem (how do you write 1,000 of them with consistent quality?), and a measurement problem (how do you know which of the 1,000 actually fire?). I'm using empirical lessons from my workflows during this run and broader thinking to shape what I'd build for each. Also note that scaling differs depending the slice and failure mode — this plan is for tabular reconciliation and exception-blindness specifically.
The 5 firing mechanisms I committed to (boundary trap, distractor cascade, scope-placement pivot, unit conflation, schema-narrow) are generative templates, not one-off designs. Each one is a (data-shape) × (ambiguous rule) × (strict-schema output) tuple, where each tuple has many instances in the real world. The public data I would lean on:
- Finance/accounting. SEC EDGAR (10-K and 10-Q workbooks, which have the actual as-filed XBRL), official FASB ASC examples, and public CFO close packages. Each gives a reconciliation pattern that can be templated against synthetic transactions.
- Logistics/supply chain. Federal customs filing data, FMCSA freight data, DOT shipment registries, USAID procurement (I used this). These provide multi-source join shapes naturally.
- Healthcare operations. Hospital cost reports (HCRIS), CMS claims data (DUA agreement could be a blocker), 340B compliance writeups (I used this), pharmacy benefit data. These are generally strict schema and ambiguous rules.
- Equity/corp-dev. SEC Form 4 filings, public ASC 718 examples, public Carta RSU schedule samples.
- Workflow primitives. Each task has the same outline: 3–6 input CSVs, one .xlsx output with 3–6 sheets and a strict-schema cell layout. A task generator that takes (real public data + a 3-rule template + a target firing-mechanism) and produces a Harbor task format directory is what should scale.
Some lessons and things I noticed when running QA in my own workflows are part of what I'd build the QA loop around:
- Instruction over-specification. Round 1 produced 18 candidates that all hit pass@3 = 100% in pilot. I would set hard rules for scale: target 50–70 line instructions, business-stakeholder voice, schema literals appear only once in the reference files. An LLM linter on instruction.md (read by Sonnet/Haiku or an open-weight model, trained against a few-shot of good/bad examples) is cheap to run on 1,000 tasks and catches over-specification before any expensive run.
- Skill-file value leakage. This was caught in 4 candidates during my runs (ground-truth numerics smuggled into a skill file the agent reads). Set hard rule for scale: mandatory grep audit — for each task, grep all skill files and reference files for any numeric token that appears in the verifier oracle.
- Schema-by-enumeration. Listing every required column header verbatim in the instruction defused any FM-1 — Flash just reproduced it. For scaling: schema literals stay in
reference/*.csvfiles that the agent has to inspect. Ratio of asserted string-literals to instruction line count should stay under 30%.
The 3-gate that every task went through in this run scales cleanly:
- Mechanical —
harbor checkagainst the Terminal-Bench 3 task-implementation rubric (I replicated this manually for the run that produced these logs; see notes at the bottom). - Oracle = 1.0 —
harbor run -a oraclemust produce reward = 1.0 - Nop = 0.0 —
harbor run -a nopmust produce reward = 0.0
Plus a 4th automated gate at scale: pilot-fire at n = 3. Any task that scores pass@3 = 0/3 (either broken, verifier or data wrong, or genuine task difficulty) OR pass@3 < 3/3 (headroom exists) routes back through review — either LLM or human. This was the criterion in my own loop and turns "1,000 tasks built" into "1,000 tasks that fire." Anything that scores pass@3 = 3/3 can either be discarded for not finding a failure mode or looped through review to ensure authoring is correct (but this should not be an issue if authoring works properly, so therefore discarding should be the response).
Two things don't auto-scale:
- Trajectory attribution. Finding "what mechanism fired" and whether a task failure represents a genuine failure mode requires reading 3,000+ trajectories at 1,000 tasks. The best answer is an LLM-judge over a structured taxonomy: read the trajectory, label which mechanism fired (or which task-design defect led to failure), and route mis-labels back through the review cycle. This LLM/human labeling and reporting review is what I did for the 20+ candidate tasks in my runs. This is the most expensive piece of the scale plan and the highest-value to nail.
- Statistical power. Three trials per task is enough to compute pass@3 but coarse. For training data — where you want to know the actual pass probability of each task, not just a binary at n = 3 — you want more trials and a confidence interval. With 5 trials per task you can give a Wilson-score CI on pass@1 instead of a binary pass@3 measurement. I would recommend n = 5 in the QA loop and n = 10 only for tasks the LLM-judge flags as marginal.
- Augmentation by mechanism. Take T1's boundary-trap template (as an example), parameterise the boundary threshold + the exception count + the pause-window inclusivity rule, and generate 30 variants. Creating different surfaces for the same failure mode like this is cheap and high-leverage.
- Cross-industry rotation. Take T3's per-shipment-rounding mechanism and swap it onto T5's pharmacy-claim domain. This kind of rotation tests whether the mechanism generalises or whether it's industry-coupled.
- A "deceptively simple" lane. The most informative tasks in this run scored 70–94% sub-test correct — Flash is almost right. I'd put focus on this region, where Flash gets 85%+ but trips on one boundary or scope condition. This is a high-leverage deterministic training signal that can save LLM-judge cost.
This will show that the failures in each of the five tasks come from genuine task difficulty, not from task-design bugs (such as ambiguous instructions, broken environment, or over-strict verifier). The strongest evidence is within the agent trajectories themselves, particularly the long deliberation passages where Flash visibly considers multiple interpretations and picks the wrong one. Across all 5 failing tasks, every failure is the same structural shape (exception-blindness) in a different domain.
| Task | Trials | Verdict | What fired |
|---|---|---|---|
| T1 SaaS rev-rec | 0 / 3 | ON-FM (boundary trap) | 3/3 byte-identical: 8 extra pause_applied = True subs (Held subs treated as paused) |
| T2 Parcel landed cost | 1 / 3 | ON-FM (FX distractor) | 2 trials apply FX conversion to USD-denominated thresholds; trial 3 recognises and skips the trap |
| T3 Carrier settlement | 0 / 3 | ON-FM (scope-placement pivot) | 3/3 byte-identical: $0.03 drift on C002 past the $0.02 dispute tolerance — fuel surcharge rounded at invoice scope instead of per-shipment |
| T4 Equity vesting | 0 / 3 | ON-FM (FM-2 cascade) | Trial 1: bps unit-conflation zeroes the 114-grant cohort. Trials 2–3: Flash deliberates "as of the termination date" interpretation and chooses the wrong cumulative-vested branch |
| T5 Pharmacy 340B | 0 / 3 | ON-FM (schema-narrow) | 3/3 byte-identical: Flash deliberates sum_rebate_usd vs prose-style metric names and picks prose; the verifier asserts the snake_case token verbatim |
All five firings are on the intended mechanism. Deep-dives below.
Task: a monthly-close revenue recognition workbook with a paused-subscription rule. The pause_applied column on the output sheet is True for subscriptions paused under the credit-and-resume policy, and False for the 8 subscriptions in status = Held; Held subscriptions should sit outside that policy.
Failure: across all 3 trials, Flash marks all 18 paused subscriptions pause_applied = True, missing the 8 Held carve-outs. Verbatim from trial 1, step 7 (substantively identical in trials 2 and 3):
"
RR-02: 'status=paused', 'halt: recognized=0 for paused days', 'hold: deferred unchanged during pause', 'Recognition halts on pause_date; deferred balance held; no proration across pause window'. Wait! Does 'halts on the pause date' mean the pause date itself has 0 recognition?"
Flash interprets the pause-policy rule uniformly across every paused-status sub (including the 8 Held subs) even though RR-02 is silent on Held (which is a separate status). The senior-analyst catch is that Held implies a hold on the deferred balance without changing the recognition state, so pause_applied = False. Flash hits the same boundary the same way every time (3/3).
Task: reconcile ~500 carrier invoice lines against shipment manifests. Fuel surcharge is base_charge × fuel_factor_pct, and the verifier asserts dispute classification on carriers that drift more than $0.02 from claimed_amount. Required scope: round per-shipment first, then sum the rounded values.
Failure: 3/3 byte-identical 16/17 — Flash computes the fuel surcharge per shipment but sums first and rounds last (rounding at the invoice total instead of per-shipment). Verbatim from trial 1, step 16:
"Step 2: Process manifests to compute base and fuel surcharge for each invoice. For each row in
shipment_manifests.csv:fuel_surcharge_shipment = base_charge × (fuel_factor_pct / 100).shipment_subtotal = base_charge + fuel_surcharge_shipment. Then group byinvoice_idand sum:fuel_surcharge = sum of fuel_surcharge_shipment."
The $0.03 drift on carrier C002 because of rounding methodology error trips the $0.02 dispute threshold by exactly $0.01, which is what keeps every trial at 16/17. The senior-analyst catch is that rounding at the scope where the rule is applied is the accounting convention, and the verifier enforces this rule. The 16/17 ceiling across all 3 trials is the cleanest evidence that the difficulty is the boundary, not the data or the verifier. Flash gets every other check right.
Task: a Q1 close workbook with a vesting + termination rule. Terminated employees forfeit "any unvested balance as of the termination date." The verifier expects the forfeiture to use cumulative-vested at the termination date (cum_term), not at the prior quarter-end (cum_prior).
Failure: trial 1 commits an FM-2 cascade (bps unit-conflation zeroes the cohort, secondary). Trials 2 and 3 produce an extremely clean deliberation trajectory — Flash walks through both interpretations explicitly, weighs them, and picks the wrong one. Verbatim from trial 2, step 17:
"'The forfeiture sheet lists each terminated employee's grants with the unvested share count being forfeited.' Wait, how do we calculate the unvested balance being forfeited as of the termination date? And what is the cumulative vested as of the termination date?"
At step 33, Flash names both branches by hand and picks one:
"Under
forfeited_prior, the forfeited shares areshares_granted - cum_prior. Underforfeited_term, the forfeited shares areshares_granted - cum_term. Wait, let's re-read the exact phrasing of the task description: 'terminations forfeit any unvested balance as of the termination date and zero out the terminated employee's vesting-this-quarter.'"
Flash explicitly identifies both interpretations of "unvested balance as of the termination date," recognises that one treats the mid-quarter vesting as forfeitable and the other doesn't, and picks the wrong branch. This is a textbook case of exception-blindness. Faced with real ambiguity, Flash picks the most-uniform reading (treat termination like any other end-of-quarter snapshot) rather than the boundary-aware reading (the termination-date-specific cumulative). A senior analyst would consult ASC 718, whereas Flash defaults to the uniform interpretation.
Task: a 340B rebate workbook with a Wholesaler Reconciliation sheet that asks for "the total rebate dollars across all NDCs" (National Drug Codes). The verifier checks the metric column for the exact token sum_rebate_usd. The instruction prose never uses sum_rebate_usd — it describes the metric in plain English.
Failure: 3/3 byte-identical 5/6. Verbatim from trial 1, step 22, Flash explicitly weighs prose vs snake_case:
"Let's verify the exact metric names to put in Sheet 4. To cover all bases, can we put both? No, Sheet 4 has a list of rows with keys and values. ... Let's use the exact names from the text or standardized versions. Let's think if we can write them like this: 1) 'Total Claims Processed' 2) 'Eligible Claim Count' 3) 'Count of Duplicate-Discount Exceptions' 4) 'Distinct NDCs Earning a Rebate' 5) 'Total Rebate Dollars'."
Flash picks the title-case prose interpretation in every trial. The senior-analyst catch is that "total rebate dollars" is the description; the token the downstream system uses is the snake_case identifier in the data dictionary. Industry reconciliation work makes this distinction constantly (column tokens are stable and prose labels can drift) and the verifier reflects that.
T2 is the only task where pass@3 > 0 and yet is very revealing because it shows what not falling into the trap looks like. The task has two FX traps: landed_cost_config.csv exposes USD-denominated thresholds, and fx_rates.csv is present in /root/data but never needed. Trials 1 and 2 (both at 14/28 byte-identical) apply the FX conversion. In Trial 3, Flash reads the config file carefully and reasons that the thresholds are already USD:
"
landed_cost_config.csvhas de-minimis thresholds for countries (e.g. US, GB, DE, CA, AU, JP) as well as dimfactor for carriers ... Wait, does the task description mention usingdimfactoror isbillable_weight_kgjust the max ofactual_weight_kganddim_weight_kg? ... We don't even need the dimfactor from the configuration."
The trap fires deterministically on the 2 misreads; the passing trial is the case where Flash re-reads carefully enough to identify which inputs are distractors.
This shows that the headroom is real, the verifier is fair (T2 passes; T3 hits 16/17), and the failures are interpretive, not arithmetic. Across all five tasks, Flash chooses the most-uniform reading of an ambiguous business rule — this is the unifying exception-blindness failure mode. It shows up across SaaS rev-rec, logistics rounding scope, equity termination handling, pharmacy schema labels, and parcel FX conversion. None of these are a task design failure — they are all the kind of judgment call a senior data analyst would properly (and is paid to) make.
All 5 tasks ship with deterministic, binary verifiers (tests/test.sh writes 1 or 0 to /logs/verifier/reward.txt). Each task has a single correct artifact — an .xlsx workbook with named sheets, named cells, and fully-derivable numeric values — so the verifier's job is structural conformance and tolerance-bounded numeric equality, not judgment. LLM-as-judge would add stochastic noise to a problem with unambiguous ground truth, contaminating the pass@k math. Fractional rewards were not used for the same reason.
harbor check requires an Anthropic API key with quota. I substituted by running the same prompt template and the same Terminal-Bench 3 task-implementation rubric (from the TB3 GitHub) against the same Claude Sonnet 4.6 model via a Claude Max subscription instead. Methodology and outputs are exactly (or near-exactly) equivalent to what harbor check -r <rubric> would have produced.
As an additional note, harbor check (or my equivalent) actually fails on each of these 5 tasks with a shared-verifier diagnostic — the verifier is mounted in a way the linter flags as "agent could read the verifier". I left the shared-verifier mount in place because Harbor's own reference task family uses the same mount pattern, so I read that as the convention. The agent can't actually read into the verifier directory at runtime (I validated this empirically based on Harbor docs), so the diagnostic is a false positive for this task family. Everything else passes.
n = 3 trials per task is enough to compute pass@1 and pass@3 but coarse. At scale (for training data) and for an open-source contribution, I would instead estimate a Wilson-score confidence interval on the underlying pass probability per task — five trials per task is roughly where the CI tightens enough to discriminate marginal tasks (this is also what I recommended in §4). This fits what the Terminal-Bench 3.0 Call for Contributions asks for: "at most 30% solve rate" for contributing tasks.
The most consistent confrontation throughout the project was the relationship between instruction prose and failure mode. Round 1's pilot finding (where all 18 tasks all hit pass@3 = 100%) wasn't a model-strength finding, but rather an instruction-format finding (the over-specified spec sheets defused every trap). Re-writing the instructions (to a more standard format) changed performance while keeping the model the same, and allowed me to properly isolate the consistent failure mode. I thought this was interesting and ties into recent discussion around the fact that the open-source bugs Mythos claimed to find were also discoverable on older models, and the solution to the Erdős problem that the internal OpenAI model claimed to write has also been replicated on existing models. I read this article while I was setting this up. An interesting idea to me is how prompt sophistication compares in leverage to model sophistication for ambiguous reasoning tasks. I believe I addressed this cleanly with how I wrote the instruction.md files in this project, but it is an important question in the benchmarking field going forward.
tabular-reconciliation-eval/
├── README.md this file
├── samples/ 5 Harbor task directories
│ ├── 01-saas-revenue-recognition-monthly-close/
│ ├── 02-parcel-landed-cost-margin-audit/
│ ├── 03-logistics-carrier-settlement-audit/
│ ├── 05-equity-vesting-true-up-close/
│ └── 06-pharmacy-340b-rebate-workbook/
├── logs/ canonical run set: 5 tasks × {oracle, nop, flash × 3}
└── report/figures/ figure scripts + rendered PNGs
The numbering (01/02/03/05/06) preserves the lineage of the build phase — the gap at 04 is a task that didn't make the final cut.
# Install Harbor (uv recommended)
uv tool install harbor
# Set the model API key (Gemini for the model under test)
export GEMINI_API_KEY=...
# Sanity gates — these must pass before trusting any real run
harbor run -p samples/<task-slug> -a oracle # expected reward = 1.0
harbor run -p samples/<task-slug> -a nop # expected reward = 0.0
# Real run: 3 trials against terminus-2 + Gemini 3.5 Flash
harbor run -p samples/<task-slug> -a terminus-2 -m gemini/gemini-3.5-flash -k 3 -o logs --job-name flash-<NN>August Andersen