Can a small open model match a frontier model?

1. Summary

2. Hypothesis and the four gates

A fine-tuned open model small enough to run 5–10× cheaper per task than GPT-5.6 matches its accuracy, generalises beyond its training threads, and knows when to hand a trade to a person. The hypothesis is disproved if no model size passes all four gates below.

Where the small model could win outright: spotting unusable input. A fine-tuned model learns what a complete, bookable thread looks like; general models tend to fill the gap and book anyway. If its rejection score beats GPT-5.6's, fine-tuning has added domain expertise, not only lower cost. Reported, not gated.

The thresholds are proposals; challenge them. The bar is GPT-5.6, the latest frontier model.

3. The task

The agent turns one email thread, with its attachments, into structured bookings.

3.1 The agent and where it sits

Kestrel Energy Trading GmbH (invented) trades German power and gas. Deals are agreed by phone or chat; what reaches trade support is email: broker recaps with PDF or Excel attachments, counterparty confirmations, corrections, cancellations and forwards, often in long reply chains. About 150 such emails arrive a day, and trade support re-keys every trade by hand. A missed correction or a wrong legal entity surfaces only when the confirmation or invoice fails to match.

The trade-capture agent reads each email thread when a new email arrives and books what the thread actually confirms. Each booking carries a confidence score: above a threshold it is booked straight away; below it, a person reviews it first. When the thread itself is unusable (an attachment that can't be read, a company we don't know) the agent books nothing and says what to fix.

Every thread needs four reasoning steps:

3.2 A worked thread

A short thread to teach the idea. The right column says how to read each part: the green row makes the trade, the red row cancels the add-on. Terms are under Trading terms.

3.3 A realistic thread

Input at full scale: a test-set thread with 5 emails, 3 attachments and 5 bookings, read against a 214-entry counterparty list. The list under the answer shows where a small model slips.

E1  16 Feb 17:42  From: Tom Reyes <recaps@halden-brokers.example>
    To: trade-support@kestrel.example
    Subject: EOD recap 16-Feb - Kestrel
    Hi all, EOD recap attached. Row 2 credit approved by VEH at 09:40. Any issues shout.
    Tom Reyes | Halden Power Brokers | +44 20 ... | This email and any attachments are
    confidential ...
    Attachment: EOD_recap_HPB_16Feb.xlsx
    Row | Ref    | Buyer       | Seller       | Product         | MW | Price | Status
    1   | H-4471 | Kestrel     | Veltrona ET  | GER Base Mar-27 | 5  | 78.85 | Done
    2   | H-4472 | Kestrel     | Veltrona EH  | GER Base Mar-27 | 5  | 78.95 | Done (sub credit)
    3   | H-4473 | Kestrel     | Brenntal     | GER Base Q2-27  | 5  | 68.50 | Done
    4   | H-4474 | Kestrel     | Nordhafen    | GER Base Apr-27 | 10 | 69.40 | CXL
    5   | -      | Kestrel bid | -            | GER Base Q2-27  | 10 | 68.40 | Working
    6   | H-4476 | Kessel      | Kestrel      | GER Base Q3-27  | 10 | 71.50 | Done
    7   | H-4477 | Kestrel UK  | Arcadia Mkts | UK Base Cal-28  | 2  | 74.20 | Done

E2  16 Feb 17:55  From: confirmations@veltrona-eh.example
    Subject: Trade Confirmation VEH-2027-00871
    Please find our confirmation attached. Kindly countersign within two business days.
    Attachment: VEH-2027-00871.pdf (2 pages)
      "... Veltrona Energiehandel GmbH (the Seller) and
       Kestrel Energy Trading GmbH (the Buyer) ...
       Delivery Period: 01.03.2027 00:00 – 01.04.2027 00:00 CET
       Contract Capacity: 5 MW | Price: EUR 78,95/MWh
       This Confirmation is governed by the EFET General Agreement ... clearing: none ..."

E3  16 Feb 18:03  From: ops@brenntal.example  Subject: RE: our trade today (H-4473)
    Hello, please note the contracting party on H-4473 is Brenntal Energie Vertrieb GmbH,
    not the AG.
    > From: Tom Reyes ... (quoted recap follows)

E4  16 Feb 18:10  From: Tom Reyes  Subject: RE: EOD recap 16-Feb - Kestrel
    Apologies, row 6 price was wrong. Corrected file attached, all else unchanged.
    Attachment: EOD_recap_HPB_16Feb_v2.xlsx   (identical except row 6: Price 71.05)

E5  16 Feb 18:20  From: J. Kahn <jk@kestrel.example>  To: trade-support  Subject: FW: Mar peak
    pls also book this, agreed yday
    ---------- Forwarded message ----------
    From: desk@kessler-strom.example
    We confirm the reduction of trade T-88213 (GER Peak Mar-27) from 5 MW to 3 MW, price
    unchanged at 88.10.

CP-0417  Veltrona Energie Trading GmbH   [Veltrona ET, VET]     active
CP-0453  Veltrona Energie Trading GmbH   [VET]                  inactive, merged into CP-0417
CP-0418  Veltrona Energiehandel GmbH     [Veltrona EH, VEH]     active
CP-0452  Veltrona Energy Trading UK Ltd  [Veltrona UK, VET UK]  active
CP-0611  Nordhafen Power AG              [Nordhafen, NHP]       active
CP-0630  Brenntal Energie AG             [Brenntal, BEAG]       active
CP-0631  Brenntal Energie Vertrieb GmbH  [Brenntal Vertrieb]    active
CP-0702  Kessler Strom GmbH              [Kessler]              active
CP-0703  Kessel Energy SE                [Kessel]               active
CP-0811  Arcadia Markets Ltd             [Arcadia Mkts]         active
IN-0002  Kestrel Energy Trading UK Ltd   [Kestrel UK, KET UK]   internal
BR-0007  Halden Power Brokers            [Halden]               broker
BR-0012  Arcadia Energy Brokers          [Arcadia]              broker

T-88213  buy   5 MW  DE-PEAK-M-2027-03   88.10  CP-0702 Kessler Strom GmbH
T-88214  buy   5 MW  DE-PEAK-M-2027-03   88.30  CP-0611 Nordhafen Power AG
T-88209  sell  5 MW  DE-PEAK-M-2027-03   88.25  CP-0703 Kessel Energy SE
T-88197  sell 10 MW  DE-BASE-Q-2027-Q2   68.90  CP-0703 Kessel Energy SE

Gold answer, computed by code:

Not booked: row 4 (cancelled), row 5 (working order, never filled), row 7 (Kestrel UK's trade, booked by that entity). Nothing unsure; thread processed.

Where a small model slips, part by part:

• Two versions of one recap: v2 replaces v1, so row 6 is 71.05, not 71.50 (trap 3).
• Side from columns: we are the buyer in rows 1–3 and the seller in row 6.
• Status column: CXL, Working and "Done (sub credit)" mean three different things; the credit approval sits in the E1 body (traps 1, 3).
• Near-duplicate companies: VET is CP-0417, not the inactive CP-0453 with the same name or the UK entity; Brenntal's email moves C to the Vertrieb entity; "Kessel" in row 6 is not "Kessler" in E5 (trap 4).
• Rows that are not ours: row 7 belongs to Kestrel UK, an internal entity (trap 9).
• An amendment among look-alike trades: the forwarded Kessler email changes T-88213, not the Nordhafen or Kessel March peak trades (trap 8).
• Noise everywhere: signatures, disclaimers, quoted history and two pages of confirmation boilerplate that mention clearing (trap 7).
• Four hour counts: March base with the clock change, March peak on weekdays, Q2 and Q3 in full (trap 5).

3.4 Input and output

The agent gets what a trade-support person sees: the thread with its attachments as extracted text, plus our reference data. It returns what that person would book, leave out, flag or send back.

// input
{
  "thread_id": "TH-20270216-0412", "processed_at": "2027-02-16T09:55+01:00", "timezone": "Europe/Berlin",
  "emails": [
    {"email_id": "E1", "sent_at": "2027-02-16T09:20+01:00",
     "from": "recaps@halden-brokers.example", "to": ["trade-support@kestrel.example"], "cc": [],
     "subject": "Recap 16-Feb: Kestrel buys GER Base Mar-27",
     "body": "Hi team, recap attached. Note the 3 MW add-on with Veltrona ET is subject to their credit approval. Tom",
     "attachments": [{"file_name": "recap.pdf", "type": "pdf",
       "text": "Row | Buyer | Seller | Product | MW | Price | Status\n1 | Kestrel | Veltrona ET | GER Base Mar-27 | 5 | 78.90 | Done\n2 | Kestrel | Veltrona ET | GER Base Mar-27 | 3 | 79.00 | Done (sub credit)"}]},
    {"email_id": "E2", "sent_at": "2027-02-16T09:55+01:00",
     "from": "recaps@halden-brokers.example", "to": ["trade-support@kestrel.example"], "cc": [],
     "subject": "RE: Recap 16-Feb: Kestrel buys GER Base Mar-27",
     "body": "Credit declined on the add-on (row 2), please disregard it. Row 1 stands.\n> Hi team, recap attached. ...",
     "attachments": []}
  ],
  "counterparties": [
    {"id": "CP-0417", "legal_name": "Veltrona Energie Trading GmbH", "short_names": ["Veltrona ET", "VET"], "role": "counterparty", "active": true},
    {"id": "CP-0418", "legal_name": "Veltrona Energiehandel GmbH",   "short_names": ["Veltrona EH", "VEH"], "role": "counterparty", "active": true},
    {"id": "BR-0007", "legal_name": "Halden Power Brokers",          "short_names": ["Halden"],            "role": "broker",       "active": true}
  ],
  "prior_trades": []
}

// output
{
  "thread_status": "processed",
  "trades": [{
    "status": "new", "amends_trade_id": null, "side": "buy", "commodity": "power",
    "product_code": "DE-BASE-M-2027-03",
    "delivery_start": "2027-03-01T00:00+01:00", "delivery_end": "2027-04-01T00:00+02:00",
    "delivery_hours": 743, "volume_mw": 5, "volume_mwh": 3715,
    "price": 78.90, "currency": "EUR", "counterparty_id": "CP-0417",
    "broker_firm": "Halden Power Brokers", "desk": "Power DE", "exchange_cleared": false,
    "evidence": ["E1/recap.pdf/row 1", "E2/body"], "confidence": 0.97
  }],
  "not_booked": [{"what": "add-on 3 MW at 79.00", "why": "subject to credit; credit declined in E2",
                  "evidence": ["E1/recap.pdf/row 2", "E2/body"]}],
  "unsure": [], "input_problems": []
}

4. What makes it hard: the fifteen traps

A trap is a kind of thread where a shortcut, learned in place of the rule, gives the wrong answer. These are common failure modes of LLM agents reading messy business text. Traps 1–12 are about getting the answer right; 13–15 about knowing when to hand a thread to a person or send it back. The dataset is built so every shortcut fails somewhere.

Grouped by the reasoning steps in 3.1. ✗ is the shortcut's answer, ✓ the gold answer. A twin is a copy of a thread with one edit (6.3).

Levers that help a small model, each tested as a variant:

• reasoning traces that restate what is still live;
• a calendar tool for counting hours;
• schema-constrained decoding;
• the rulebook kept in the prompt;
• look-alikes in every training counterparty list;
• a confidence score fitted on the calibration split (7.4).

5. The dataset

The dataset is 1,900 synthetic threads, delivered as JSONL. Four controls shape it: splits (what is held out), the variation matrix (what varies), reasoning ingredients (what forces reasoning) and tiers (how many ingredients one thread combines).

5.1 Splits

Blue is training data (300 clean single trades, 450 trap threads, 150 with a messy surface, 100 with deliberately noisy labels), grey is calibration, amber is test. Traps are oversampled so each shortcut fails often enough to measure.

5.2 Variation matrix

Scale per thread: 1–12 emails with 0–4 attachments (about 1,000–20,000 tokens), a counterparty list of 150–300 entries with 2–4 near-duplicates per active name and about 10% inactive, and 0–40 of yesterday's trades. Each training thread draws one level per dimension at the shares below. Quotas then make every trap occur with every delivery period and every way a trade is stated.

5.3 Reasoning ingredients by tier

An ingredient is a feature that forces a reasoning step; a tier is how many a thread combines: tier 1 none (300 plain threads), tier 2 one, tier 3 two or three, tier 4 four or more. Each cell counts training threads where the ingredient decides the answer. Every ingredient appears alone and combined, so a failure traces to the ingredient or the combination.

The realistic thread in 3.3 is a tier-4 case: it combines revisions, look-alike parties, conflicting versions, several trades, an amendment and four hour counts.

5.4 Agent answers: right and wrong

Every sample carries a realistic agent answer, the kind of output an agent run logs, so it can feed preference tuning. The agent answer is fully right in 80% of threads. In the other 20%, the error is one of the kinds below, most tied to a trap in section 4. About a quarter of the wrong answers also carry a routing error (traps 13–15): high confidence on a wrong trade, a booking from an unusable thread, or a usable thread rejected.

6. Building the dataset

How the threads are generated and checked.

6.1 Pipeline

• Paraphrase with a model that is not under test. A frontier baseline that also wrote the threads would be scored on its own style.
• Step 4 is a blind second read. A second model answers the thread without seeing the gold answer. Any disagreement goes to a person or the thread is discarded. Target: under 2% of threads discarded for a wrong gold answer.
• Train and test never share a template, a paraphrase seed, or any five-word sequence other than standard closing phrases.

6.2 Rulebook v0

The eleven rules that code computes the gold answers from. v1 is a deliverable (8).

6.3 Twins and wrong agent answers

A twin is a copy of a thread with one edit. Flip twins change one detail and must change the answer. Invariance twins change only the surface and must not. Traps 1–9, 11 and 14 each get 20 flip-twin pairs, spread across the test splits; a pair counts only if both threads are fully right. A wrong agent answer is the gold answer with one error applied.

6.4 Quality gates and data rules

Every generated batch must pass six quality gates (separate from the four gates in section 2):

1. Schema-valid, 100%.
2. The blind second read agrees with the gold answer, or the thread is fixed or dropped.
3. Human spot check of 5% per batch (50 for the first). The reviewer checks: realistic (scored 1–5), gold answer correct, trap present, only one answer possible. The gold answer is correct in at least 98%.
4. Distribution within 2 points of the variation matrix.
5. No train–test overlap (templates, seeds, long n-grams).
6. Name scan: no real company or person names, zero hits.

Data rules:

• The generator reads only the rulebook; no real data goes in.
• No people appear, and every company is invented; the name scan (gate 6) blocks any thread with a real name.

7. Evaluation

7.1 The results table

One column per gate in section 2. Each row is a model at one size, averaged over three seeds; the cells stay empty until the study fills them.

Every row uses the same prompt and rulebook, temperature 0, and schema-constrained decoding where supported. The three fine-tuned rows train on: gold answers only; gold answers plus distilled reasoning traces; preference pairs of the gold answer against the wrong agent answer.

7.2 Generalisation splits

The 600 test threads form seven splits. In-distribution (grey) is the reference; each of the other six (amber) holds out one thing that training never showed.

Read each split as its gap from the in-distribution score. A model that is strong in-distribution and drops 20 points on composition or rule swap has memorised the training data.

7.3 At par, cost and ablations

• At par (the Accurate gate): the share of threads fully right is within 5 points of the best frontier model. The test: the lower end of a paired bootstrap 95% interval on the difference is above −5. 600 test threads give about ±3 points; 200 would give ±5.5.
• Cost: for a frontier model, measured tokens × list price. For a small model, GPU cost at realistic throughput, plus training and fees spread over 12 months, plus hosting. The result is plotted as accuracy against cost per task, one point per model size.
• Ablations: training size 125 / 250 / 500 / 900; with and without each of reasoning traces, the rulebook in the prompt, and the noisy slice.

7.4 Routing to a human: confidence and unusable input

Each booked trade carries a confidence between 0 and 1. A threshold turns it into a route: above it, the trade is booked straight away; below it, or with any unsure entry, a person reviews it first. A useful model is confident when right and unsure when wrong, so reviewers see the risky trades and little else.

How it is measured:

1. Fit the threshold on the 300-thread calibration split: the lowest value at which auto-booked trades are at least 99% fully right.
2. Apply it to test unchanged and report auto-book precision, coverage (share of trades booked straight away) and the wrong trades that slipped through.
3. Risk–coverage curve: error of auto-booked trades as the threshold sweeps; its area compares models without picking a threshold.
4. Calibration: expected calibration error and a reliability diagram per model and per test split.
5. Flags that matter: recall of gold unsure fields, and the share of reviewed trades that were actually wrong.

Two ways to produce confidence, compared as variants: (a) the log-probability of the booked fields, temperature-scaled on the calibration split; (b) a stated confidence the model is fine-tuned to give, trained on out-of-fold labels of whether its own earlier answers were right. Frontier models are scored the same way, with stated confidence and log-probabilities where the API exposes them.

Unusable input is the second route: the thread goes back to whoever can fix it, with the problem and the fix. Measured by rejection precision and recall (does it reject the right threads, and only those?), and by diagnosis: the right code and fix_target. Flip twins pair a usable thread with the same thread made unusable, so a model can't pass by rejecting everything that looks odd. One problem type, a missing attachment, appears only in test.

Why this is hard for a small model: fine-tuning on gold answers teaches it to sound certain, because every target is stated with full confidence. Calibration must be fitted or trained separately, and it must hold on the held-out splits, where errors are most likely.

7.5 Threats to validity

8. Deliverables and acceptance