My Code, My Test, and My Prompt All Agreed. All Three Were Wrong.

A friend handed me a grocery receipt to test my receipt-scanning app, Receipt Tracker, which turns a photo into categorised line items. It was a dense Massy Stores run, 37 items, and the photo wasn't the sharpest. The app got most of it, but a few items were missing and the totals didn't add up. He was generous about it, impressed at how much it had pulled off a bad photo. I was less impressed: a scan you have to double-check by hand for missing items isn't saving you the work.

The model doing the reading was Google's Gemini 2.5 Flash, and on that receipt it kept dropping lines and inventing quantities. Upgrading to Gemini 3.5 Flash fixed it cleanly: all 37 items, every run, reconciling to the printed $254.03 to the cent. The upgrade wasn't cheap. Gemini 3.5 Flash lists at $1.50 per million input tokens against 2.5 Flash's $0.30, five times the price, and enough of a jump that plenty of developers complained when it landed. I paid it for the quality and kept it.

The model is the app's only real running cost, and I'd just chosen to pay more for it. So when I noticed DeepSeek's V4 models going for a fraction of Gemini's token price, the obvious question was whether the trade could run the other way: the same receipt quality for a fraction of the cost.

What followed turned up two real bugs and improved the prompt for the model I wasn't trying to replace. Most of it was me being wrong. The uncomfortable part is why I nearly shipped the wrong call anyway: my code, my test, and my prompt all agreed, and I trusted that agreement without noticing all three came from the same place.

DeepSeek can't even see

The investigation started with DeepSeek, whose V4 models go for a fraction of Gemini's price, and ended about thirty seconds later. On OpenRouter, every DeepSeek model lists text as its only input modality. V4 does have a "Vision" mode, but it only lives in the chat app at chat.deepseek.com; the API itself is text-only. The capability exists; you just can't build on it.

Qwen looks great, then fails by $88

The real candidate was Qwen3-VL-32B: a proper vision model, strong on document OCR benchmarks, and about 18× cheaper than Gemini per receipt. I already had an eval harness (14 receipt fixtures, 135 assertions) so I pointed it at Qwen.

It passed about 85%. I trusted Gemini as the model that just worked, and honestly I'd have guessed it passed everything, though I wasn't checking that closely. So Qwen coming in lower made me wary, and one failure in particular jumped out, because it was about money. Reconciliation is a check that runs in code after the model: add up the line items it pulled out, and see whether they match the printed total. This was the same dense receipt Gemini reconciles to the cent, so when Qwen's numbers didn't add up on it, it looked like a straightforward Qwen problem. I checked directly, summing each item's price × quantity:

Qwen3-VL-32B on the 37-item Massy receipt:
  printed total:   $254.03
  sum of items:    $341.99
  reconciliation gap: $87.96   ← ouch

Eighty-eight dollars off. I dug in and found a clean culprit: promotional quantities. On the "Gala Apples" line, Qwen set quantity: 8 but kept the line's printed total ($5.52) in the price field, so price × quantity came out to $44.16 instead of $5.52. I wrote it up as a systematic Qwen failure, ran the bigger 235B model to confirm it wasn't just a small-model problem (it wasn't; the same gap showed up at 235B), and started drafting the verdict: cheaper, but the totals don't add up. Keep Gemini.

I'd run this whole investigation with my coding agent, Claude Code, so that verdict was as much its conclusion as mine, and I almost took it. But first I asked it the question I'd been skipping: was the prompt itself tuned for Gemini, giving it an unfair edge?

The bug was mine

Answering that meant running Gemini through the same probe. Gemini extracts this receipt to the cent, so it should have passed clean. Instead my probe put Gemini just as far off, at $337.11, an $83 gap on a receipt I knew reconciles perfectly. That was the giveaway: Gemini's extraction hadn't changed, so the same line items that reconcile to $254.03 could not suddenly be $83 off. The probe was doing the wrong math, on both models.

In my schema, an item's price field holds the line total (the extended price actually charged), rather than a unit price. The prompt's own examples say so: Drumsticks, quantity: 0.862, price: 17.20. So the correct reconciliation is Σ price, full stop. My probe computed Σ (price × quantity), which double-counts every multi-quantity line and manufactures ~$80 of phantom gap out of thin air.

With the correct metric, Σ price:

Model	Σ price	Gap vs printed total
Gemini	$254.03	$0.00 ✓
Qwen	$258.68	$4.65 (one extra line, not promo leakage)

The "$88 Qwen failure" never existed. It was an artifact of multiplying the wrong two numbers. And the same bug was hiding in my actual test suite's reconciliation assertion, inflating the sum on any multi-buy line exactly the way my probe did.

The promotional quantity: 8 I'd flagged as a Qwen defect? Production already handles it: a deterministic step re-derives the quantity by dividing the line total by the per-unit price the model extracts into its own field from the receipt's "N @ price" line, and the integrity guard sums line totals rather than price × quantity. I'd benchmarked raw model output and ignored the production code that normalises the exact field I was panicking about. Gemini and Qwen emit the same promo quantities; production fixes both.

There was a third instance, too, in the prompt itself. A few lines apart, it handed the model two contradictory rules:

"Do NOT multiply quantities — the receipt already shows the correct totals."

"RECONCILIATION CHECK: sum every extracted item's price × quantity."

The same wrong formula, written three times, by someone who plainly knew better. That's the real tell: you don't fix this by being more careful. The field is called price but it holds a line total, and "price × quantity = total" is the formula your hands type on autopilot. The durable fix is to name the field for what it holds, lineTotal, so that multiplying it by a quantity reads as the obvious nonsense it is.

And there's a deeper reason all three copies survived: no reconciliation test summed line totals on a multi-quantity receipt. My one reconciliation check was gated to quantity-1 restaurant receipts, where lineTotal × quantity and Σ lineTotal give the same answer. The bug lived in the exact blind spot where no test was looking, so I added the missing one: sum the line totals on the dense 37-item Massy receipt and confirm it matches the printed total.

Confirm what a field means before you compute with it; a line-total-vs-unit-price mix-up fabricates gaps that look exactly like a model defect. And benchmark the production pipeline rather than raw model output, because deterministic post-processing can erase the very differences you're measuring.

The real gap was my prompt

With the metric fixed, I ran the honest comparison: both models, the same 135 assertions, three runs each.

Model	Overall (of 135, 3-run mean)
Gemini 3.5 Flash	131.7
Qwen3-VL-32B	116.0

A real ~15-assertion gap, but not in reconciliation (Qwen passes the money guard). When I bucketed Qwen's failures, the largest cluster was category assignment: of seven category checks, it failed six. And the pattern was almost comically specific:

Receipt	Expected	Qwen gave
Chefette, KFC	Fast Food	Dining
Heaven Sent (baby store)	Baby Care	Baby & Children
Massy (formula)	Formula	Baby & Children > Formula

Qwen wasn't miscategorising. It understood every receipt fine. It was returning the parent category (or the full Parent > Sub path) instead of the leaf subcategory my taxonomy wanted.

My prompt has a rule for exactly this: "use the subcategory, never the parent." But look at how it's written: "never return just 'Groceries', always pick the specific subcategory under it." Every example is a grocery example. That rule lives in a prompt I've revised across more than 40 commits, nearly every one a reaction to something Gemini got subtly wrong on a Bajan receipt. Gemini generalised the rule to all categories. Qwen read it literally, applied it to groceries, and returned parents for everything else.

The benchmark wasn't measuring "which model is better at receipts." It was measuring which model better fits a prompt I'd unconsciously shaped around Gemini.

The fix that helped both models

If the gap is prompt-fit, it should be promptable. I added one closing rule that restates "leaf subcategory only" for every tree, not just groceries:

• restaurant / fast-food → Fast Food (not Dining, and not the ingredient)

• baby → Baby Care

• hardware → Home

Then I A/B'd it, three runs per condition, both models:

Condition	Category checks (of 7, 3-run mean)	Overall (3-run mean)
Qwen baseline	~1 / 7	116.0
Qwen + leaf rule	7 / 7	121.3 (+5.3)
Gemini baseline	~6 / 7	131.7
Gemini + leaf rule	7 / 7	132.7 (no regression)

The cheap model's category adherence went from roughly 1/7 to 7/7, and because those are 3-run means, that climb is worth about the +5.3 the overall score gained: the whole improvement was category, with nothing regressing elsewhere. The rule wasn't a Qwen-specific patch, either. It nudged Gemini's flaky case to solid too, with no regression anywhere. A model-neutral improvement that I only discovered by trying to onboard a model my prompt had quietly excluded.

That fix shipped, to the Gemini path I'm actually running.

So did I switch?

No. I'll be honest about why, because the honest reasons are the useful ones: this app is parked, I'm effectively its only active user, and at my volume the 18× price difference is cents per receipt. There is no production case for swapping. Qwen3-VL-32B turned out to be a genuinely viable, far cheaper model, about 15 checks behind a frontier model and most of those look tunable, but "viable and cheaper" doesn't beat "already working" when the savings round to zero.

What I actually got out of it was better than a model swap:

• Three copies of one bug, found and killed: the probe, the test, and the prompt all summed price × quantity.

• The field renamed price → lineTotal where the code does money math, plus the reconciliation test the bug had been hiding from.

• A model-neutral prompt improvement for the model I kept.

• A sharper picture of what my benchmark was really measuring.

When a challenger model "fails," the first suspect should be your own measurement: the metric, the pipeline, the prompt you wrote for one model without realising it. I didn't switch in the end, but that was because Qwen is narrowly behind and the savings are cents at my volume. It was never because it failed a test it actually passed.

What I'd actually tell another developer

I've argued before that catching where the AI is wrong is the core of the job. This is me catching myself. My coding agent generated the original schema, the first prompt, and the tests, and I revised that prompt across 40-plus commits afterward without ever going back to question what it had built underneath. So the same wrong assumption about what price meant was sitting in all three, and all three agreed with each other.

That agreement was worthless, and it helps to see why. The wrong formula was price × quantity: the prompt told the model to sum it, the reconciliation code computed it, and the test asserted it. Watch what it does to two kinds of line, remembering that price already holds the line total:

Line	price × quantity	Result
single item (quantity: 1)	3.99 × 1 = 3.99	✓ matches the line total
multi-buy (quantity: 8)	5.52 × 8 = 44.16	✗ line total is 5.52

Every test fixture I had was a single item at quantity 1, where price × 1 and the line total are the same number, so the formula looked correct and the tests stayed green. It only breaks on a multi-buy line like the Gala Apples (8 @ 0.69), and no test ever contained one. The prompt taught the model the formula, the code checked with it, and the test agreed because it never met the input that exposes it.

When the code, the test, and the prompt all trace back to one source, their agreement doesn't verify anything. It echoes the same assumption back to you. You don't catch that by reading the code more carefully, because a careful read just runs the assumption again in your head. You catch it by checking against something the model never generated: the printed total on the receipt, a known-good incumbent, an invariant from the real world.

---

The receipt scanner is Receipt Tracker; the code is private but I'm happy to talk shop about any of the above.