Enterprise banks, payment processors, and healthcare customers needed financial documents classified at production accuracy.
The honest part.
No dedicated engineers, no data-science team on call. Every prompt, eval, and architecture decision was mine to make and mine to fix: shipped by directing AI, not by handing off.
The window between commitment and the first production traffic. The schedule shaped the architecture: buy nothing, build the smallest thing that can be trusted.
Itemize's customer names stay confidential. The numbers, the architecture, and the process are real, and verifiable in conversation.
The paper trail, redacted.
Real documents drove this: PRD, eval plan, rollout memo. Client specifics stay under NDA; the process doesn't.
How success was scored: final-output grading on single-shot extraction, one answer checked against hand-labeled ground truth. Not agent-trajectory scoring. Calling these agent evals would be overselling them.
100 documents per type across roughly 100 document types (invoices, receipts, statements, tax forms, healthcare forms, non-invoice documents), roughly 10,000 in the golden set. I hand-labeled all the ground truth; a business analyst double-checked every set. Built or refreshed at every customer onboarding, whenever a new layout showed up in production, and monthly regardless.
The bar every version had to clear before shipping: 99% per field on the golden set, plus zero regressions against the previous version. Fail either and the release was blocked. I held the go/no-go on every rollout.
Three models vote. One gate decides.
models drawn from: anthropic · openai · gemini · aws in-house · ocr: amazon
Accuracy was climbed, not found.
Before / after.
drag the handle · arrow keys work too
- Every document touched by a human
- Hours of turnaround
- Accuracy capped by fatigue
- Humans see only flagged exceptions
- Seconds of turnaround
- 98.51% in production, measured not vibes
The day I stopped the line.
A healthcare customer needed 26 line-item fields and 7 header fields extracted from claim forms, delivered to their production SFTP as a zip: pipe-delimited CSVs, plus every claim split into single-page PDFs tagged with GUIDs.
A personal habit, not a ticket: about two hours a day auditing random documents in production. One of those audits caught 400 to 500 of every 1,000 documents landing in the low-confidence queue at under 60% accuracy, while the rest ran around 95% per line-item field on the same claims. I stopped the production line myself, before the customer saw it. Then I sized the problem: a hand audit of roughly 400 failing documents, three buckets.
Asked the customer to rescan, best effort. Then changed extraction to a two-way check: Amazon OCR output compared against the LLM extraction, with the LLM making the final call per field. That lifted even the best-effort low-quality documents to the 99% per-field bar.
The model was seeing a blank claim form. The developers added a PDF flattening script at the ingestion stage, so form data is never dropped before extraction.
Routed to human review, by design. Not every document should be automated.
Today under 5% of this customer's claims route to human review, decided by one number: a per-document confidence score below 90%.
What I'd do next.
The hard gate is deliberately conservative: it routes anything ambiguous to humans, which costs automation rate. The next lever is a confidence-calibrated gate trained on six months of production disagreements, which should recover most of the flagged volume without giving back accuracy. After that: the same pattern, applied to the next document family.