Illustrative scenario. An operations team tests two extraction pipelines on twenty clean digital PDFs from its three largest suppliers. Both look accurate in the demo. Weeks after rollout, exceptions spike on skewed warehouse scans, merged-column freight tables, and new regional vendors absent from the pilot folder.
Preventing that gap starts with an OCR evaluation dataset for mixed supplier scans built before you choose a vendor or tune thresholds. Stratify real documents by layout, legibility, and supplier familiarity, freeze the labeled holdout, and compare per-field error rates across candidate pipelines on the exact same files.
This guide draws on third-party documentation (not Insonl clients) from Google Cloud Document AI and Amazon Textract. All document counts, manifest entries, supplier labels, and error figures below are illustrative and made up, not from a real company, benchmark, or Insonl engagement.
Why random samples hide supplier-scan failures
Pulling the latest hundred files from an inbox rarely tests what breaks production. Three distortions appear right away:
- Supplier concentration: A few high-volume suppliers often generate most monthly files as clean digital PDFs, crowding out long-tail layouts and degraded scans.
- Template leakage across splits: Random splitting can put documents from the same supplier template in both training and test sets. Google Cloud Document AI’s dataset guide supports
Training,Test,Auto-split(random shuffling into train and test), andUnassigned. To measure generalization to unseen suppliers, assign entire suppliers to the holdout. - Frequency-weighted averages: Google Cloud Document AI’s evaluation guide weights
All labelsprecision, recall, and F1 by label frequency. Repeated line-item columns swamp rare header fields likeamount_dueorsupplier_tax_id, while clean PDFs mask errors on poor scans.
Stratify the holdout by layout, legibility, and supplier
Tag every candidate document across three independent axes before assigning it to the holdout:
- Layout complexity. Separate single-page forms from multi-page tables and irregular tables. Amazon Textract’s best practices note that tables extract best when visually separated from graphics and upright, and warn of inconsistent results on merged multi-column cells or tables whose rows, columns, or cells vary within the same table (AWS recommends text detection as a workaround). Tag document subtypes—invoices, delivery notes, credit memos—on this axis if your workflow mixes them.
- Legibility and capture quality. Separate native digital PDFs from clean scans and degraded scans (skew, stamps over numbers, handwritten notes, low contrast, or low resolution). Amazon Textract advises upright pages in a supported language, ideally at least
150 DPI, and keeping files already inPDF,TIFF,JPEG, orPNGintact without re-converting or downsampling. Explicit legibility tiers isolate capture defects from extractor bugs. - Supplier familiarity. Separate recurring high-volume suppliers from supplier-disjoint (unseen) vendors kept strictly out of prompts, few-shot examples, custom rules, and training splits.
(Illustrative stratification plan for a 120-document holdout—made-up quotas, not a universal minimum or real benchmark.)
| Axis | Bucket | Illustrative count | What it isolates |
|---|---|---|---|
| Layout | Single-page flat header + simple table | 40 | Baseline field localization |
| Layout | Multi-page or dense line-item table | 50 | Page breaks and row pairing |
| Layout | Irregular table (merged cells, overlays) | 30 | Table-parser breakdown and fallbacks |
| Legibility | Native digital PDF | 30 | Parsing and schema mapping |
| Legibility | Clean scan (upright, >=150 DPI) | 50 | Standard OCR accuracy |
| Legibility | Degraded scan (sub-150 DPI, skewed, stamped) | 40 | Noise and capture resilience |
| Supplier | Recurring known suppliers | 70 | Core-volume template stability |
| Supplier | Unseen long-tail suppliers (disjoint) | 50 | Generalization to new layouts |
Store split membership, strata tags, and immutable file hashes (SHA-256) in an external manifest outside any vendor console. Document AI’s labeling docs say that Document AI exports labeled JSON Document files, and that the JSON omits dataset metadata, including processor schema, split assignments, and labeling status. The export places files in Train, Test, and Unassigned folders according to their assignments.
Write ground-truth labeling rules before touching a model
Anchor label names and canonical formats to your downstream integration contract—detailed in a JSON Schema for invoice extraction fields—so decimal precision, currency codes, dates, and null values are unambiguous before labeling begins.
Four rules drawn from Document AI’s dataset and labeling docs prevent label drift:
- Specify common and corner cases. State whether currency symbols belong inside
amountlabels, how credit notes are signed, and how missing fields are marked. Label every occurrence even forREQUIRED_ONCEorOPTIONAL_ONCEfields (such as aninvoice_idrepeated on each page). For parent-child tables, label child entities only; parent containers update automatically. - Block unreviewed auto-labels from the holdout. Document AI forbids using auto-labeled documents in test or training sets until manually reviewed, corrected, and marked as labeled. Unchecked auto-labels bake baseline errors into the benchmark.
- Keep cross-engine ground truth separate from vendor training annotations. For processor training, Document AI warns against manually overriding an OCR misread, which renders the example unusable for training. For a vendor-neutral holdout, ground truth must match what is printed on the scan so an engine that reads a smudged digit accurately is not penalized.
- Calibrate and dual-review annotators. Have trainees independently label a shared batch and check their work against a reference key. Google recommends reviewing about the first 10 documents each new annotator labels for the use case. For this holdout, we recommend having a second trained annotator verify every test document.
(Illustrative manifest entry—made-up IDs, hashes, and values.)
{
"holdout_version": "v1.0",
"doc_id": "scan_0047",
"sha256": "a3f9c2e1...illustrative_hash",
"source_format": "tiff",
"strata": {
"layout": "irregular_table",
"legibility": "degraded_scan",
"supplier_group": "unseen_long_tail"
},
"ground_truth": {
"invoice_number": "INV-2026-0814",
"currency_code": "VND",
"amount_due": "18450000",
"purchase_order_number": null
}
}
Compare per-field error rates on the same frozen holdout
Run every candidate extractor—cloud OCR, multimodal model, or new processor version—on the same source files, and keep any conversion or downsampling consistent across candidates. AWS advises preserving supported files as-is when uploading them to Textract. Lock four scoring controls:
- Audit document counters for silent drops. In Document AI,
get_evaluationexposesdocument_counters(input_documents_count,invalid_documents_count,failed_documents_count,evaluated_documents_count). Score every expected field on invalid or failed files as a miss rather than shrinking the denominator. - Pin exact vs. fuzzy matching rules. Document AI matches non-tabular entities on
entity.typeandentity.mention_text(orentity.normalized_value.text), ignoring bounding boxes. Fuzzy Matching normalizes whitespace and newlines, strips leading/trailing punctuation (!,.:;-"?|), lowercases text, and strips leading/trailing currency symbols onmoneylabels, while still treating1vs.1.00as distinct. Measure exact match against your canonical schema first, with normalized scores alongside it so formatting bugs remain visible. - Separate single-occurrence headers from tabular rows. Under Document AI, single-occurrence labels count at most one match per document, while multi-occurrence labels score every instance. For tabular evaluation, the guide describes one level of parent-child nesting in Preview. Document AI compares the combined bounding boxes of each parent’s children, with an automatic match when there is exactly one predicted parent and one annotated parent. It applies ordinary entity matching to children within matched parents and skips child matching for unmatched parents. Separating headers from tables distinguishes OCR errors from row-segmentation errors.
- Lock export settings before tuning cutoffs. Document AI’s
Download Metricsexports JSON at the active confidence slider. Compare raw or identically constrained outputs first, then tune per-field routing cutoffs on the chosen extractor—covered in how to set OCR confidence thresholds for invoice review.
(Illustrative comparison on the same 120-document holdout—made-up pipelines and error counts, not a real benchmark. Failed scans count as misses on every field. Metric definitions follow Document AI’s evaluation docs.)
| Slice / field (denominator) | Pipeline A errors | Pipeline B errors | Takeaway |
|---|---|---|---|
| Failed / unscored scans (120 docs) | 0 (0.0%) | 2 (1.7%) | Pipeline B drops 2 degraded unseen scans |
amount_due — all scans (120) |
3 (2.5%) | 6 (5.0%) | Pipeline A holds tighter on posting totals |
amount_due — degraded scans (40) |
2 (5.0%) | 5 (12.5%) | Capture noise drives Pipeline B’s gap |
invoice_number — unseen suppliers (50) |
2 (4.0%) | 7 (14.0%) | Pipeline B overfits recurring layouts |
line_net_amount — irregular tables (150 rows) |
24 (16.0%) | 9 (6.0%) | Pipeline B handles merged columns better |
A single blended average would hide that trade-off. If you are scoping a document-capture build measured on your own samples, see Insonl’s computer vision AI practice.
Version the holdout without leaking production fixes
Treat the evaluation holdout like versioned code:
- Freeze
v1.0for regression checks. Never swap files inside an existing tag, or score changes cannot be attributed to the model versus the dataset. - Promote production misses into
v1.1. Capture verified edits from your exception queue—using the field-level audit trail in what an invoice review screen should show before approval—into a dated release while preserving supplier-disjoint rules. - Re-run evaluation after every change. Document AI evaluates automatically on train or up-train, but requires a manual run (console or
evaluate_processor_version) after editing the test set or evaluating a pretrained version. Log holdout version, extractor version, preprocessing commit, and matching settings with every scorecard.
Next step
Bring a sample of your supplier scans and posting rules to Insonl’s free AI workflow audit. You receive a written audit with a workflow map, recommendations, and a fixed-scope estimate. The audit is free; Insonl charges only for follow-on work if you decide to proceed.
Sources
- Google Cloud Document AI, Evaluate performance: https://docs.cloud.google.com/document-ai/docs/evaluate
- Google Cloud Document AI, Create dataset: https://docs.cloud.google.com/document-ai/docs/create-dataset
- Google Cloud Document AI, Label documents: https://docs.cloud.google.com/document-ai/docs/label-documents
- Amazon Textract, Best practices: https://docs.aws.amazon.com/textract/latest/dg/textract-best-practices.html
- Insonl, Set OCR Confidence Thresholds for Invoice Review: https://insonl.com/journal/ocr-confidence-thresholds-invoice-review
- Insonl, A JSON Schema for Invoice Fields Your ERP Can Validate: https://insonl.com/journal/json-schema-invoice-extraction-fields
- Insonl, What an Invoice Review Screen Should Show Before Approval: https://insonl.com/journal/invoice-review-screen-before-approval
- Insonl, Computer vision for operations and inspection: https://insonl.com/services/ai/computer-vision-ai
- Insonl, Free AI Workflow Audit: https://insonl.com/ai
