When AP teams run OCR or computer vision AI in front of an ERP, a single shared confidence cutoff can fail both ways: high-confidence misreads on totals or invoice IDs reach staging, while clerks spend review time on low-risk text fields.
Setting OCR confidence thresholds for invoice review is a cost-routing decision, not one fixed constant. Test per-field cutoffs whenever review cost and bad-posting cost differ, using field-level errors measured on a labeled batch.
Using two third-party references (not Insonl clients)—Google Cloud Document AI’s evaluation guide and Amazon Textract’s best practices—this post covers field-level error measurement, cost-based cutoffs, and deterministic ERP gates.
Why one global confidence threshold can fail AP teams
Amazon Textract defines a confidence score as a 0–100 number indicating the probability a prediction is correct. For applications sensitive to false positives, AWS advises enforcing a minimum threshold and discarding or flagging lower-scoring results for higher human scrutiny, noting that archival tasks such as handwritten notes might use thresholds as low as 50%, whereas business processes involving financial decisions might require 90% or higher.
Even one financial-grade cutoff is blunt inside a single invoice. A misread amount_due, tax_total, or invoice_number can mispost cash, distort tax filings, or evade duplicate checks; a dropped character in a payment note rarely changes the ledger.
Google Cloud Document AI highlights two traps in aggregate scores:
- Aggregate metrics weight evaluable labels by frequency. Document AI computes
All labelsprecision, recall, and F1 across evaluable labels—excluding any label the processor version cannot extract or the test set does not annotate—weighted by label frequency. Repeated line fields can dominate the headline score and hide weak precision on single-occurrence fields likeamount_due. - Default cutoffs maximize F1, not posting safety. Document AI picks an optimal threshold maximizing F1 (
2 * (Precision * Recall) / (Precision + Recall)), weighting precision and recall equally and choosing the lowest value on a tie. In AP, an accepted false positive written to the ERP can cost far more than a false negative routed to manual review.
Raising a threshold generally increases precision while lowering recall. One shared cutoff forces fields with different error costs onto one operating point.
Measure field-level errors on a labeled batch first
Measure field-level errors on a labeled batch of your own supplier PDFs first. In Document AI, evaluation compares predicted entities against test-set annotations; without a test set, create a dataset and label test documents first. Evaluation runs automatically on train or uptrain, and a manual evaluation is required for updated metrics after modifying the test set or when evaluating a pretrained version (console or evaluate_processor_version).
Stabilize capture quality first so scan defects do not skew baselines. Amazon Textract’s best practices list key input checks:
- Confirm document text is in a supported language and pages are upright.
- Use a high-quality image, ideally at least
150 DPI, and do not convert or downsample files already inPDF,TIFF,JPEG, orPNG. - Keep tables upright and visually separated from graphics; expect inconsistent results on merged multi-column cells or irregular table structures (Textract recommends text detection as a workaround).
Account for four Document AI evaluation rules that shift threshold numbers:
- Single- vs. multi-occurrence labels. Single-occurrence labels (like invoice ID) expect one value per document even if annotated on multiple pages: matching any annotation counts once, and extra annotations are treated as duplicate mentions rather than extra TP/FP/FN counts. Multi-occurrence labels (like line items) score each instance independently.
- Fuzzy matching masks format errors, not numeric differences. Matching checks
entity.typeandentity.mention_text(orentity.normalized_value.text), ignoring bounding boxes on non-tabular labels. Fuzzy Matching normalizes whitespace (including newlines), strips leading/trailing punctuation (!,.:;-"?|), lowercases text, and strips leading/trailing currency symbols onmoneylabels, but does not apply to numeric values (1and1.00still do not match). Measure with fuzzy matching off—or compare both—when a JSON Schema for invoice extraction fields expects exact formats or separate currency codes. - Interpret
False Negatives (Below Threshold)carefully. Document AI ignores predictions below the cutoff even when right and lists those suppressed matches underFalse Negatives (Below Threshold). This count covers correct predictions suppressed by the cutoff; review volume also includes low-confidence wrong predictions and missing values under this routing policy. - Watch tabular parent/child coupling. For tabular entities (Preview, single-level nested tables), parent rows match by combined child bounding boxes (auto-matching when one predicted and one annotated parent exist) and apply the parent’s threshold to all children. If a parent misses, its children are not matched even when child text matches.
In Document AI, Download Metrics exports JSON reflecting the confidence-threshold slider at export, so download a separate export per tested cutoff. For Textract, compare the confidence values for your extracted fields against labeled answers at each cutoff.
Set per-field thresholds from the cost of a bad posting
Replace equal-weight F1 with a per-field cost rule. Estimate two internal figures per field with AP and finance:
C_review: cost to inspect one flagged field crop and confirm or edit it.C_error: expected downstream cost of posting a wrong value (rework, vendor dispute, tax fix, or misapplied payment).
At cutoff t, predictions below t (plus blanks) route to review, while values at or above t auto-pass with FP(t) errors:
batch_cost(t) = field_checks_routed(t) * C_review + auto_accepted_errors(t) * C_error
Calculate batch cost at each tested cutoff and choose the lowest-cost cutoff that satisfies any hard error cap. Fields split into three tiers:
| Field tier | Examples | Threshold posture | Backstop gate |
|---|---|---|---|
| Posting-critical header | amount_due, tax_total, document_currency_code, invoice_number, supplier_tax_id |
High precision cutoff; never optimize F1 | Cross-foot math, master data, duplicate check |
| Matching and coding | purchase_order_number, issue_date, due_date, line quantity, unit_price, line_net_amount |
Medium-to-high cutoff tuned to PO checks | Decimal math and three-way invoice matching |
| Non-posting descriptive | payment_note, description, remit_to_address |
Low cutoff or policy-only review | Never block staging when header math passes |
Illustrative worked example: tuning two fields on a 200-invoice batch
(Illustrative example—not a real company, invoice batch, OCR benchmark, or Insonl client. All counts and unit costs are made up, assuming one ground-truth annotation and at most one prediction per invoice, and that reviewers catch routed errors.)
An AP team labels 200 invoices and tests three cutoffs (0.80, 0.90, 0.97) on two single-occurrence fields:
amount_due: illustrativeC_review = $0.50,C_error = $40.00(wrong posted total forces rework or payment adjustment).payment_note: illustrativeC_review = $0.50,C_error = $1.50(free-text note; minor cleanup if wrong).
Under Document AI’s definitions, a wrong prediction above cutoff counts as both an FP and an FN, while suppressed correct predictions appear under False Negatives (Below Threshold):
| Field | Cutoff | TP/FP/FN (below) | P/R/F1 | Field checks routed | Review + Error = Total |
|---|---|---|---|---|---|
amount_due |
0.80 |
190/6/10 (2) | 96.9%/95.0%/96.0% | 4 | $2.00 + $240.00 = $242.00 |
amount_due |
0.90 |
182/3/18 (10) | 98.4%/91.0%/94.5% | 15 | $7.50 + $120.00 = $127.50 |
amount_due |
0.97 |
168/1/32 (24) | 99.4%/84.0%/91.1% | 31 | $15.50 + $40.00 = $55.50 |
payment_note |
0.80 |
184/5/16 (8) | 97.4%/92.0%/94.6% | 11 | $5.50 + $7.50 = $13.00 |
payment_note |
0.90 |
162/2/38 (30) | 98.8%/81.0%/89.0% | 36 | $18.00 + $3.00 = $21.00 |
payment_note |
0.97 |
134/1/66 (58) | 99.3%/67.0%/80.0% | 65 | $32.50 + $1.50 = $34.00 |
Two findings stand out among the three tested cutoffs:
- F1 picks the costliest cutoff for
amount_due. Both fields peak in F1 at0.80(96.0%and94.6%), yet0.80passes6bad totals ($242.00). Raisingamount_dueto0.97drops F1 to91.1%, adds22False Negatives (Below Threshold), and cuts total cost to$55.50. - One global
0.90cutoff loses on cost and workload. Flat0.90routes51field checks (15 + 36), passes3bad totals, and costs$148.50. Splitting cutoffs (0.97onamount_due,0.80onpayment_note) cuts total cost to$68.50, trims bad totals to1, and lowers review volume to42field checks (31 + 11).
When only amount_due fails its cutoff, highlight that field beside its PDF crop so clerks verify one value instead of re-keying the header.
Backstop confidence scores with deterministic gates and drift checks
A high score never proves accounting validity: OCR can score highly on a misclassified credit note, a swapped decimal separator, or an inactive vendor ID. Keep deterministic gates between extraction and ERP writes:
- Schema and e-invoice gates. Normalize absent optional values to explicit
nullwhere the schema permits it, then validate extracted PDFs against a strict JSON Schema for invoice extraction fields. For XML e-invoices, skip OCR and enforce schema and business rules as in Peppol invoice validation before ERP import. - Exact decimal cross-footing. Recompute line totals, tax subtotals, and
amount_duein exact decimals. Any mismatch overrides a high score and forces review. - Stateful ERP controls. Run duplicate supplier invoice detection and three-way invoice matching before staging or posting.
Do not confuse operational telemetry with accuracy monitoring. Document AI’s monitoring dashboard tracks throughput and latency—Number of successfully processed pages and Total number of docs sent to processing (by location, processor type, processor_id, processor_version_id, status, and processing_type), plus Sync processing latency, Batch processing latency per document, and Batch processing operation count (by location, processor_type, processor_id, processor_version_id, and status)—not live field precision. Field accuracy can fall even when throughput and latency look stable. Log clerk edits, sample them into your labeled test set, and re-run per-field evaluations after processor updates or supplier onboarding.
Audit your invoice review thresholds before scaling automation
Insonl is a Ho Chi Minh City engineering studio building computer vision AI and ERP systems. If your AP queue is overloaded with low-value flags—or passing OCR misreads to posting—start with our free AI workflow audit.
The audit is free; work after it is quoted separately.
Sources
- Google Cloud (third-party docs), Document AI: Evaluate performance: https://docs.cloud.google.com/document-ai/docs/evaluate
- Amazon Web Services (third-party docs), Amazon Textract: Best practices: https://docs.aws.amazon.com/textract/latest/dg/textract-best-practices.html
- Insonl: Computer Vision AI · ERP Systems · Free AI workflow audit · JSON Schema for Invoice Extraction Fields · Three-Way Invoice Matching · Duplicate Supplier Invoice Detection · Peppol Invoice Validation
