Before changing the prompt, model, or lookup behind an operations workflow, run the current version and the candidate on the same historical cases. Sample enough routine work to see whether the change makes daily handling worse. Include past failures where a wrong decision meant missed work, avoidable rework, or a delayed escalation. Agree which failures stop a release before anyone sees the results.
Illustrative scenario. A purchasing team wants an AI assistant to triage supplier emails about purchase orders. It reads a message and the order record, then proposes a route for a buyer to approve. Most date changes can go into the normal buyer queue. If a proposed delay affects an order already flagged for a production shortage, purchasing policy requires escalation to production planning. If the order cannot be linked reliably, the case goes to manual triage. The test ends at the routing suggestion and its supporting evidence. Buyers still own purchase-order updates.
Define the decision and reconstruct its inputs
Pick the action the workflow is expected to take and write down what the next operator needs to act on it. In this example, the response needs a route, the matched purchase-order line where available, and the recorded fact that supports the route. For a shortage escalation, check whether the assistant found the right order line and used the shortage flag. A correct queue paired with evidence from a different order line fails the case.
Rebuild the view available when each case reached the queue. Capture the supplier message and attachment, the order-line state, and the planning flag as of the decision time. Use timestamps on the source records to exclude facts added later. If a planner flagged a shortage after the email arrived, that flag cannot justify an immediate escalation in the replay. If the necessary state cannot be reconstructed, record the gap and leave the case out of any check that depends on it. Keep the record references in the case manifest, and give the runner only the approved fields needed for the decision.
Sample routine work and costly failures
Start the routine sample from the intake log. Choose a window that covers the operating conditions you intend to support. Group cases by differences that can change the decision, such as whether the order reference was clear, whether the message covered several order lines, and whether it came with an attachment. Draw cases from each group and record the source population and selected counts. Include straight-through work and ordinary manual reviews, even when both ended without an incident.
Build a second sample from escalations, buyer overrides, reopened work, and corrections in the order history. Trace each case to the original input and write down the consequence the release should guard against. A costly case may be a shortage escalation that a buyer missed, or one a planner caught before it caused a delay. Keep both if the decision can be reconstructed. Report results for this deliberately enriched sample on its own. Use the routine sample for conclusions about everyday handling, and account for the sampling mix if you estimate a production rate.
OpenAI recommends designing evaluations for the actual task, drawing on historical and production data where relevant, and including ordinary and edge cases with expert labeling. For an example of choosing sample groups when incoming documents vary in layout and legibility, see Insonl’s OCR test-set guide. OpenAI, Evaluation best practices · Insonl, Build an OCR Test Set for Mixed Supplier Scans
Label the action the policy requires
Have a buyer or planning lead review the frozen inputs against the policy intended for the release. Record the expected route, the evidence required, and the severity of an incorrect action. Store what staff actually did in a separate field. A historical override can help the reviewer find a bad decision, and the policy owner can confirm whether the override should become the expected behavior. When several routes are explicitly allowed, record the permitted set. Send unresolved labels to the policy owner before counting those cases in the gate.
Keep a holdout for the release decision
Split by the unit that ties related records together. In the purchasing example, keep the original email, replies, revised attachments, and related order lines in one split. Reserve an untouched holdout containing routine and costly cases. Keep its inputs and labels out of prompt examples, rule tuning, model training, and retrieval data used for tuning. If you reserve a later time window to test newer intake patterns, check for open orders or threads that cross the cutoff before assigning cases.
Give developers a working set for debugging. Have the person responsible for the release control the holdout and log when individual cases are disclosed. When engineers inspect a failing holdout case and tune against it, move that case into the working regression set and replace it with a fresh case that checks the same requirement. Version the case list, labels, and scoring rules so a release report identifies exactly what was tested. OpenAI illustrates using a held-out set with explicit success metrics. The grouping and replacement procedure here is for this kind of historical replay. OpenAI, Evaluation best practices
Agree what blocks the release
Write the gate with the workflow owner before opening the holdout. Set a floor for acceptable routine routing and a limit on how far the candidate may fall below the current version on the same cases. Include a ceiling for unnecessary manual reviews based on the team’s capacity. List the errors that block release on their own and say how to handle incomplete runs. Use the following rules as an illustrative gate for the purchasing workflow.
| Check | When to block |
|---|---|
| Required escalation | A case with a recorded shortage flag is sent to the normal buyer queue or held without the required planning handoff. |
| Evidence and fallback | A proposed route cites the wrong order line, or an unresolved match proceeds without manual triage. |
| Routine routing and review load | The candidate misses the agreed routing floor, exceeds the allowed drop against the current version, or sends more cases to unnecessary review than the team agreed to handle. |
| Run completeness | Required cases lack a scorable result because the runner, fixture, or scorer failed. Repair the test and rerun it before deciding. A timeout produced by the workflow itself is scored against the expected fallback. |
Report counts and denominators for each check. Show cases the current version passes and the candidate fails, alongside improvements in the other direction. If the candidate routes more routine messages correctly but misses a mandatory planning escalation, block the release. If both versions miss the same mandatory escalation, the candidate still fails the gate. An evaluation result applies to the cases tested. Use operational data for decisions about staffing or cost per accepted task. Insonl, Calculate the Cost per Completed Task in an AI Pilot
Run the same cases against each change
Replay both versions against identical stored records in a test environment. Capture model and prompt versions, routing rules, lookup fixtures, and raw outputs. Use exact checks for the route and order-line reference when the policy supplies a fixed answer. When judging a written reason, use examples that reviewers have scored to check that an automated grader applies the rubric as intended. If outputs can vary between runs, agree on the repeat count and how any critical failure will be scored before testing.
Save the pass or block decision with its failed case IDs. When a new production failure belongs in scope, preserve the input available at the decision and have the workflow owner review the label before adding it. Recheck the gate when the workflow changes. OpenAI recommends calibrating automated scoring with human feedback, running evaluations on every change, and growing the eval set over time. OpenAI, Evaluation best practices
If the evaluation is part of a broader implementation, use the workflow decision and its release gate as requirements for the build. Insonl, Generative AI
Sources
- OpenAI, Evaluation best practices
- Insonl, Build an OCR Test Set for Mixed Supplier Scans
- Insonl, Calculate the Cost per Completed Task in an AI Pilot
- Insonl, Generative AI
- Insonl, Free AI workflow audit
Request an audit for one workflow
Bring one operational decision, the records your team uses to make it, and a past failure you want a release to catch. The audit is free. Follow-on work is charged separately. Request an AI workflow audit from Insonl
