insonl
← journal

An AI Readiness Checklist for One Operations Workflow

date
23 November 2026
by
insonl
read
6 min read

Before proposing an AI pilot, ask your operations lead and system owner to walk one real case through four checks. Can a pilot read the necessary records with approved access? Can reviewers say what a correct result looks like? Who owns the rules for exceptions? What happens to open work if the pilot stops? Score each answer against something the team can inspect or repeat.

Illustrative scenario. A support team wants help routing inbound delivery complaints. Today a coordinator reads the message, checks the linked order, and assigns the ticket to the team that can act. The proposed pilot suggests a queue and a reason. A coordinator accepts or corrects the suggestion. The scope is small enough to test with existing tickets and broad enough to expose gaps in data access, rules, and recovery.

Define the decision and when it is complete

Pick a single intake channel, a set of eligible cases, and the point at which the next team can take over. In the delivery example, the workflow begins when an email becomes a support ticket and ends when the right queue accepts it or a coordinator takes ownership of an exception. A suggested queue, a saved assignment, and an accepted handoff are three distinct states. Record which one the pilot is allowed to produce and which one you will count as complete.

Write down what stays in the existing process. For this example, staff handle refunds, carrier claims, and customer replies. For tickets missing an order reference, the pilot can recommend manual triage. If you still need to trace what operators open, what gets written, or where cases wait, use the Insonl AI workflow audit guide.

Score the four readiness checks

Score the proposed scope as it works today. Use 0 for missing evidence, 1 for partial evidence, and 2 when the team demonstrates the check end to end. Record the evidence, the person who owns the next action, and the date of the check beside each score. A missing permission, an unresolved routing rule, and an untested stop procedure each need their own fix.

Check 0 · Missing 1 · Partial 2 · Demonstrated
System access The team has screenshots or a permission request. An approved export or test account covers only part of the proposed flow. The intended account can retrieve the necessary fields and perform the exact action in scope in an approved test environment.
Example quality Sample inputs have no reviewed outcomes. Reviewers have labelled routine cases, with gaps in common exceptions or handoffs. An approved, labelled set covers the proposed case mix and material exceptions, with a held-out set for the decision.
Policy ownership Nobody can settle a disputed route. An owner is named, and some exception rules still depend on verbal instructions. The owner signs off on versioned rules, resolves sample disputes, and names who can override a suggestion.
Reversibility There is no tested way to stop the pilot or recover its work. The team can stop new suggestions, and open cases have yet to be reconciled in a rehearsal. An operator stops the pilot and accounts for pending and previously assigned cases under the proposed scope.

Prove the exact lookup

With an account approved for the pilot, retrieve a permitted test ticket and follow its order reference to the shipment status. Repeat with a missing reference and with a record the account has no permission to read. Write down how operators will distinguish missing data, denied access, and a system timeout. Each should lead to a defined next action. A manager’s screen can identify the fields, while the approved account proves what the pilot can actually retrieve.

If the pilot will write assignments after approval, test the write in a sandbox and inspect the resulting ticket under the receiving team’s account. Check that it arrived in the intended queue and that the reviewer’s approval is visible. If all you can get today is an approved export, frame the next step as an offline replay and document which lookups it cannot reproduce.

Label each case using the information available at intake

Give reviewers the message and the order information the coordinator could see when the ticket arrived. Record the expected queue, the reason, and whether staff should take over. An eventual resolution can help reconstruct a case, but a carrier update posted the next morning could make a historical routing decision look easier than it was. If a case lacks the original context, keep that gap visible and use it only for checks the available evidence supports.

Include the routine delivery question, the missing order reference, and a ticket that mentions both a delay and a payment dispute. Keep related replies together when splitting cases for development and evaluation to avoid testing on another message from a thread already used to tune the workflow. OpenAI’s guide recommends task-specific evaluations that reflect real usage and include typical and edge cases, and use expert human annotators for ground-truth labels. OpenAI, Evaluation best practices

Ask the policy owner to settle a disagreement

Ask two coordinators to route the same mixed complaint using the written rule. If they disagree, have the support policy owner decide the correct action or explicitly allow either route. In this example, the owner might require the payments team to handle a disputed charge first while the delivery team keeps responsibility for the shipment. Save that decision with the rule version and use it to label similar cases. If the owner has yet to decide, mark the case as requiring staff review. NIST’s generative AI profile calls for clear roles in human oversight and recommends defining acceptable use. NIST AI 600-1, GOVERN 3.2

Recover the work after a stop

Run a stop drill with one ticket awaiting review, one already assigned, and one whose write returned a timeout. Have an operator turn off suggestions, return eligible new tickets to the manual intake queue, and reconcile the three cases against the ticket system. Check the current assignment and its history before retrying a timed-out write. If the receiving team already owns the case, coordinate any reassignment with that team. Record who can stop the pilot and who confirms every open case has a human owner.

NIST recommends establishing criteria for deactivating a generative AI system and describes testing fallback procedures that can include manual processing. For this pilot, one stop condition could be an assignment that staff cannot trace or account for during reconciliation. NIST AI 600-1, GOVERN 6.2 and MANAGE 2.4

Decide what the evidence permits

  • Prepare the workflow. If the team lacks approved examples, capture and label cases in the existing process. If a rule has no owner, assign one and resolve the disputed cases before testing that decision.
  • Run an offline replay. If the approved inputs, labels, and rules support a test but live access or recovery is incomplete, replay historical cases in isolation. Keep results in the test environment and list the missing evidence for a supervised pilot.
  • Propose a supervised pilot. Proceed when every check scores 2 for the scope you plan to run. Compare it with manual routing on the same case mix. Agree in advance how to judge correct queue acceptance, review and rework time, and missed escalations. Name who reviews the results and who can stop the pilot.

Save the selected scope, evidence, and operating limits together so the next person can repeat the decision. Once you have a viable pilot scope, use the build-versus-buy checklist to compare implementation options against it.

Sources

Request an audit for one workflow

Bring one recurring task, the system where it ends, and an exception that your team handles by hand. The audit is free. Any follow-on work you choose is charged separately. Request an AI workflow audit from Insonl