AI Pilot Exit Criteria: What a Pilot Must Prove

A pilot is ready for production when its output reconciles to the system of record on real volume, its exception queue fits the team that has to clear it, there is a tested way back, and a named person has accepted it — not when it reaches an accuracy number. The accuracy figure is an input to the second gate. On its own it decides nothing, which is inconvenient, because it is the only one of the five that is easy to put on a slide.

The textbook version of this is the stage gate: run a pilot, measure performance against a threshold, promote it if it clears. That works when the thing being promoted is a model in a benchmark. It does not survive contact with a process that has a deadline, a control environment and a team of people who will absorb whatever the system does not.

I have run go-live gates on enterprise finance systems for eighteen years. The questions below are not AI questions — they are the ones a cutover decision has always asked, pointed at a component whose output varies. That is the whole trick, and it is why I would not write a new framework for this.

Why the accuracy threshold is the wrong gate

Ask what "95% accurate, therefore ready" means operationally and it dissolves.

Five per cent of what? On a workflow processing four hundred items a day, that is twenty exceptions — a person's morning. On one processing forty thousand, it is two thousand, which is a department you do not have. The same percentage is a rounding error in one process and a hiring plan in another, so the percentage is not the criterion. The criterion is the absolute exception volume measured against the capacity of the team that clears it, and that is a number you can only get by multiplying through.

There is a second problem with the threshold framing. Accuracy measured on the pilot's sample is a claim about the sample. If the pilot ran on three clean weeks, the figure describes the easy part of the distribution, and the interesting cases have not happened yet. This is the same reason a vendor's accuracy number needs its denominator before it means anything.

"

The same accuracy figure is a rounding error in one process and a hiring plan in another. The percentage is not the criterion; the queue is.

The five gates

Each one is a question with a factual answer, deliberately. A gate that can be argued is not a gate.

1. Does it reconcile? The output has to agree with the system of record — not approximately, and not after somebody fixed the differences by hand. Run it in parallel with the existing process on the same real volume and compare the results item by item, which is what a parallel run has always been for. Differences are the finding. A pilot that produces plausible output nobody reconciled has proved that it produces plausible output.

2. Does the exception queue fit? Take the measured exception rate, multiply by real daily volume, multiply by the average minutes to clear one, and compare that to the hours the receiving team actually has. Do it at peak, not at average — a close-period spike is where this fails. If the answer is that the queue needs 1.4 people and you have 0.3, the workflow is not ready regardless of how good the model is. Most of the design work here is in the exception path, and this gate is where you find out whether that design was optimistic.

3. Is there a way back, and has it been used? What reverses a wrong output, who can trigger it, how long does it take, and has anyone actually done it during the pilot? A rollback that exists on a slide is not a rollback. This is a straightforward import from cutover practice, where a fallback nobody has rehearsed is treated as absent.

4. Is there an owner and a support model? A named person who owns the output, and a defined route for what happens when it misbehaves at an inconvenient hour — who is called, what they can do, and what the process falls back to meanwhile. Operational readiness has always meant this, and an AI step does not get an exemption because it is new. Ownership by "the AI team" is the same as no owner, which the pilots that never became operating systems demonstrate repeatedly.

5. Can a stranger reproduce one result? Pick a single output from the pilot and ask somebody who was not involved to reconstruct how it was reached, from what the system recorded. If that requires a person who was there, the audit trail is a memory aid rather than evidence — and the difference matters the first time an outcome is challenged. What has to be captured to make this true is the subject of observability, and it is far cheaper to add before go-live than after.

The gates that get skipped

In practice the same two go missing, and for the same reason: they are the ones that produce bad news.

The parallel run on peak volume. Running in parallel is expensive — the work gets done twice, by people who are already busy, during the period when they are busiest. It is also the only gate that produces a real exception rate rather than an estimated one. Programmes that skip it discover the true number in week two of production, when the fallback has already been decommissioned.

The rehearsed rollback. Nobody wants to spend a day of a pilot deliberately breaking something and undoing it. And the first time you find out whether reversal works should not be the day you need it. In treasury the equivalent lesson is old and expensive: a reversal path that has never been exercised is a hypothesis.

There is a third, subtler omission. Pilots are usually evaluated on quality of output and almost never on whether the reviewers could keep up. If the design assumes a human checks each proposal, measure how long a check takes and whether attention holds over an afternoon of them. An approval step that is physically possible but practically skimmed has not added a control; it has added a signature.

A go/no-go you can actually hold

The decision meeting works when the answers were collected beforehand and the meeting only reads them.

Not one of those rows is about the model. That is not an accident: the model is the part most likely to improve on its own, and the four organisational rows are the ones that will still be unresolved in six months if nobody forces them now. The same asymmetry drives where to start — the expensive decisions are the ones the technology does not make for you.

A no-go is also a valid outcome, and worth saying because in practice it rarely is one. If the queue does not fit, the honest options are to narrow the scope until it does, fund the queue, or stop. Promoting anyway means the receiving team absorbs the gap quietly, and it will not appear in any report until something is late.

What I would decide

Write the five gates down before the pilot starts, because criteria authored afterwards are written by the result. Derive the accuracy requirement from the queue arithmetic rather than picking a number. Insist on a parallel run that crosses a close. Rehearse the reversal once, on purpose. Put the accepting signature with the person who owns the output afterwards, and give them the right to say no.

And treat a passed pilot as the beginning of hypercare, not the end of the project — the period where the real exception mix arrives is the one after go-live, and it is where the ownership decision either holds or is revealed to have been nominal. My cutover and go-live checklist covers the general version of that transition; the five gates above are what I would add when the component in the middle is probabilistic.

See also why most AI pilots never become operating systems and how to measure ROI from an AI workflow.