← Blog

Your AI pilots don't have a model problem. They have an evidence problem.

Enterprises are running dozens of agent pilots, and most of them will die this year. Not because the model hallucinated, not because latency was bad — because when the budget review came, nobody could prove the pilot was used.

July 21, 2026 · 5 min read

The pilot graveyard has a pattern

Walk through any large enterprise and count the agent pilots: a support copilot, an HR bot, a SQL assistant, two coding agents, something in procurement nobody fully remembers approving. Each one launched with a demo that impressed a steering committee. Twelve months later, most are gone.

The post-mortems rarely say "the model wasn't good enough." They say things like "unclear value," "couldn't quantify impact," "sponsor moved on." Translate those, and they all mean the same thing: when someone finally asked "is anyone actually using this, and what does a successful outcome cost?", the room went quiet.

Demos start pilots. Evidence keeps them alive.

A demo answers "can this work?" — and modern models answer that question so convincingly that pilots get approved in a single meeting. But the question that decides renewal is different: "did this work, for whom, how often, and at what cost per outcome?" That question can't be answered with a demo, an anecdote, or a screenshot of a great conversation. It needs boring, continuous, comparable numbers:

• Task success rate — completed ÷ (completed + failed). Does the agent finish the job?
• Escalation rate — how often it hands off to a human. The honest deflection number.
• Cost per conversation — the unit economics of running it at all.
• Cost per completed task — the headline: what a successful outcome costs, comparable across pilots and against the human baseline.
• User sentiment — would the people it serves keep it?

Notice what's not on the list: eval scores, token counts, latency percentiles. Those live in your tracing tool and matter enormously — to engineers. A steering committee cannot read a trace, and it shouldn't have to. (We wrote a full guide on this split: how to measure AI agent pilots.)

Why nobody has these numbers

Three reasons, all fixable:

1. Every pilot reports differently. One team quotes thumbs-up rate, another quotes "deflection," a third quotes tokens saved. Nothing is comparable, so nothing is decidable — and the committee defaults to whoever tells the best story.
2. Cost is invisible. Inference spend is real money, but it lands in a cloud bill, not next to outcomes. Cost per completed task is the only cost number that supports a fund-or-kill decision, and almost no pilot reports it.
3. Measurement was somebody's Q3 promise. Instrumenting telemetry felt like a sprint of work, so it slipped. Then the review arrived first.

The fix is smaller than you think

You don't need an analytics team per pilot. You need four events from the agent's backend — conversation started, capability used (with its cost in USD), an outcome (task-completed, task-failed, or escalated-to-human), and a user verdict. Wire those into any measurement layer and every pilot in the company becomes comparable on one table, sorted by cost per completed task. The fund/fix/retire conversation mostly runs itself from there.

The uncomfortable corollary

If a pilot can't survive honest adoption numbers, it shouldn't survive the budget review — and that's the system working. The point of evidence isn't to save every pilot. It's to make sure the ones that die deserve it, and the ones that scale can prove they do. In 2026, "we believe it's helping" is not a line item. Bring a number or bring your farewell slide.