How to measure AI agent pilots
Every enterprise is running agent pilots — support copilots, HR bots, SQL assistants, coding agents — and almost none of them roll up into a number a steering committee can act on. Demos impress, anecdotes circulate, budgets renew on vibes. This guide is the practical alternative: five metrics, the instrumentation conventions to collect them, and the review cadence that turns pilots into fund-or-kill decisions.
Why pilot measurement fails today
- The engineering layer measures the wrong altitude. Tracing and observability tools (Langfuse, LangSmith, Arize and friends) are excellent at spans, evals, token counts, and latency. A steering committee cannot read a trace. Those tools answer "is the agent working correctly?" — not "is this pilot worth continuing?"
- Every pilot reports differently. One team quotes thumbs-up rate, another quotes deflection, a third quotes tokens saved. Nothing is comparable, so nothing is decidable.
- Cost is invisible. Inference spend is real money, but almost no pilot reports cost per outcome — the only cost number that supports a decision.
The five metrics that decide fund-or-kill
| Metric | Definition | What it tells the committee |
|---|
| Task success rate | completed ÷ (completed + failed) | Does the agent actually finish the job it was given? |
| Escalation rate | escalations to a human ÷ conversations | How often the agent gives up — the honest deflection number |
| Cost per conversation | total inference/tool cost ÷ conversations | Unit economics of running the pilot at all |
| Cost per completed task | total cost ÷ completed tasks | The headline number: what a successful outcome actually costs — comparable across pilots and against the human baseline |
| User sentiment | thumbs up/down, ratings, periodic CSAT/NPS | Whether the people it serves would keep it |
Rank every pilot by cost per completed task next to its success rate and sentiment, and the fund/fix/retire conversation mostly runs itself: fund the leaders, fix the frustrating-but-used, retire the unused and the uneconomic.
Instrumentation conventions (portable to any stack)
You need exactly four event types from the agent's backend, regardless of framework:
- Conversation started — one event per session, tagged with the entry surface (
/slack, /web-chat, /vscode).
- Capability used — one event per tool call or skill invocation, carrying its cost in USD (tokens, API fees). Name capabilities with stable, kebab-case identifiers.
- Outcome — exactly one of
task-completed, task-failed, or escalated-to-human per task. Reserved, unambiguous names are what make pilots comparable.
- User verdict — thumbs or a rating at conversation end; a periodic 0–10 CSAT for regulars.
Two rules that keep the data honest and the lawyers calm:
- Pseudonymous users only. Track a salted hash of the employee id — never emails or names. You need distinct-user counts and repeat-usage, not identities.
- Fire-and-forget telemetry. Measurement must never break or slow the agent. Every emit is asynchronous and failure-tolerant.
Hook these four events wherever your framework exposes "run started / tool called / run ended" — Claude Agent SDK loops, LangChain callbacks, OpenAI Agents run hooks all fit the shape. Concrete adapters: docs → Track AI agent pilots.
Run the review like a portfolio, not a science fair
- Quarterly (or monthly early on), one page per pilot: the five metrics, their trend, and the owner's one-paragraph narrative. Nothing else.
- Side-by-side ranking: all pilots on one table, sorted by cost per completed task. The ranking — not any single number — is the decision instrument.
- Pre-agree the thresholds: e.g. "a pilot below 40% success or above 3× the human cost baseline after two quarters is retired by default." Defaults beat debates.
- Keep engineering metrics out of the room. Latency and eval scores belong in the tracing tool and the team retro; the committee sees outcomes and money.
Where Morvero fits: Morvero treats
each agent pilot as a product. Three REST calls from the pilot's backend produce the scorecard automatically — task success rate, escalation rate, cost per conversation, cost per completed task — plus thumbs feedback flowing into the same triage and AI-backlog loop as your web apps, and a steering-committee report that ranks every pilot side by side. It sits
above your tracing tool, not instead of it. And since your coding agent can wire the telemetry itself via
Morvero's MCP server (and verify data is arriving with
verify_install), instrumenting a pilot is an afternoon, not a sprint.
What this approach does not cover
- Deep tracing, evals, prompt debugging — use Langfuse, LangSmith, Arize, or your platform's tooling. The scorecard is the business-review layer above them, and it deliberately stays there.
- Model quality benchmarking — offline evals are an engineering practice, not a governance metric.
- Consumer-facing chatbot marketing analytics — this guide (and Morvero) targets internal and workforce-facing pilots.
morvero