For AI program leads, transformation offices, and PMOs: everything a pilot needs to arrive at budget review with evidence instead of anecdotes — the metrics, the gates, the instrumentation, and the report. Built to be adopted mid-pilot; it's never too late until the review is on the calendar.
Five numbers decide fund-or-kill. Everything else is engineering telemetry (valuable, but not for this audience — see the evidence problem):
| Decision metric | Definition |
|---|---|
| Task success rate | completed ÷ (completed + failed) |
| Escalation rate | handed to a human ÷ conversations — the honest deflection number |
| Cost per conversation | total inference + tool cost ÷ conversations |
| Cost per completed task | total cost ÷ completed tasks — the headline, comparable across pilots and against the human baseline |
| User sentiment | thumbs, ratings, periodic CSAT — would its users keep it? |
Vanity list — banned from the review pack: conversations held (activity ≠ outcome), tokens processed, "hours saved" estimates without a measured baseline, demo transcripts, and eval scores (they belong in the team retro). Also record the human baseline — what a completed task costs today via the existing process — in week one, while nobody has an incentive to inflate it.
Gates negotiated after results are known aren't gates; they're rationalizations. Publish these at pilot kickoff (or adopt them now, dated), with two review points:
| Gate | Threshold (starting points — tune per pilot class) | Outcome |
|---|---|---|
| Continue gate (after 1 quarter) | Success rate ≥ 40% and trend improving and sentiment not hostile | Miss → kill-or-rethink by default; the owner argues exceptions, not the metrics |
| Scale gate (after 2 quarters) | Success ≥ 60% · escalation ≤ 30% · cost per completed task ≤ human baseline (or a published multiple of it) · sentiment positive | Pass → scale case goes to committee with the numbers attached |
The point isn't the specific thresholds — it's that defaults beat debates. A pilot that misses its gate can still be saved, but the burden of proof flips to the sponsor, where it belongs.
From the agent's backend, emit exactly four event types. This shape is portable across Claude Agent SDK loops, LangChain callbacks, OpenAI Agents hooks, or a hand-rolled loop:
morvero/slack, /web-chat, /vscode).task-completed / task-failed / escalated-to-human per task. Reserved names are what make pilots comparable.Non-negotiable hygiene: pseudonymous user hashes only (never emails — you need distinct-user counts, not identities), and fire-and-forget emission that can never slow or break the agent. Concrete adapters per framework: docs → track AI agent pilots.
One page per pilot, then one ranking table. Per pilot: the five metrics with quarter-over-quarter deltas, a sparkline each, the human baseline alongside cost per completed task, and the owner's narrative in one paragraph. Then the table the meeting is actually about: every pilot, one row each, sorted by cost per completed task, with a signal chip per row — Scale candidate / Iterate / Kill-or-rethink / Insufficient evidence. The chips are reading aids computed from the published gates, not verdicts; the committee decides, but it decides looking at comparable rows instead of competing slide decks.
Bring the ranking to the committee quarterly (monthly early on). Keep the meeting to outcomes and money; leave latency and evals with the engineers.