Why AI pilots fail, and how to reach production

Most AI pilots fail because they prove the wrong thing. A pilot demonstrates that a model can do the task — on demo data, with one enthusiastic user, nothing external, nobody watching. Production demands identity, permissions, approval routing, logging, exception handling and evals, and none of that was ever in the pilot's scope. The gap between the two is an unpaid integration tax, and it comes due whether you budgeted for it or not.

What do the AI pilot failure statistics actually say?

Two numbers dominate every conversation about failed AI pilots. MIT's NANDA initiative report (2025) found that 95% of generative-AI pilots showed no measurable P&L impact. Gartner (June 2025) predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027 — and estimates that of the thousands of vendors claiming agentic capability, only about 130 actually have it, a practice it calls agent washing.

Both numbers deserve a moment of honesty before anyone builds a slide on them. The MIT figure's methodology has been questioned, and it measured something deliberately broad: whether generative-AI pilots of any description produced measurable P&L impact. It is not a controlled study of well-run agent deployments. Operators keep quoting it anyway, because its shape matches what they see from the inside: a striking demo in week two, applause in week five, silence by month six. The Gartner number is a forecast rather than a measurement, and the agent-washing estimate carries its own lesson — some proportion of "failed pilots" never contained a real agent in the first place. If you are mid-procurement, the buyer's guide to agentic AI covers how to tell the difference.

As of July 2026, the Gartner cancellation prediction is enjoying a second wind — Forbes re-ran it on 7 July 2026 — and nothing in the intervening year has blunted it.

So take the statistics as directionally honest and move past them quickly, because the interesting question is not whether the true failure rate is 95% or 40% or something kinder. It is what, specifically, fails. And it is almost never the model.

What does an AI pilot conveniently skip?

A pilot is designed to succeed. Nobody does this cynically; it is simply what "pilot" means — strip away every source of friction that is not the model, and check whether the model can do the task. Write down the conditions a typical pilot actually ran under and the pattern is hard to miss:

• •Demo data, or a hand-curated export. The agent read a clean spreadsheet someone prepared, not the live system with its duplicates, half-migrated records and permission quirks.
• •One enthusiastic user. A champion who forgave the rough edges, re-prompted around failures and quietly did the awkward parts by hand.
• •Nothing external. No email actually left the building. No supplier, tenant or customer was genuinely contacted. Consequence was simulated.
• •No identity or auth model. The agent ran on the champion's login, or on a broad-scope API key that security never reviewed.
• •No approval routing. The champion was judge and jury; every action was implicitly approved because they were watching it happen.
• •No logging or audit. Nobody asked what the agent did last Tuesday, because the answer was always "whatever the champion watched it do".
• •No exception handling. When the agent hit something odd, a human shrugged and dealt with it. The shrug never appeared in the write-up.
• •No evals. The demo was the eval.

None of these omissions is dishonest. Collectively, though, they mean the pilot proved the cheap, demonstrable core of the problem and deferred the expensive remainder. The champion's attention was load-bearing infrastructure, and attention does not scale. There is a quieter omission underneath all of it: the pilot treated the AI as a tool one person drives, when production means handing the workflow to something that owns it — a different operating model, judged by a different bar.

What is the integration tax, itemised?

The integration tax is the gap between "the model can do the task" and "the organisation can depend on the system". It is every item the pilot skipped, priced honestly. It is payable whether or not you planned for it; the only choice you get is whether you pay it deliberately before go-live or discover it line by line after the pilot stalls.

Read the MIT statistic from the other side and it stops being a horror story: the roughly 5% of pilots that did show measurable impact exist too. Our read, from building this for a living, is that those are overwhelmingly the teams that paid the tax up front — identity, access, approvals and evals were scoped into the pilot itself, so "go live" was a decision rather than a second project.

This is also, plainly, what the Usermode platform is: the integration tax paid once, as a product. Identity, permissioning, approval routing, signed external sends, the audit ledger, the eval harness and the scheduling layer are standing infrastructure, and every new workflow inherits them. That — not heroics — is why a first workflow can be live in 14 days: the fortnight is spent on your workflow, not on rebuilding the plumbing every deployment needs.

Why is governance a feature of production, not a blocker?

Mid-pilot, governance feels like the queue you are stuck in: the security review, the DPO's questions, the approvals workshop nobody schedules. The reframe that unsticks teams is simple. Governance is not what stands between a pilot and production. Governance is most of what "production" means.

Your security team declines to sign off the pilot as-is, not out of obstinacy but because the pilot has no answers to the questions production asks. What did the agent do last Tuesday — show me. Who authorised the email it sent, and could that authorisation be replayed against a different recipient? What happens when a hostile inbound message tries to prompt-inject it? A pilot was never asked these questions. A production system is the set of answers.

Governance is not the tollgate in front of production — it is what production is made of.

Concretely, the governance stack that lets an agent act unattended looks like this. Human approval gates sit on spend and sensitive actions, so autonomy stops exactly where consequence starts. Every external send carries a signed (HMAC-SHA256), time-limited, recipient-bound authorisation, so an approval for one message to one recipient cannot be replayed as anything else. Read-only roles fail closed at the tool layer — enforced in code, not requested in a prompt. Every action lands in a tamper-evident, append-only audit ledger. Untrusted inbound mail is sandboxed against prompt injection. And a run cannot end without either a logged outbound action or an explicit escalation, so silence is treated as a failure rather than a neutral outcome.

Notice what each mechanism buys: permission, not restriction. The approval gate is what makes it safe for a director to say yes to autonomy. The audit ledger is what turns "the AI did something odd" from a crisis into a ticket. Teams that treat governance as a blocker stall in review; teams that treat it as the product ship through it.

Where do AI pilots quietly die on data access?

Data access is where AI pilots die quietly rather than loudly. Nothing fails in a way that produces a post-mortem; the project just stops moving. The pilot ran on an export the champion pulled one Friday afternoon, so the question "how does the agent read the live system?" was never in scope — and when it is finally asked, every answer is slow.

Whose credentials does the agent use? The champion's login works until they change their password, leave, or security notices an interactive account making API calls at three in the morning. A shared service account with broad scopes sails through week one and then fails the access review — correctly. The defensible answer, the agent as its own identity with least-privilege, auditable scopes per system, is precisely the piece nobody scoped, and standing it up touches identity teams, system owners and, in older estates, licensing.

Then there is the shape of the access itself. Exports and screen-scraping are how pilots read data; production needs governed, typed connections into the systems of record, which is exactly what MCP (Model Context Protocol) integrations exist to provide. Building those connectors against a real estate — Microsoft 365, the finance package, the property system, the document library with its permission inheritance — is unglamorous work, and no demo budget ever includes it.

The failure mode here is a stall, not a refusal. The access request sits in a queue. The integration estimate comes back in quarters, not weeks. The champion gets pulled onto something urgent. Six months later the pilot is "deprioritised" — nobody cancelled it; it starved. That is what the inside of a cancellation statistic looks like: projects rarely die at the demo. They die in the gap between the demo and the systems of record.

What does a production-readiness checklist look like?

A production-readiness checklist for an AI agent is short to read and expensive to fake. Before a workflow goes live, it should pass every line:

• •It runs on live systems under its own identity. Its own credentials and least-privilege scopes — not a borrowed login, not an export refreshed by hand.
• •Every external action is gated and logged. Approvals sit exactly where consequence sits, each outbound send is signed and recipient-bound, and the audit ledger would survive a sceptical reviewer.
• •A human knows within minutes when it is blocked. Escalation is a designed path with a named owner and a channel, not an inbox nobody watches. Silent stalling is treated as failure.
• •Evals are green in CI. Behaviour is proven against realistic cases before go-live and re-proven on every change. The demo is never the eval.
• •Someone owns it like they would own a new hire. A named human is accountable for its output, reviews its work closely in the early weeks and can answer for it in a management meeting.
• •The workflow has a number attached. A baseline was taken before go-live and a measure agreed, so "did it work?" has an answer — measuring AI agent ROI covers how to set that up honestly.

A pilot that ticks all six is not a pilot any more; it is production with a modest scope. The list also doubles as a diagnostic for a stalled one: whichever line you cannot tick is where your integration tax is still outstanding.

It is equally the sequence behind how we work. A fixed-price AI Readiness Audit comes first — £2,500, credited in full against any engagement signed within 30 days — and itemises exactly this bill against your estate before anyone writes integration code. Then a first workflow goes live in 14 days on a platform where the tax is already paid. It is the same stack our named agents run on today inside two real businesses — Legacie in property, WH Scott Group in industrial lifting and inspection — under the same identities, gates and ledgers described above.

The statistics will keep circulating, and most pilots will keep proving the wrong thing. If you would rather see the paid-up version — a governed agent running a real workflow on live systems — you can book a demo and we will walk through it against your own estate.