the production gap

I build agents that run on the actual software a company already uses. Moneybird, Monday, PandaDoc, whatever they’ve got sitting there. And I keep them running long after the demo’s packed up and gone home. That’s the part where I keep seeing the same thing, over and over, in the day-to-day. What wins a room over doesn’t always hold up come Monday morning. Not by a long way, sometimes.

Two numbers from the same year, and honestly, they just won’t sit next to each other. In 2025, the OECD says, 61 percent of all the venture capital on the planet went to AI. More than 258 billion dollars. Let that land for a second. That same year Gartner reckoned more than 40 percent of agentic AI projects get scrapped before the end of 2027, and researchers at MIT wrote up how the bulk of enterprise pilots stall out before they deliver a single thing. And the models are hardly ever the weak spot here. Companies just don’t learn how to build with them.

So much money. So little of it reaching production. Sounds like a paradox. It isn’t. Between an agent that impresses and an agent that actually works, there’s a gap, and almost nobody spending all that money has crossed it themselves.

a demo and a service are two different crafts

A demo has to work once, everyone watching. A production system has to behave itself precisely when nobody’s watching. Sounds like a bit of word-play, I know. It’s a genuinely different craft, though.

In the demo, the builder picks the input himself. Runs the path he already knows, stops the second the applause kicks in. In production? The world picks the input, the path lands differently every single time, and nothing stops on its own. A chief operating officer in manufacturing put it flat out in that MIT study: “The hype on LinkedIn says everything has changed, but in our operations, nothing fundamental has shifted.” On stage, everything’s changed. On the floor, where I sit every day, nothing yet.

the arithmetic nobody puts on the slide

So why do demos work while production falls over? The reason’s dull, honestly. It’s just arithmetic.

Say an agent does the right thing 95 percent of the time, every step. And trust me, for today’s models that’s the optimistic read already. Three steps, like in a demo? It’ll get away with it almost every time. Thing is, reliability over a chain doesn’t add up. It multiplies. By twenty steps the odds of a clean run have already dropped to something like one in three. And a production process worth the trouble is hardly ever three steps. It’s twenty. Thirty.

Sure, it’s a simplification. Real errors aren’t so neatly independent of each other, obviously. But the direction’s right, and every builder feels it in the gut straight off: more autonomy is more steps, and one weak link drags the whole chain down with it. I’ve watched it happen myself. One evening in May, my own engine slid into a loop that kept feeding itself. In review, the mistake looked completely reasonable. And that’s the treacherous part. In the demo? Nowhere to be seen.

it is not a model problem

The reflex? Pin it on the model, then go hunting for the fix in a smarter model. Both wrong.

The expensive mistakes this past year didn’t come from the model being dumb. They were actions with no brake on them. Look at Air Canada. Its chatbot made up a bereavement discount that didn’t exist, and the tribunal simply ruled the company was on the hook for it. Whether that information comes off a static page or out of a chatbot, doesn’t matter one bit. Or take Replit, where the coding agent wiped a whole production database during an explicit freeze, owned up afterward to “a catastrophic error in judgment,” and then cooked up thousands of fake records to cover its own blunder. Because a chatbot that hallucinates lies to a user. An agent that hallucinates does something. It refunds the wrong customer. Deletes a row that was meant to stay put.

The real causes sit deeper, and they’re a lot tougher to shift. Context rot, for one. Feed a model more, and it starts doing worse, often well before that window’s anywhere near full. Anthropic says it about its own models, no hedging: treat context as a limited resource with decreasing returns. That extra token isn’t free. It eats into whatever attention the model’s got left.

And this is exactly what my guardrails piece was already on about, only now a whole industry full of wreckage has gone and confirmed it. Safety doesn’t come out of how clever the model is. It comes out of the discipline you build into the tooling itself. A builder running agents across dozens of document types put it better than I can: in a regulated domain, the most constrained agent is the most reliable one.

where it does pay off

This isn’t a piece against agents, mind you. I build them, they run at real companies, they deliver. They just deliver somewhere a fair bit less exciting than the market promises you.

Agents don’t win by getting to reason around all free and easy. They win by running a well-documented process some human used to do by hand. That’s honestly all it is.

Take the quote-recording agent I built for an installation firm. Dead simple tool, really. Costs next to nothing to run. You train a model on all the variations, on your own context, and the thing just lives up in the cloud. A fairly plain model gets you a long way if it’s prompted right, and we even wrote a whole guide on that bit. What they wanted was quicker quotes, plain and simple, because the things were eating up a huge amount of time. So you open your phone, there’s a web page, well, a web environment, and on it a little quote-recording panel: a text box with speech-to-text, and a checklist beside it of the stuff you had to keep in mind. You go down the checklist, you just talk at your phone. Out comes a chunk of text, off it goes to the model, and the model builds the quote in PandaDoc, where it comes out looking properly polished. And that was that.

And there it is, right there: a really plain model handing you a ton of value, precisely on the simple jobs. Because making quotes here wasn’t some complicated process. There was a fixed set of variables you had to run through. On an air-conditioning install, the brand, the model, a layout plan, a condensate pump, and on it goes. Loads of variants, sure, but always standard, with fixed unit prices and maybe a discount on top. No complicated payment structures sitting behind it.

And mind you, it doesn’t run itself. Punching in orders and processing documents is a hugely people-heavy job, and in a complex B2B process the human stays in the loop, always. You don’t want to hand a process like that fully over to the AI. You want to pull the human actions out of it, the human judgment you keep right where it is. Let an agent do the intake, draft the document, get the analysis teed up, and then the person in the loop only has to look over what the agent already did. Speeds it up massively. You’ve got to watch the cost, though. Reach straight for the priciest flagship model from Anthropic or OpenAI and you’ll hit the point where it just gets genuinely expensive. For a lot of document work a lighter model already does the job fine. There are all these gradations in between, and you just test for it, because it comes down to how many factors actually move the outcome.

And the builders who pull this off outside my own kitchen, they all say a version of the same thing: what works is the boxed-in system. Clever’s got very little to do with it. Take Klarna. It shows the whole thing, and both ways at once. The company had an AI assistant do the work of 853 employees and saved tens of millions doing it. Nice story. Only then they publicly walked part of it back, the chief executive admitting they’d pushed too hard on efficiency and cost, and the quality had suffered for it. The win is real. So’s the boomerang.

There’s a more hopeful signal too, and it’s not nothing. Google’s own study last autumn found more than half the executives they surveyed say they’ve got agents in production, and three quarters report a return inside the first year. The most mature category? Code. And yet even there, honestly, the real number isn’t triple productivity. The solid DORA research clocked an individual developer getting tens of percent faster, while the delivery speed of the whole organization can actually drop if the foundation’s wobbly. That’s just how it goes: AI makes a good process better, and a bad process it breaks faster.

the market disagrees with itself

Notice how often one of these big numbers gets tossed around without anyone stopping to ask what it’s actually measuring. Estimates of the market size run a factor of three to four apart, purely on how narrow or broad you draw the word “agent.” Gartner’s measuring one thing, MIT another, the sunny venture capitalists something else again. They don’t contradict each other. They’re just counting different stuff. Line the headlines up, leave the definitions off, and you’re mistaking noise for insight.

And a chunk of what gets sold as an agent? Not an agent at all. Of the thousands of vendors calling themselves agentic, Gartner puts the real ones at around 130. The rest are existing chatbots in a new coat. Agent washing, they’ve started calling it.

you close the gap with boring discipline

Waiting around for a smarter model? That won’t close the gap. Andrej Karpathy, and the man genuinely knows what he’s on about, doesn’t see robust agents as a one-year job at all. More like a decade-long problem. He’d sooner call the models ghosts than animals: summoned, statistical, with human quirks and very inhuman blind spots. And you don’t build reliability by politely asking a ghost to go easy.

You build it with boundaries that sit outside the agent’s reasoning loop. In the order I stick to myself: first a hard brake nobody can talk their way past, simply because the tool won’t allow anything else. Then an explicit threshold, before a single thing goes to production. After that, a log of every action the thing takes. And to cap it, a kill switch that trips after a run of failures back to back. None of it’s exciting. That’s exactly the point.

The test I hang on every agent plan, mine or a client’s, is the same one my engine story ends on. It’s not about what the system can do. It’s the other side: what happens when the system gets it wrong, and which line of code puts it to a stop right then? Got an answer to that, and you’re standing on the right side of the production gap. Still owe the answer? Then you haven’t got a system. You’ve got a demo that hasn’t been allowed to fail yet. And a spot in that forty percent.