Your AI Agent Was Good Last Month. Why Is It Worse Now?

Your AI agent was good last month. Why is it worse now?

Usually because the inputs moved, not the model. An agent that worked well on the data it was built against will degrade quietly as that data changes shape. The prompt looks the same, the setup looks the same, and the results get worse anyway.

This is the most under discussed problem in production AI work. People plan for the launch and not for month four, and month four is when the thing that was impressive starts producing output somebody has to quietly fix by hand.

The good news is that drift has a small number of causes, and they are distinguishable if you look in the right order. Here is how I work through it.

What does drift actually mean when nothing was changed?

It means the gap between what the agent produces and what you would accept has widened. Something in that loop moved: the inputs, the prompt's fit to those inputs, the surrounding systems, or the model behind the API. Only one of those is outside your control.

Notice that drift is defined against your standard, not against some absolute. If your expectations rose because you got better at the task, the agent did not get worse. That sounds like a technicality and it is not, because the fix is completely different.

So the first question is always whether the output changed or the judgement changed. I have caught myself getting this wrong, deciding an agent had degraded when what had actually happened is that I had learned what good looked like.

Which causes are yours and which belong to the provider?

Input drift, prompt fit and downstream changes are yours. Model behaviour behind an API is the provider's. In my experience the first three explain most cases, and people reach for the fourth first because it requires no self examination, which is the wrong instinct.

Input drift is the common one. The records coming in are longer, or messier, or now include a category that did not exist when you built the thing. The agent is handling a population it was never shown, and doing so about as well as you would expect.

Prompt fit is the second. A prompt encodes assumptions about the input, and those assumptions decay silently. Nobody edits a prompt to make it wrong. It becomes wrong because the world around it moved while the text stayed still.

How do you notice drift before a client does?

Sample the output on a schedule and actually read it. Not the error rate, the output itself. Drift almost never triggers an error, which is exactly why monitoring built around failures misses it completely. A weekly read of ten real outputs will catch it.

I read a small sample of real output every week for anything running in production. It takes a few minutes and it is the only check that reliably catches the slow kind of degradation, because the slow kind looks fine in aggregate and wrong in particular.

This is the same blind spot behind automations that fail silently. A system that reports success while producing nonsense is worse than one that crashes, because nobody investigates a green dashboard.

What does a usable evaluation set look like?

Twenty to fifty real examples with the output you would accept for each, saved somewhere stable. Small enough that you will actually maintain it, real enough that it reflects the inputs you get rather than the inputs you imagined. That is the whole specification.

The temptation is to build something comprehensive. Resist it. A giant evaluation set you never run is worth nothing, and a small one you run every time you touch the prompt is worth a great deal. Coverage matters less than habit here.

Include the awkward cases deliberately: the record with missing fields, the one in the wrong language, the one that is three times longer than normal. Those are where drift shows up first, because those are the cases nearest the edge of what the prompt assumed.

Should you pin the model version you call?

If the API you are calling lets you name an exact model identifier rather than an alias that can move, use the exact one. It converts a whole class of invisible change into a change you make deliberately, which is the entire goal.

The trade is that you stop getting improvements automatically. That is usually the right trade for anything client facing, because a system that gets silently better is also a system that can silently change in ways your downstream steps did not expect.

Whatever you choose, write down which identifier you are calling and where that is configured. The version of this that hurts is the one where output changed, nobody knows what the system is calling, and the person who set it up has moved on. That is the same problem as keeping an automation alive when the tool underneath it changes.

What do you do on the day output quality drops?

Stop the automatic path before you investigate. Route the output to a queue a person reviews rather than letting it keep writing to live systems. You can diagnose calmly once nothing further is being committed on a guess. Nothing you learn is worth more bad records.

Then run your evaluation set. If the saved examples still produce acceptable output, the model and prompt are probably fine and your incoming data has changed. If the saved examples now fail too, something upstream of your inputs moved, and that narrows it a great deal.

That single comparison, saved examples versus today's inputs, separates the two most common causes in about ten minutes. It is the main reason to keep an evaluation set at all, and it is worth more on a bad day than any amount of logging.

When should the agent stop and ask a person?

When it is operating outside the range you tested, and when the cost of being wrong is high. Confidence is not a reliable signal here, so build the escalation around input characteristics you can measure rather than around how sure the output sounds.

Length is a good proxy, oddly. An input much longer or shorter than your normal range is a decent flag that you are outside tested territory. So is a missing field that is usually present, or an unfamiliar value in a category column.

Design the handoff so it is genuinely easy to act on. An escalation that dumps raw context on somebody gets ignored within a fortnight. One that says what it was unsure about and what it would have done gets used, which is the practical side of deciding when to keep a human in the loop.

What should you do next?

Pick the agent you rely on most and save twenty real inputs with the output you would accept. That is an hour of work and it converts drift from something you argue about into something you can check. Do it before you need it.

Then put a recurring reminder in your calendar to read ten real outputs. Not to look at a dashboard, to read the actual text. The habit is what catches the slow degradation, and no tooling substitutes for it.

If you have something in production that feels like it used to be better and you cannot prove it, that is a solvable problem and a common one. Tell me what it does and I will tell you where I would look first. Let's chat.