What Should You Log When an AI Step Makes a Routing Decision?

What do I need to record when AI decides where a lead goes?

Record the input the model saw, the decision it made, the reason it gave, its confidence, the prompt and model version, and what the automation did next. With those six things, anyone can answer "why did this lead go here?" months later. Without them, an AI routing step is a black box nobody can defend or fix.

More teams now put a language model in the middle of lead routing. A form comes in, an AI step reads the message and company details, and it decides whether the lead goes to sales, to nurture, or to a partner. It works well until a rep asks why a great lead landed in the wrong queue.

At that moment, a normal automation log is not enough. It will show the run happened and which branch fired. It will not show what the model was thinking, which is exactly what everyone wants to know.

Why is a regular automation log not enough for AI decisions?

A regular log records that a step ran and what it returned. That works for fixed rules, because the rule itself explains the result. An AI step is different. The same rule, meaning the same prompt, can produce different results for similar inputs, so you need the reasoning and context, not just the outcome.

With a fixed rule like "company size over a threshold goes to enterprise," you can rebuild any decision from the data. With an AI step, the decision depends on the wording of the prompt, the model behind it, and how the model read that particular message. Change any of those and the result might change.

That is why I treat AI routing decisions like small judgments that need a paper trail. The goal is not to distrust the model. It is to make its choices reviewable, so you can tell a good decision from a lucky one and a bad decision from a bad prompt.

What should the input record contain?

Store exactly what the model received: the cleaned form fields, any enrichment data added before the AI step, and the final text sent to the model. Store it as it was at decision time, not as it looks in the CRM today. Records change after the fact, and you need the snapshot the model actually judged.

This matters more than people expect. A week later, a rep may have edited the company size or added notes. If you rebuild the decision from today's CRM record, the model's choice can look strange when it was perfectly sensible given what it saw. The snapshot protects both the model and the person reviewing it.

Be careful with personal data. Log what you need to explain the decision and no more, and follow your privacy obligations for how long you keep it. A decision log is a system of record, so it deserves the same care as your CRM.

How should I capture the decision and the reason?

Ask the model to return a structured answer with separate fields for the decision, a short reason, and a confidence level, then log each field on its own. A free-text answer you have to parse later is hard to search and easy to misread. Separate fields make every decision easy to filter and count.

In practice, I ask for something like a route value from a fixed list, a one or two sentence reason in plain language, and a confidence label such as high, medium, or low. The fixed list matters. If the model can invent a new route name, your automation has nowhere to send it.

The reason field is the one people skip and later regret. It is what lets a sales manager read a decision and immediately see whether the logic holds up. "Routed to enterprise because the message mentions a procurement process and a large team" is something a human can agree or disagree with in seconds.

Why do the prompt and model version need to be logged?

Because they are part of the decision. If you change the prompt or switch models, past decisions were made under different rules. Logging a prompt version and the model name with every decision lets you compare before and after a change and see whether the change helped or hurt routing quality.

I give every prompt a simple version label and bump it every time the wording changes, even slightly. That label goes into the log with each decision. When routing quality shifts, the first question is whether the prompt or model changed around that date, and the log answers it immediately.

This ties into how you manage prompts in general. Treat them like code: versioned, reviewed, and changed on purpose. I wrote about the setup in how to version prompts inside a production automation.

What should happen with low-confidence decisions?

Log them like any other decision, then send them to a person instead of acting automatically. The confidence field makes this possible. A low-confidence routing choice goes to a review queue with the model's reason attached, and the human's final decision gets logged next to the model's suggestion for later comparison.

That side-by-side record becomes your best training material. Every time a person overrides the model, you learn something about where the prompt is weak. After a few weeks of overrides, patterns usually appear, and those patterns tell you exactly what to change in the prompt.

The review queue also keeps sales trust intact while the system is new. Reps see that uncertain leads get human attention instead of disappearing into a wrong queue. If you are building that queue, my notes on routing low-confidence AI classifications to a human cover the mechanics.

Where should the decision log live?

Put it somewhere searchable that people outside the automation tool can read, such as an Airtable base, a Google Sheet for low volume, or a database table for high volume. Then write the decision and reason back to the CRM record as properties, so a rep can see why a lead landed with them without opening another tool.

The split is deliberate. The CRM gets the short version: route, reason, confidence, and a link to the full log entry. The log holds the full snapshot, prompt version, and model details. Reps get context in their daily view, and whoever maintains the automation gets everything needed to investigate.

Zapier, Make, and n8n all keep their own run histories, and those are useful for debugging. But run histories are built for the person maintaining the automation, not for a sales manager asking a business question. A dedicated decision log answers business questions without making anyone learn the automation tool.

How do I review the log so it actually improves routing?

Review a sample each week. Read a handful of high-confidence decisions to confirm they are right, every override from the review queue, and anything sales has flagged. Note patterns, change the prompt on purpose, bump the version, and check the next week whether the pattern improved.

Keep the review small enough to actually happen. Fifteen minutes with the right sample beats an hour-long review that gets skipped. The aim is a steady loop of small corrections, not a big audit every quarter.

The same habits apply to automation logs in general, and I covered the broader version in what to log when an automation runs. AI decisions simply add a few fields on top, because the step doing the work can explain itself.

What should you do next?

If you have an AI step in your routing today, add structured output with a fixed route list, a reason, and a confidence label. Log the input snapshot, prompt version, and model name with every decision. Send low-confidence results to a person, write the short version back to the CRM, and review a sample weekly.

None of this slows the automation down. It just makes every decision explainable, which is what keeps sales trusting the system long enough for it to get good.

If you want help building AI routing with a decision log your sales team can actually read, reach out at pravinkumar.co. I build websites and the automations behind them for lead generation, and making them explainable is part of the job.