How Do You Test an AI Classification Step Before It Routes Leads?

How do you know an AI classifier is ready to route real leads?

Test it against a labeled set of past leads before it touches live data. Pull real submissions, have a person label each one with the correct category, run the AI step on the same set, and compare. If it agrees with the human labels often enough on the categories that matter most, it is ready for a limited launch.

AI classification steps are everywhere in GTM automation now. A form comes in, a model reads the free-text field, and the workflow decides whether the lead is a fit, which segment it belongs to, or which rep should get it. It feels like magic in a demo. It feels very different when a good lead gets routed to the wrong queue and nobody notices for a week.

The fix is not a better prompt. It is a test before launch, done the same way you would test any decision a person used to make.

What does an AI classification step actually do in lead routing?

It reads unstructured input, such as a message field, a job title, or a company description, and assigns a label from a fixed list. That label then drives a routing rule. The model is replacing a human judgment call, so it should be held to the standard of the person who used to make that call.

Common examples include sorting inbound messages into sales, support, partnership, and spam. Others include tagging a lead's industry from a company description, or flagging whether a job title is a decision maker. Each one is a small classification problem with a fixed set of answers.

The risk is that these steps fail quietly. A model does not throw an error when it picks the wrong label. It returns a valid answer that happens to be wrong, and the automation carries on. That is why testing has to happen before launch, not after a complaint.

How do you build a test set?

Pull a few hundred real past leads, remove anything sensitive you do not need, and have a person who knows the business label each one with the correct category. Include easy cases, hard cases, and junk. The test set should look like what actually arrives, not like a cleaned-up example from a demo.

Size depends on how many categories you have and how rare the important ones are. If enterprise leads are a small fraction of inbound but matter most, make sure the test set contains enough of them to judge performance on that category specifically. A test set that is almost all small leads tells you little about how the model handles the ones you care about.

Labeling is the slow part, and it is worth doing carefully. Have one person label, then have a second person review a sample. Where they disagree, discuss and write down the rule you agreed on. Those rules often become part of the prompt, because if two humans disagree, the model will too.

Store the test set somewhere stable, like a table in Airtable or a sheet, with the input, the human label, and a column for the model's answer. You will reuse it every time the prompt or the model changes.

Which numbers should you look at?

Look at overall agreement with the human labels, then at each category separately, and pay most attention to the costly mistakes. A model that is right most of the time overall can still be wrong on the one category that matters. Per-category results and a simple confusion table show you where the errors land.

A confusion table is just a grid of human label against model label. Each cell counts how often the model gave one answer when the human gave another. It instantly shows patterns, like enterprise leads being labeled as small business, or partnership requests being labeled as sales.

Then weigh the errors by cost. Sending a spam message to sales is annoying. Sending an enterprise lead to the self-serve queue can cost a deal. Decide in advance which errors are acceptable and which are not, and judge the model against that, not against a single overall score.

Write down a pass bar before you run the test. It is too easy to decide after the fact that whatever result you got is good enough.

How do you improve a classifier that fails the test?

Read the errors one by one. Most failures come from a few patterns: categories that overlap, instructions that are vague, or inputs the prompt never anticipated. Tighten the category definitions, add short examples of tricky cases, and give the model an explicit option for uncertain inputs. Then rerun the same test set.

Overlapping categories are the most common problem. If "partnership" and "sales" can both describe the same message, the model will split them unpredictably. Either merge the categories or write a clear rule for which one wins.

An "unsure" option is the most useful single change. Instead of forcing a guess, let the model say it cannot tell, and route those leads to a person. A smaller, more accurate automated share plus a human review queue is better than a confident model that is often wrong.

Keep each version of the prompt and its test results together. I wrote about how to version prompts inside a production automation, and test results are what make versioning useful. Without them, you cannot tell whether a new prompt is better or just different.

How should you launch once the test passes?

Launch in shadow mode first. Let the AI step label live leads without acting on the label, while routing continues the old way. Compare its answers to what actually happened for a week or two. If the live results match the test results, switch routing over, starting with the categories where it performed best.

Shadow mode catches the gap between your test set and real traffic. New kinds of submissions appear, seasonal patterns shift, and a campaign might bring in a type of lead you never labeled. Seeing those in shadow mode costs nothing. Seeing them after launch costs leads.

When you switch over, keep a human in the loop for the costly categories. If enterprise routing is the high-stakes decision, have the AI suggest and a person confirm for the first few weeks.

Test the whole workflow too, not just the AI step. My guide on testing a Make scenario with fake leads covers the plumbing side, which needs its own check.

What should keep running after launch?

A regular spot check. Sample a handful of classified leads each week, have a person label them blind, and compare. That catches drift when inputs change, and it catches silent changes in model behavior. The pre-launch test proves the classifier was ready. The weekly check proves it still is.

I covered the routine in spot-checking an AI classification step each week. The two practices work together: the pre-launch test sets the baseline, and the weekly spot check tells you when you have drifted away from it.

Rerun the full test set whenever you change the prompt, switch models, or add a category. It is the only reliable way to know that a change improved things rather than moving the errors somewhere else.

Which mistakes should you avoid?

Testing on examples you wrote yourself, judging only on overall accuracy, skipping the "unsure" option, and launching straight into live routing. Each of these hides problems until they cost something. The other common mistake is never retesting, so a classifier that worked at launch slowly degrades without anyone noticing.

Self-written examples are the subtle one. When you write test inputs yourself, you write them the way you think about the categories, so the model looks better than it is. Real submissions are messier, shorter, and more ambiguous. Always test on real data.

Another mistake is putting sensitive personal data into prompts or test sets that do not need it. Strip what the classifier does not use. It reduces risk and usually improves results, because the model has less noise to read.

What should you do next?

Pick one AI classification step that routes leads, pull a few hundred past submissions, and have someone label them. Run the model on the same set, look at per-category results and the costly errors, and set a pass bar. Fix the prompt until it passes, then launch in shadow mode before it routes anything live.

Keep the test set. It is the most valuable asset you will build for that automation, because every future change can be checked against it in minutes.

If you want help testing an AI step in your Make, n8n, Zapier, or HubSpot workflows, or building a labeled test set for your lead routing, reach out. Let's chat about making your automations as careful as the people they replaced.