Should You Trust AI Visibility Scores From Tools?

Can you trust the AI visibility score your tool reports?

Trust it as a trend, not as a ranking. AI answers change from run to run, so a single position or a single snapshot means little. A visibility score is useful only when it is built from many prompts, each run several times, across named engines. Without that, it is a number dressed up as data.

AI visibility tools have become a standard line in B2B marketing budgets. They promise to tell you how often ChatGPT, Gemini, Perplexity, or Google's AI features mention your brand. Founders forward the dashboards to their boards. Marketing teams set targets against them.

I work on AEO and GEO for B2B sites, and I use these numbers too. But I have learned to read them carefully. The research on how AI tools recommend brands should make every marketer cautious about what a single score can claim.

Why are AI answers so hard to measure?

AI answers are hard to measure because they are not fixed. The same prompt can return different brands, in a different order, with a different number of items each time. A score built on one run per prompt captures a moment, not a pattern. That is the core problem every tracking tool has to solve.

SparkToro and Gumshoe.ai tested this directly in research published in January 2026. Six hundred volunteers ran 12 prompts through three AI tools a combined 2,961 times. SparkToro reported a less than 1 in 100 chance that ChatGPT or Google's AI would give the same list of brands if asked 100 times.

Order was even less stable. SparkToro said it was more like 1 in 1,000 runs before two lists appeared in the same order. If your tool tells you that you rank third for a prompt, that research suggests the number may not survive the next run.

What does a trustworthy visibility score measure?

A trustworthy score measures how often your brand appears across many runs of many prompts, not where it ranks in one answer. SparkToro's research concluded that visibility percentage across dozens to hundreds of prompts, run multiple times, is a reasonable metric. Position within a list is the part to ignore.

Think of it like polling. One phone call tells you nothing about an election. A thousand calls, sampled sensibly, tell you something. The AI version is the same: appearance rate over enough samples starts to reflect which brands a model tends to associate with a buying question.

So the first thing I check in any tool is whether it reports appearance rate or rank. If the headline number is an average position, I treat it as noise. If it is a share of runs where the brand appeared, with the sample size shown, it is worth tracking.

What should you ask your tool vendor before trusting the score?

Ask five questions. Which prompts are in the set, and who chose them? How many times is each prompt run? Which engines and modes are queried? Are runs logged in or anonymous? Can you export the raw answers? If a vendor cannot answer these clearly, you cannot know what the score represents.

The prompt set matters most. A score built on prompts that already contain your brand name will look great and mean nothing. A score built on real buyer questions, the kind your sales team hears on calls, tells you something about consideration. Ask to see the list and edit it.

Raw answer export is the second test. Whatever tool you use, whether Peec AI, Otterly, Profound, or another, I want to read the actual responses behind any number I report. Reading twenty raw answers often teaches more than a month of charts, because you see how the model describes you, not just whether it named you.

Is a single snapshot ever useful?

A single snapshot is useful for spotting errors, not for measuring visibility. If one answer states the wrong price or confuses you with a competitor, that is worth fixing no matter how rare it is. But a snapshot cannot tell you whether your visibility went up or down. Only repeated sampling can.

This is where manual checks still earn their place. When a founder asks ChatGPT about their own company and sees something wrong, the instinct is to panic about visibility. The better move is to treat it as a factual accuracy problem and trace where the model might be getting the wrong idea, such as an old page or a stale directory listing.

Before any of this, set a baseline. If you change your site and then start measuring, you will never know what the change did. I walk through that in setting a baseline for AI visibility before you change anything.

Should AI visibility scores drive your marketing targets?

Not on their own. Use them as one signal alongside pipeline, branded search, and what buyers say on sales calls. A visibility score can rise while revenue stays flat, and vice versa. Set targets on outcomes you control and can verify, and use visibility trends to explain them, not replace them.

The risk of targeting the score directly is that teams start gaming the prompt set. They add easy prompts, drop hard ones, and report progress. The dashboard improves. Nothing changes in the market. I have strong opinions on this: a metric you can move without changing reality is not a metric worth setting goals against.

A better habit is to ask new leads how they found you, and to note when someone mentions an AI assistant. That self-reported signal is imperfect too, but it connects visibility to real buyers. Combined with appearance-rate trends, it gives a far more honest picture than either alone.

How can you check visibility without an expensive tool?

Pick ten to twenty real buyer questions, run each several times across two or three AI tools, and record whether your brand appears. Log the results in a spreadsheet with the date and tool. Repeat on a fixed schedule. It is slower than a dashboard, but you will know exactly how every number was made.

The manual method has a hidden benefit. Reading the answers yourself shows which sources the models lean on and which competitors keep appearing. That tells you where to spend effort, whether on your own pages, on review sites, or on third-party mentions.

I laid out a fuller version of this process in tracking AI search visibility without enterprise tools. For teams that later buy a tool, the manual log becomes the benchmark to check the vendor's numbers against.

What makes a visibility tool worth paying for?

A tool is worth paying for when it saves you the sampling work without hiding the method. That means editable prompts, many runs per prompt, clear engine coverage, raw answer access, and appearance rates with sample sizes. If it adds a proprietary composite score you cannot decompose, treat that score with suspicion.

Composite scores are the part I distrust most. When a tool blends mentions, sentiment, position, and citations into one number, you lose the ability to see what moved. A drop could mean you were mentioned less, or described less warmly, or that the prompt set changed. You need the parts, not just the total.

If you want a broader framework for what to measure, my guide on how to measure AI visibility for your brand covers the metrics I would track first. The short version is to keep the method visible and the claims modest.

What should you do next?

Open your current AI visibility tool and find out how the headline number is built. Check the prompt list, the runs per prompt, and whether you can read raw answers. Ignore rank positions. Track appearance rate over time, pair it with pipeline and buyer feedback, and fix factual errors whenever you spot them.

If your tool cannot explain its method, run your own small manual sample for a month and compare. The gap between the two will tell you how much to trust the dashboard. Either way, you will understand your AI visibility far better than a single score could show.

If you want help building an AI visibility tracking setup you can actually trust, or improving how AI engines describe your company, reach out. I do this work for B2B teams and I am happy to review what you are measuring today. Let's chat.