Why Do AI Answers Change Every Time You Ask the Same Question?

Why do you get a different AI answer every time you ask?

Because these systems are non deterministic by design and, to a degree, by accident. The model providers document this themselves. Identical inputs can produce different outputs, and that remains true even when you remove every setting that is supposed to introduce randomness. Variation is the normal state, not a malfunction.

This matters far beyond curiosity. A large amount of AEO and brand monitoring work rests on asking an AI system a question and treating the answer as a measurement. If the answer moves on its own, a single observation tells you very little, and a lot of confident conclusions are being drawn from single observations.

I want to go through what the vendors actually document, because this is one of those areas where the folk explanation and the documented behaviour differ, and the difference changes what you should do.

What do the model providers actually say about this?

They say it plainly in their own documentation. OpenAI states that Chat Completions are non deterministic by default, which it explains as meaning that model outputs may differ from request to request. That is the vendor describing the baseline behaviour of its own product, not a third party observation.

OpenAI also describes what it offers as partial mitigation. It gives access to a seed parameter and a system fingerprint response field, and says that to receive mostly deterministic outputs you set the seed to an integer and use the same value across requests, while ensuring all other parameters such as the prompt and temperature are exactly the same.

Notice the word mostly, which is OpenAI's own hedge and not mine. Even with every input pinned, the vendor describes the result as mostly deterministic rather than deterministic. That gap is the whole subject of this article.

Does setting temperature to zero fix it?

No, and Anthropic says so directly. Its glossary states that users may encounter non determinism in APIs, that even with temperature set to zero the results will not be fully deterministic, and that identical inputs may produce different outputs across API calls. This applies to its own service and to third party providers.

This is worth sitting with because temperature zero is widely treated as a determinism switch. Anthropic describes temperature as a parameter controlling the randomness of predictions, where lower temperatures produce more conservative outputs that stick to the most probable phrasing. Lower randomness is real. Zero randomness is not on offer.

So if you have been setting temperature to zero in an evaluation harness and treating the output as a fixed value, the vendor documentation says that assumption does not hold. You have reduced the variance, which is genuinely useful, and you have not eliminated it, which means your harness still needs to handle variation rather than assume it away.

What else changes the answer besides randomness?

The system underneath you changes. OpenAI documents that determinism may be impacted due to necessary changes it makes to model configurations on its end, and exposes the system fingerprint field so you can detect this. If that value differs, OpenAI says you may see different outputs because of changes made on its systems.

That is a category of variation most people never account for. Your prompt is identical, your parameters are identical, your seed is identical, and the answer changed because the provider adjusted something. From outside, this is indistinguishable from your content having improved or declined, which is exactly the confusion that ruins AEO measurement.

For consumer facing AI search products the picture is murkier still, because retrieval is layered on top of generation. What the system found this time, how fresh the index is, and what it chose to ground the answer in all move independently of the model. Two of those can change without anything about the model changing at all. I looked at a related problem in how AI answer engines handle conflicting information.

Why does this break brand visibility monitoring?

Because a single query is a sample of one from a distribution, and people treat it as a reading from an instrument. You ask whether an AI system recommends your product, it does not mention you, and you conclude you are invisible. Ask again tomorrow and you might be there. Neither answer was wrong and neither was a measurement.

The failure mode I see most often is the celebratory screenshot. Somebody queries a chatbot, sees the brand named, screenshots it, and it circulates internally as evidence the AEO work is paying off. That screenshot is one draw. It is not nothing, and it is not a result, and the distance between those two things is where a lot of budget gets misallocated.

The inverse is more damaging. A team runs one query, does not appear, and concludes a strategy has failed. Given documented non determinism, a single absence is extremely weak evidence. Before you change direction on the basis of an AI answer, you need to know whether you are looking at a trend or at noise.

How many times should you check before you believe a result?

Enough times that you can talk about frequency rather than presence. I would not act on fewer than ten runs of the same question, spread across days rather than fired in a minute, and I would record how often you appeared rather than whether you appeared.

Frequency is the right unit here and almost nobody uses it. Appearing in seven of ten runs and appearing in one of ten are wildly different situations that both get reported as we show up in AI answers. The first is a position worth defending. The second is a coin flip you happened to win.

Spreading the runs across days matters because it catches the provider side changes rather than just the sampling noise. Ten queries in one minute measure one configuration ten times. Ten queries across two weeks measure something closer to what your buyers actually experience, which is the thing you were trying to learn about. I set out a fuller approach in how to measure AI visibility for your brand.

What does this mean for how you write content?

It raises the value of being unambiguous. If the system is sampling from a distribution of plausible continuations, then content that states a fact one way, clearly, in one place, gives it fewer ways to go wrong than content that hedges the same fact across several paragraphs.

This is an argument for a particular kind of writing discipline. One canonical term per concept rather than three synonyms. A direct answer near the top rather than a build up. Facts stated once and completely rather than assembled across a page. None of this is new advice, but non determinism gives it a sharper justification than tidiness.

It is also an argument against over interpreting any single quotation of your work. If an AI system paraphrases your page badly once, that is worth noting and not worth panicking about. If it paraphrases it badly seven times out of ten, your page has an ambiguity problem and you can usually find it. The underlying selection question is one I covered in how AI search picks between two pages that say the same thing.

When should you care and when should you not?

Care when you are measuring, building, or making a decision on the output. Do not care when you are using these tools to think. Variation is a problem for instruments and a feature for collaborators, and most of the frustration I see comes from wanting one thing while using the other.

If you are building an automation where a model classifies or extracts something, variation is a correctness problem and you should design for it explicitly, with validation, constrained outputs, and a defined behaviour when the result fails the check. Assuming stability that the vendor documentation explicitly denies is how automations quietly produce wrong data for months.

If you are drafting, exploring, or pressure testing an idea, variation is doing you a favour. Asking the same question twice and getting two framings is a cheap way to see the shape of a problem. I would not want that determinism even if it were available.

What should you do next?

Take whatever AI visibility claim you are currently working from and ask how many observations it rests on. If the answer is one, it is not yet a finding. Re-run it ten times across a week and write down the frequency, then decide what you think.

If you run any automation that depends on a model returning the same shape of answer every time, add a validation step this week that checks the output before anything downstream uses it, and define what happens when the check fails. The vendors document that identical inputs can produce different outputs, so build as though they will.

And when someone shows you a screenshot of an AI system recommending a competitor, treat it the way you would treat one customer's opinion. Interesting, worth following up, not yet a fact. If you want help building a measurement approach that survives this problem rather than ignoring it, reach out and let's chat.