The short answer
AI answers change because they are written fresh each time, from sources that vary, for a person whose setup varies. Nothing is broken when two runs of the same question name different businesses. That is how the technology normally behaves.
For anyone trying to check their visibility, this has one big consequence. A single run proves very little. If you ask once and see your business named, you cannot conclude that you are reliably visible. If you ask once and do not see it, you cannot conclude that you are invisible. You have one draw from a range of possible answers, and the range is wider than most people expect.
This page explains the main causes, then moves to the practical question: how many runs make a fair reading. If you want the surrounding method for checking, our page on the checking method itself covers it, and this page can be read as the explanation for one of its rules, which is to run each question more than once.
The main causes of change
Five causes account for most of the variation people see. They stack, so the total variation is larger than any one of them suggests.
1. The answer is generated, not looked up
An assistant does not retrieve a stored answer. It writes one, word by word, choosing among likely continuations. That process includes a degree of chance, by design, so that replies do not read like stamped copies. Ask the same question twice and the sentences differ. When the question is about who to choose, the list of names can differ too, especially for borderline candidates.
2. The sources it reads can differ
Many assistants search the web or a database at the moment you ask, then write from what they found. The pages returned can change between runs, because search results shift and because the assistant may phrase its internal search differently each time. A different set of pages in front of it means a different set of names in the answer. Our page on how LLMs choose sources goes into the mechanics. An LLM, or large language model, is the kind of AI that writes these answers.
3. The wording of your question
"Best accountant in Denver" and "who should I hire for small business accounting in Denver" sound close and can produce different lists. Even a change of one word can steer the answer toward a different type of source.
4. The person asking
Location, language settings, past conversation and account history can all influence an answer. A logged-in assistant that remembers earlier chats may lean on them. Two people asking the same question can legitimately receive different replies.
5. The assistant itself changes
The companies update their models, their search connections and their rules about showing sources. An answer last month and an answer today can differ because the system behind it moved, not because anything about you did.
Why one check proves so little
Put those causes together and a single check is a sample of one from a distribution you cannot see. It is like judging a restaurant from one dish on one night.
Consider what a single run can and cannot tell you. If you appear, the real chance of appearing may be high, or it may be one in five and you were lucky. If you do not appear, the real chance may be zero, or it may be four in five and you were unlucky. The run cannot distinguish these. Only repeated runs can.
This matters in two directions, and both are costly. A false positive makes you complacent: you saw your name once, told your team, and stopped working on it. A false negative makes you despondent or sends you chasing a problem that is not there. Agencies and tools that show a single screenshot as evidence are, knowingly or not, relying on this effect. A screenshot of a good answer is an easy thing to produce, because with enough tries nearly any business can be shown named once.
The remedy is not to distrust everything. It is to treat each run as one observation and to collect enough observations that the pattern shows through the noise.
How many runs make a fair reading
There is no magic number, and anyone offering one with false precision is guessing. A practical rule of thumb for a manual check is this.
Run each question at least three times, and five if you can. With fewer than three, you cannot see variation at all. With five, you can tell the difference between a business that appears every time, one that appears sometimes, and one that never does.
Treat the share of runs as the result. If you appeared in two of five runs, the result for that question is two in five, or forty percent. Do not round it up to yes.
Use more runs for the questions that matter most. If a handful of questions are close to your sales, give them extra runs, say ten, and keep the rest at three to five.
Read across questions as well as across runs. Twenty questions at three runs each gives sixty answers per assistant, which is a much steadier picture than any one question. The overall share is more reliable than the share for any single question.
Our page on how to measure AI visibility shows how to turn these counts into a figure you can repeat. A larger sample reduces the wobble, but it never removes it, so keep showing the number of answers behind each percentage.
What counts as a real change
Because answers vary, you also need a rule for deciding whether a movement between two rounds is real.
First, establish your own noise. Run the identical set of questions twice in one week, with no changes made. Compare the two rounds. The difference you see, with nothing changed, is your natural variation. A later movement smaller than that is not evidence of anything.
Second, look for agreement. A real change usually appears in several questions in the same group and often in more than one assistant. A change that shows up in one answer is probably chance.
Third, look for persistence. A change that holds over two or three rounds is more believable than one that appears once. And a movement that you can tie to something you did, such as a new page on the topic, is more believable than one that appears from nowhere.
Finally, read the sentences, not only the counts. A shift in how you are described can be a real signal even when the counts look flat. Our page on reading AI visibility results shows what to make of the different outcomes.
Reducing the noise you add yourself
You cannot stop the assistants varying. You can stop adding variation of your own.
Freeze the questions. Same words, same order, every round. A new wording is a new test.
Use the same setup. A logged-out browser, the same country, the same assistants, and where an assistant offers a choice of modes, the same mode each time. Write the setup down.
Run in a short window. Do the whole round in a day or two, so that updates and news events do not slip in halfway through.
Record everything. The full text of the answer, the date, and the links. If a result looks odd, you can then check it against the record.
Keep a note of outside changes. A model update, a new feature, a new version of a source page. They are not your doing and you will want to remember them when reading the trend later.
A simple log is enough, and our page on an AI visibility tracking spreadsheet suggests the columns.
What tends to stay steady
It is not all chaos. Some things are far more stable than others, and knowing which helps you read a round.
Businesses with strong, consistent coverage across many independent sources tend to appear on most runs. Their placement within the answer may move, but they are usually in it. Businesses on the edge, with thin or scattered coverage, flicker in and out. That flicker is itself information. It tells you that you are close, and that a bit more clear, consistent material might tip you into reliable territory.
Facts about you tend to be steadier than opinions. An assistant that has picked up your location and what you do will usually repeat it. What it says about whether you are the best choice will move more.
And broad category questions tend to be steadier than narrow ones, because there are more sources agreeing. Narrow questions can swing widely with a single new page.
So when you see a business that is named every time, it is not luck. And when you see yourself named about half the time, you are not invisible. You are borderline, which is the most workable place to be.
Conclusions to avoid
Four tempting conclusions that the variation does not support.
"The AI is broken or unreliable, so none of this matters." Variation does not mean the answers are random. The pattern across many runs is real and informative. The skill is in reading the pattern, not any one answer.
"We were named, so we are fine." One mention is one draw. See how often you are named before relaxing.
"We were not named, so our work failed." The same applies in reverse. A single miss after real work is not evidence the work did nothing.
"A competitor appeared once, so they have overtaken us." Count how often they appear. A business named in one of five runs is not ahead of one named in three of five.
If you want a plain explanation for a colleague, say this: assistants write a fresh answer each time, so we judge by how often we appear across many runs, not by any one run.
A ten-minute experiment to see it for yourself
You do not have to take any of this on trust. Pick one question a buyer might ask about your category. Open one assistant in a logged-out window. Ask the question, and copy the list of businesses named. Then start a fresh conversation and ask the identical question again. Do it five times in all.
Lay the five lists side by side. You will almost certainly see some names on every list, some on two or three, and a few on only one. The names on every list are the assistant's steady answers. The ones that come and go are the edge cases. Now find where you sit. If you are in the steady group, good. If you are in the flickering group, you are borderline. If you are in none, you have learned that more than luck is involved.
Do the same with a slightly reworded version of the question and notice how much the lists move. That is the effect of wording alone. This small exercise teaches more about visibility than any screenshot, and it takes about ten minutes.
Claims to be careful with
Variation also gives you a way to test what suppliers tell you.
If someone claims to have put you in the answer, ask how many runs it was based on and whether they will repeat it in front of you. If a report shows you ranked third in AI answers, ask how rank can exist when each answer is written fresh. If a tool shows a score that moves by a few points week to week, ask how it separates real movement from run-to-run noise.
Honest suppliers have good answers to these questions. They show the number of runs, they show the spread, and they tell you when a change is within noise. Vague answers are a useful signal in themselves.
None of this is a reason to avoid the work. It is a reason to ask for evidence that holds up when the same question is asked again.
How RunOctopus treats variation
Because one answer proves little, RunOctopus does not rely on a single screenshot. Its reports on where ChatGPT, Perplexity, Claude and Gemini name a business are part of the paid service, and they are meant to be read as patterns across answers. There is no free automated AI visibility checker on the site, so a quick free look would be a single run, which is exactly what this page warns against.
The service itself is done for you. Ollie, the company's AI, researches the business and writes and publishes pages on its own website, with a team that plans, writes, reviews and publishes alongside it. It works with Shopify, WordPress, WooCommerce and Wix, and the owner does not install, approve, upload or schedule anything. The 90-day promise is more visibility on Google and in AI answers within 90 days, measured against the 90 days before joining, or every dollar back.
The free site read at app.runoctopus.com/start reads your site and shows the plan. It does not test AI answers. No account, no call.