How to Measure AI Visibility: What to Count and How Often

What you are actually measuring

To measure AI visibility, you ask AI assistants the questions your buyers ask, write down what comes back, and count how often your business is in it. That is the whole method. Everything else is detail about how to keep the counting honest.

It helps to be clear about what kind of measurement this is. You cannot read a number off a dashboard that the assistants publish about you, because they publish nothing. There is no equivalent of Search Console for what ChatGPT says. So the measurement is a sample. You take a fixed set of questions, you run them, and the answers are your sample. A good measurement is one where the sample is fair, the counting rules are written down, and you can repeat it next month and compare.

If you want the plain walkthrough of running a check by hand or with software, our page on the AI visibility checker covers the options. This page is about the part after you have the answers in front of you: what to count, how often, and how to end up with a figure you would be comfortable showing a partner or a client.

One warning before the detail. A measurement that flatters you is worse than none, because it makes you stop working on the problem. Every rule below exists to stop the count drifting upward for reasons that are not real.

The three things worth counting

Most people count one thing: was my business mentioned. That is a start, but it lumps together very different outcomes. Three counts, kept separate, tell you far more.

Named

Your business appears in the answer by name. This is the widest net. It includes a business named as the one to avoid, so you still have to read the sentence. Record the sentence next to the tick, even if it takes a few extra minutes.

Cited with a link

The assistant shows a source link that points to your website, or names your page as where the information came from. This is narrower than being named. A business can be named because a review site mentioned it, while its own site is never used. Being cited means your own pages are doing the work. Not every assistant shows links in every answer, so note which ones do and compare like with like.

Recommended first

When the answer gives a shortlist or a single pick, you are the first one named or the one it says to choose. This is the narrowest count and the one closest to a sale. An assistant that names you fourth in a list of five has mentioned you. It has not recommended you.

Keep a fourth column for the opposite case: absent. It sounds obvious, but a count of "named" with no count of "absent" hides how many answers had no trace of you at all. The absent answers are where the work is.

Why separate them? Because they respond to different causes and different fixes. If you are absent, the assistant has little reason to know you exist. If you are named but never cited, your own pages are not the preferred source. If you are cited but never first, something about how you compare is losing. The page on reading AI visibility results goes through what each outcome means.

How many questions make a fair sample

Too few questions and one lucky answer moves your whole figure. Too many and you will stop running the check, because it takes an afternoon. A workable range for a small business is ten to thirty questions, and the number matters less than the mix.

Build the set from the questions a buyer asks at different points. Some are broad, like what kind of provider to hire. Some are specific, like which provider suits a certain budget or a certain problem. Some compare two options by name. Some are about you directly. If every question is a broad one, you will score low and learn little. If every question names your brand, you will score high and learn nothing.

Write the questions in your buyers' words, not yours. The easiest source is your own inbox and your sales calls. How do people phrase the first message they send you? That phrasing is the question. The page on writing the question set goes through this step by step.

Once you have the set, freeze it. Do not add a question because you suspect you would do well on it, and do not delete one because you do badly. If you must change the set, start a new series and say so. Then keep the old series running beside it for a few rounds, so you can see what the change did to the numbers.

How often, and how many runs per question

Two different frequencies matter, and people mix them up.

Runs per question, per round. The same question asked twice can give two different lists of names. That is normal, and it is the single biggest source of false readings. Our page on why AI answers change explains the mechanism. The practical consequence is that one run is a coin flip. Run each question several times in a round, count the share of runs where you appear, and treat that share as the result for the question. Three to five runs is a sensible rule of thumb for a manual check. More is better if you can afford it, but fewer than three tells you almost nothing.

Rounds over time. Once a month is enough for most small businesses. Weekly is useful in the first month of a change, because you want to see whether something moved, but weekly numbers wobble and are easy to over-read. The assistants change their own behavior, and changes in your numbers can come from them rather than from anything you did.

Whatever rhythm you choose, run it on the same day of the month, with the same accounts logged out, in the same location. Small differences in setup produce differences in answers, and you want the only variable to be your own progress.

Turning the counts into one honest number

Eventually someone asks for a single figure. Give them one, but build it so it cannot lie easily.

The simplest honest figure is a share. Take every answer you collected in the round, which is questions times runs times assistants. Count how many of those answers named you. Divide. That gives a percentage of answers that include you. Do the same for cited with a link and for recommended first. You now have three percentages, and the order will always be named, then cited, then recommended, from largest to smallest.

Report them per assistant as well as combined. A combined figure of thirty percent could mean thirty percent everywhere, or ninety in one assistant and nothing in the others. Those are very different problems. If you want to see how a vendor might blend these into a single score, and what that blending hides, read our page on the AI visibility score.

Two rules keep the number honest. First, always show the denominator. Seven answers out of ten is a different statement from seventy out of a hundred, even though the percentage matches. Second, never hide the absent count. Reporting only the wins is how a measurement turns into marketing.

What not to measure

Some numbers look useful and mislead.

Referral traffic from assistants. Most recommendations never produce a click. A person reads the answer, remembers your name, and searches for you later or types your address. That visit shows up as branded search or direct traffic, with nothing marking where it began. Traffic from assistants is real but it is a small and partial slice. If it is your only measure, you will report failure during genuine progress.

A single invented position. Some tools report that you rank number three in AI answers. There is no position to hold. An answer is written fresh each time, and what looks like rank is the order of a sentence. Count whether you were in the answer and how you were described.

Sentiment scores without the quote. A score of seventy-two for tone is a figure with no way to check it. Read the actual sentence instead. It takes seconds and it is the only way to catch an assistant calling you the cheap option or the old-fashioned one.

Anything measured once. A single run of a single question is an anecdote. Anecdotes are fine for curiosity and bad for decisions.

Keeping it comparable from month to month

A measurement is only useful if next month's version can be compared with this month's. That requires a few boring habits.

Write down the rules once. What counts as named, what counts as cited, what counts as first. If you let the definition stretch, your numbers will rise for no reason. For example, if you start counting a mention in a footnote as a recommendation, you will see an improvement that is only a change in your own standards.

Record the date, the assistant, whether you were logged in, and the exact question text for every run. Store the full answer text, not only your tick marks, so that a disagreement later can be settled by reading what was said. A spreadsheet is enough. The page on an AI visibility tracking spreadsheet lays out the columns.

Note anything that changed outside your control. If an assistant launched a new feature, changed how it shows sources, or you moved country, write that beside the round. Without notes, you will later credit your own work for a movement that came from elsewhere, or blame your work for a drop that did not originate with you.

Five ways the measurement goes wrong

These come up so often that they are worth listing.

Testing while logged in. Assistants personalize. If you are logged into an account that has seen your own website many times, the answers lean toward what the assistant already knows about you. Use a logged-out window, or a clean profile, so you see roughly what a stranger sees.

Asking leading questions. "Is Acme the best plumber in Tulsa" will get you a different result than "who is a good plumber in Tulsa." Only the second tells you anything about visibility.

Checking one assistant. Each assistant draws on different sources and behaves differently. A strong showing in one says little about the others. Check at least the ones your buyers use.

Stopping after a good round. Because answers vary, any single round can be lucky. Two good rounds in a row is a signal. One is a hint.

Treating the number as the goal. The figure is a thermometer, not the treatment. If the number is low, the useful question is what the assistants are drawing on instead of you. Our page on what to do after a check picks up from there.

A small worked example

Here is the arithmetic with made-up numbers, so you can see the shape. Say you test twelve questions, run each four times, in three assistants. That is one hundred and forty-four answers. You read them all and tick the boxes.

Suppose you were named in thirty-six of those answers, cited with a link to your own site in nine, and recommended first in six. Your named share is thirty-six over one hundred and forty-four, which is a quarter. Your cited share is a small fraction of that, and your first-pick share is smaller still. The gap between the three figures is the interesting part. It says the assistants know you exist, mostly through other sites, and rarely choose you.

Next month you run the identical set. If the named share stays the same but the cited share doubles, your own pages have started to carry weight. That is a real change you can point to. If everything rises together by a few points, be cautious and run another round before you believe it.

Where RunOctopus fits in this

RunOctopus is a done-for-you service, not a measuring tool you run yourself, and it does not offer a free AI visibility checker. Its AI, Ollie, researches your business and then writes and publishes pages directly on your own website, working alongside a team that plans, writes, reviews and publishes. It works with Shopify, WordPress, WooCommerce and Wix, and there is nothing to install, approve, upload or schedule.

For customers, the service includes reports on where ChatGPT, Perplexity, Claude and Gemini name the business, so the counting described above is part of what you receive rather than something you build in a spreadsheet. The work comes with a 90-day promise: more visibility on Google and in AI answers within 90 days, measured against the 90 days before joining, or every dollar back.

If you want to see what Ollie makes of your site before deciding anything, the free site read at app.runoctopus.com/start takes your address, reads the site and shows the plan. It does not test AI answers. It needs no account and no call.