AI Visibility Benchmark: Against What, Exactly?

The comparison set is the whole decision

Benchmarking AI visibility is almost entirely a question of what you compare yourself with, and most of the value or damage is decided before a single reading is taken. The arithmetic afterwards is trivial. The choice of comparison is where the judgment lives, and it is usually made by default rather than deliberately.

The default choice is a list of named competitors, because that is what everybody has to hand and what leadership asks about. It produces a table showing who was named more often, which feels like a benchmark and behaves like a scoreboard. The trouble is that the assistants answering these questions are not running a competition between the companies in your category. They are assembling an answer from whatever material is available, and the names that appear are frequently not your competitors at all.

The more useful comparison is the whole set of sources being named across many askings, whether or not they are businesses like yours. If publications and community discussion dominate every answer, that is the single most strategically important fact available to you, and a competitor table will not show it because publications are not on the list.

The four comparisons worth running

Four comparisons produce genuinely different information and it is worth being deliberate about which you are running, because they answer different questions and only two of them survive a single reading.

Category composition. Across all your questions and all askings, what kinds of source get named. Publications, community discussion, specialist sites, vendors. This is the most useful single reading you will ever take, and it tells you whether the realistic path is your own pages or being referenced by other people.

Category stability. Whether the same names appear regardless of how a question is phrased. A stable set means entrenched sources and an expensive frontal attempt. A set that shifts with phrasing means the category is unsettled and there is room.

Named rivals over time. Whether a specific competitor is pulling away or falling back over several readings. This is legitimate and it needs months of data before it says anything, because a single reading of two companies is two samples from two distributions.

Your own movement. The only comparison that is unambiguously about you, and the one most worth the effort. Against your own fixed set, over time, with the question groups kept separate so that progress in the winnable ground is not averaged away by the broad question you were never going to take.

The comparison that misleads most

A single reading comparing you with named rivals is the most commonly produced benchmark and the least trustworthy artifact in this whole discipline, and it is worth being blunt about why.

Every element of it is a sample. Your presence is a draw from a distribution, and so is each competitor's. Comparing two draws and concluding that one company is ahead is like comparing two coin flips and concluding one coin is better. Ask again tomorrow and the table may reverse with nothing having changed anywhere.

It also invites the wrong response. A table showing a competitor named more often produces a request to study that competitor's pages, which is only useful if you have already established that you are being retrieved and passed over rather than never retrieved at all. If your access is refused, their page structure teaches you nothing you can use.

And it obscures the interesting finding, which is what else was named. A comparison table lists the companies somebody chose to compare. It cannot show you that in every answer the dominant sources were three publications and a forum, which is the fact that should be reorganizing your quarter.

Building a benchmark that holds up

A defensible benchmark takes a morning to design and an hour a month to maintain, and the design decisions matter far more than the maintenance.

Start with the question set, in customer language, fixed forever. Six to ten questions covering the three groups: ones that name you, specific long-tail ones, and the broad category question. This is the same set you use for everything else in this work and it should not be duplicated or varied for benchmarking.

Record every name that appears in every answer, not just the ones on a list you made in advance. This is the step that separates a benchmark from a scoreboard, and it costs nothing because you are already reading the answer. The names you did not expect are the ones worth having.

Ask each question at least five times per reading, and treat the proportion as the unit rather than the answer. Then read the composition of names by frequency rather than by presence, so a source appearing in every answer is visibly different from one appearing occasionally.

Take the first reading before any work starts. A benchmark without a starting point is a description rather than a comparison, and it cannot be reconstructed later.

One design decision is worth making explicitly rather than by accident, which is whether to record names that are obviously not competitors. The instinct is to filter them out to keep the table tidy, and it is exactly the wrong instinct. A publication appearing in every answer is the most important row in the dataset even though it will never buy anything from you and you will never take business from it. Keep everything, tidy nothing, and let the composition speak.

The second is how to handle a name that appears once and never again. Those are noise individually and informative in aggregate, because a long tail of one-off names indicates an unsettled category and a short tail indicates an entrenched one. Record them, count them, and read the shape of the tail rather than chasing any individual entry in it.

How to read the composition

The composition of named sources tells you which strategy is available to you, and there are three patterns that recur often enough to plan against.

If the named sources are mostly publications and community discussion, your own pages are not the constraint. You can write excellent material and remain absent, because the answers are being built from third-party discussion of your category. The realistic work is being discussed there, which has no schedule and no deliverable, and that is an unwelcome conclusion that is better reached early than after a quarter of writing.

If the named sources are mostly specialist sites, the category rewards depth and you can compete on it. This is the most encouraging pattern because it is the one your own effort can change, and it means a narrow, thorough treatment of one subject is a realistic path to being named.

If the named sources are mostly vendors including your competitors, the category is commercially contested and being named is largely a function of being reachable, quotable and deep. That is a fair fight and it is winnable with the ordinary work, and it is the pattern where a well-run program most reliably shows a result within two or three quarters.

Most categories are a mixture, and the mixture usually shifts by question type. Broad category questions tend to draw publications and long-tail questions tend to draw specialists, which is exactly why the three question groups should never be averaged into one number.

Reading that shift is where the strategy actually comes from. If your long-tail questions draw specialist sites and your broad questions draw publications, then the winnable ground is the long tail and the broad question is a multi-year proposition at best. That is a concrete planning conclusion drawn from a morning of reading, and it is considerably more useful than any score. Our page on the visibility score covers why blending the groups hides exactly this.

On share of voice

Share of voice is the phrase most often used for this and it carries an assumption worth examining, because the assumption does not hold here.

In advertising, share of voice is a proportion of a finite quantity. There is a total amount of impression and you own some percentage of it. That framing works because the denominator is real and fixed.

In generated answers there is no fixed denominator. An answer might name one source or six, and the number varies by question, by assistant and by run. Your share of a thing whose size changes every time you measure it is not a stable quantity, and treating it as one produces a number that moves for reasons that have nothing to do with you.

The version that does hold up is presence frequency. Across a fixed set of questions asked a fixed number of times, what proportion of askings named you. That is a real number with a real denominator and it is the honest form of the same idea. If somebody insists on the phrase share of voice, define it that way explicitly in the report so at least everybody is discussing the same quantity.

There is a second-order version worth collecting once the basic reading is established, which is how many names appear in a typical answer to each of your questions. A category where answers routinely name one source is a very different competitive environment from one where answers name six, and the difference decides how much room there is for you. Six names means six chances to be included. One name means somebody has to be displaced, and displacement is a far harder proposition than inclusion.

That number also tends to differ by assistant more than almost anything else you will record, which is another reason not to average across them. An assistant that names one source and an assistant that names five are running different competitions, and your presence rate in each is not comparable with the other however tempting the arithmetic looks.

The honest trade-off

A proper benchmark takes longer to read than a competitor table and produces a conclusion that is harder to act on, which is a real cost and the reason competitor tables persist.

Telling a leadership team that the dominant sources in your category are three publications and a forum is a much less satisfying answer than telling them a named rival is ahead. The first implies work with no schedule. The second implies a project.

The argument for doing it anyway is that the satisfying answer is frequently false, and acting on it costs a quarter. A category dominated by third-party discussion will not respond to a content program however well executed, and finding that out in month one instead of month nine is worth the discomfort of the conversation.

There is also a quieter benefit that nobody plans for. Recording every name that appears, rather than only the ones on a list, surfaces sources you did not know existed. Businesses regularly discover through this exercise that a publication they have never spoken to, or a community they have never participated in, is the single most influential source in their category. That is a lead rather than a metric, and it arrives free with a benchmark done properly. Our piece on citation tracking covers reading the credited sources in more detail.

Bottom line

Benchmark against the field of named sources rather than a list of rivals you chose. What else appears in the answers is more informative than where you sit against three competitors, and it costs nothing extra to collect.

Never read a single reading as a comparison. Every number in it is a sample, and comparing two samples produces a table that can reverse tomorrow with nothing having changed.

Read the composition first, because it tells you which strategy is even available. Then track your own movement over time against a fixed set, which is the only comparison that is unambiguously about you.

And take the first composition reading before any work begins. It is a morning of effort, it will probably reshape what you were about to commission, and like every other baseline in this discipline it is the one thing that cannot be produced after the fact.