LLM Brand Monitoring: One Practice, Several Assistants

Two levels, two different questions

Some questions only make sense across all assistants at once, and some only make sense one assistant at a time. Mixing them is why so much monitoring output feels informative and changes nothing.

The aggregate question is whether you are present in your category at all. That is a fact about your business and it should be read across everything, because being named in one and absent from three is a meaningfully different position from being named in all four.

The per-assistant question is whether a specific change landed. Assistants differ in how quickly they pick things up, so a rewrite can be visible in one within weeks and invisible in another for a quarter. Averaging that hides the only feedback you were going to get.

What is meaningless at either level is a single blended score across all assistants and all questions. It moves for reasons nobody can name and it invites explanations that are invented.

Why assistants disagree about the same business

Two assistants asked the same question about the same business can give genuinely different answers, and the reasons are worth understanding because they change what you do.

They draw on different material. What one has access to and reaches for is not what another does, so a source that carries you in one may be absent in the other.

They refresh at different rates. A change you make can surface quickly in one and slowly in another, which means an early reading in the slow one is not evidence that nothing worked.

Some show sources and some do not, which changes what you can observe rather than what is true.

And they compose differently. One may name several options where another names one, so the chance of appearing at all differs before anything about you is considered.

That last point is worth dwelling on because it distorts naive comparisons badly. If one assistant habitually names five sources and another names one, your presence rate in the first will be higher for reasons that have nothing to do with your standing. Comparing the raw percentages across the two and concluding you are stronger in one is a mistake that gets made constantly. What is comparable is the movement of each number over time against itself, not the numbers against each other.

What to track across all of them

Keep the collected data identical across assistants even though the analysis differs, because comparability is the thing that makes the exercise worth anything.

The same fixed question set, in buyer language, asked the same number of times in each reading. If the set differs between assistants you cannot compare them and you have lost the aggregate view.

Whether you were named, recorded as a proportion of askings rather than a yes or no.

Which other names appeared, because the stability of that set across assistants tells you whether the category has entrenched sources or is genuinely open.

Which sources were credited, where they are shown at all.

And the wording around your name, which is frequently the most useful column and the one nobody keeps.

The wording column earns its place because presence is binary and reputation is not. Being named as the obvious choice, being named as one of several, and being named as the cheaper alternative are three quite different outcomes that all count as present. Over several months the drift in that language is often the earliest sign that something is changing, in either direction, and it arrives long before the presence proportion moves enough to be convincing. It costs nothing extra to capture because you are already reading the answer.

Reading the aggregate view

The aggregate answers strategic questions rather than tactical ones, and there are three worth asking of it.

Are we present anywhere. Being named by one assistant and no others is a different situation from being named by none, and it usually means the material carrying you exists but is thin.

Is the competing set stable across assistants. If the same four names dominate everywhere regardless of phrasing, the category has entrenched sources and a narrower subject is the realistic path.

Is the gap widening or closing. Over several months, the proportion of questions naming you across everything is the single number that reflects whether the program is working, and it is the only number worth putting in front of anybody senior.

Present that number with its denominator attached, always. Named in eleven of forty-five askings is a statement somebody can interrogate and trust. Twenty-four percent is the same fact with the evidence removed, and it invites exactly the question you cannot answer, which is twenty-four percent of what. The habit of carrying the denominator through every report is the single cheapest thing you can do to make this work credible to people who were not in the room when the method was agreed.

Reading each one individually

The per-assistant view is where the work gets its feedback, and it should be read differently from the aggregate.

Use the source-showing assistants as your instrument. They tell you which page and often which passage was used, which is direct feedback on an edit and is the closest thing to a controlled experiment this discipline offers.

Expect the fast movers to show a change first. That is not the change being real only there, it is the change being visible there first, and treating the slow one as a contradiction is a common misreading.

Give any change at least two readings in the slow assistants before concluding it did not work there. One reading is a sample everywhere, and it is a particularly poor sample in a system that refreshes unevenly.

Watch for one assistant diverging sharply from the others over several readings. That usually means the material carrying you in the others is not reachable in this one, which is a retrieval question rather than a content one.

Our pages on ChatGPT and Perplexity go into what each behaves like in practice.

How many assistants to watch

More is not obviously better, and the cost of each additional one is a real ongoing tax on a small team.

Two is the practical minimum. One is not enough because you cannot tell a general change from a particular one, and everything you conclude will be about that assistant rather than about your business.

Three or four covers the realistic range of behavior without the reading becoming a morning's work. Beyond that you are adding cost for diminishing information.

Choose at least one that shows its sources and at least one that does not, because the first gives you feedback and the second is closer to how many people actually experience this.

Weight your attention rather than your coverage. Read all of them, and investigate the one that gives you sources.

Adding a fifth or a sixth is usually a decision about thoroughness rather than about information, and it has a cost that is easy to underestimate. Each additional assistant adds askings to every reading, and because each question is asked several times, the arithmetic compounds faster than people expect. A set of ten questions asked five times across six assistants is three hundred askings a month, which stops being an hour and starts being a job. Cap the coverage deliberately rather than letting it grow, and revisit the cap annually rather than every time a new assistant is written about.

The one case for going wider is a business selling into a market where a particular assistant genuinely dominates. If your buyers overwhelmingly use one, watch that one closely regardless of whether it shows its sources, and treat the others as a sanity check rather than as equals.

Keeping readings comparable over time

The most common way this work becomes worthless is drift, and it happens quietly.

Somebody edits a question because it seems unfair. A tool updates its defaults. An assistant is added or dropped. Six months later the history exists and cannot be compared with itself.

Fix the set and never edit it. If you need new questions, add a second set alongside and keep the original running, even when the original questions start to look dated.

Record the assistant and the date on every row, which sounds obvious and is routinely skipped, and makes an archive uninterpretable a year later.

And write down the method. How many askings, on what day, by whom. The person who set it up will not be the person reading it in a year, and an undocumented method quietly becomes a different method.

The version of this that survives staff changes is a single page next to the data that states the questions verbatim, the number of askings, the assistants in scope, the day of the month, and the reason each question was chosen. That last field matters more than it looks. A question whose purpose has been forgotten is the first one somebody edits when it produces an uncomfortable answer, and the reason it was chosen is usually the reason it should stay.

What this cannot tell you

Three limits are worth stating so nobody overclaims from the readings.

It cannot tell you why. You observe that a name appeared or did not. The reasoning is not inspectable and any confident account of it is reconstruction, however plausible it sounds and however senior the person offering it.

It cannot tell you what it was worth. Presence is not traffic and not revenue, and the visit that might follow arrives as direct traffic with no trace of its cause. Any vendor offering to close that loop is describing something the plumbing does not currently support.

It cannot tell you that your question set is wrong. The set is a model of buyer behavior and monitoring cannot audit the model, which is why it is worth rebuilding from fresh customer language occasionally even though that breaks comparability.

The practical resolution to that tension is to run two sets permanently. The original, frozen, however dated it looks, because it is the only thing that gives you a comparable multi-year line. And a rolling set rebuilt annually from language customers used that year, which tells you what is actually being asked now. The two will diverge, and the divergence is itself informative, because it shows you how fast the vocabulary of your category is moving.

The honest trade-off

Watching several assistants multiplies the reading time and the analysis, and it produces a picture that is harder to summarize for anybody senior.

One assistant is much cheaper to watch and produces conclusions that may be about that assistant rather than about you. That is the real risk of economising here, and it is not obvious until a decision has already been made on a false general.

The workable compromise is a full reading across three or four with a short summary, and deep investigation only in the one that shows sources. That keeps the aggregate honest and the tactical feedback sharp without turning monitoring into somebody's main job.

It is also worth accepting that this practice will look disproportionate for the first two quarters. You are building a measurement apparatus for a channel that may not move for months, which is a hard thing to justify while it is happening and an obvious thing to have done once somebody asks whether any of the work paid off. The businesses that end up with a defensible answer to that question are the ones that started collecting before they had anything to show.

Bottom line

Read presence across all assistants and read change one assistant at a time. Those answer different questions and blending them into one number answers neither.

Watch at least two, include one that shows its sources, and keep the question set and the method identical across all of them so the readings stay comparable.

And never edit the original set. Add a second one alongside it if you must, because a history that cannot be compared with itself was an expensive way to collect nothing.

Carry the denominator into every report. Named in eleven of forty-five askings survives being questioned. A percentage on its own does not, and the first person who asks what it is a percentage of will be the person deciding whether this work continues.