The questions are the measurement
When you test AI visibility, the questions are the measuring instrument. The assistant is only the thing being measured. If the questions are wrong, the result is wrong, however carefully you record the answers.
Two failures come up over and over. The first is a set that flatters you, built from the phrases on your own site, which assistants will often match because the phrases are yours. The second is a set that keeps changing, so that each month's result is a reading of a different thing and no trend can ever be seen.
This page is about avoiding both. It covers where the questions come from, how to write them, how many you need, and why you should freeze the set. If you want the whole check around it, from asking to recording, start with our complete check from start to finish, and for a quick run with ten questions, the free manual check shows the routine.
Write ten to thirty questions in your buyers' words, leave your name out, cover several kinds of question, and never change the wording once the first run is done.
Where the questions should come from
You already own the best source of questions. It is the way your customers talk before they have decided anything. Mine these places, in roughly this order.
- Emails and messages from customers. The first line of a pre-sale inquiry is often a perfect question.
- Phone calls and in-store questions. Ask the person who answers the phone to write down the five questions they hear most.
- Reviews, yours and your rivals'. Reviews show what people were worried about and what they were glad to have found.
- Forum threads and community posts in your category. People ask in plain language there, with all the doubt included.
- The "people also ask" boxes and suggested searches in Google for your main terms.
- A friend outside your industry. Ask them, "How would you search for what I sell?" The answer is often humbling.
Notice what is missing from that list. Your own website, your product descriptions and your slogans are not sources. They are written to sell, not to ask. A question copied from your homepage will probably succeed in matching your homepage, which proves nothing.
For another view on the way buyers phrase things, our page on ChatGPT prompts for SEO covers how people actually type to assistants, though it is aimed at a different job.
Cover the different kinds of question
A buyer does not always ask the same way. A good set spreads across the ways they ask, so that you see where in the decision you are visible and where you are not.
The shortlist question. "Best X for Y." The buyer wants names.
The problem question. "My X keeps doing Z." The buyer does not yet know what to buy or whom to hire. Informational pages tend to matter here.
The comparison question. "X versus Y." The buyer has narrowed down and wants help choosing.
The constraint question. "Cheapest option that still does A." A budget, a size, a deadline or a requirement is added.
The place question. For local work, the town or area. Our page on checking AI visibility for a local business shows how to build these in.
The doubt question. "Is X worth it." "Is it safe to." The buyer is close to buying and wants reassurance.
For an online store, shopping-specific shapes such as gifts and where-to-buy are described in checking AI visibility for an online store.
Aim for at least four of these kinds in any set. If every question is a shortlist question, you are measuring one stage of buying and calling it visibility.
How to write each question
Write each question the way a person would say it out loud to a friend who knows the area. A few habits help.
Use full sentences. Assistants are built for conversation. "What should I look for when buying a standing desk for a small apartment" tests more than "standing desk apartment."
Include the real constraint. The detail is what makes a question specific, and specific is where assistants name particular businesses instead of giving generic advice.
Use the plain word your customers use. If they say "fix," do not write "repair." If they say "vet," do not write "veterinary practice."
Leave your brand out. Always. A question that contains your name is a test of what the assistant knows about you after being told who you are. That is a different and much easier test.
One idea per question. Do not stack three questions into one. It confuses the answer and makes the result hard to record.
Avoid leading phrasing. "Why is X the best option" assumes the answer. Ask "which option is best" and see what comes back.
A small test: read each question aloud. If it sounds like a search box entry rather than a sentence a person would say, rewrite it.
Good and bad versions of the same question
Examples make this clearer. Here are pairs for a few kinds of business. The first line in each pair is the version that tends to teach you nothing, and the second is the version worth keeping.
- Weak: "Best accountant." Better: "Who is a good accountant for a small landscaping business that just started hiring."
- Weak: "Handmade soap online." Better: "Where can I buy handmade soap without fragrance for sensitive skin."
- Weak: "Acme Plumbing reviews." Better: "Who do people recommend for a plumber in a small town, someone who comes out the same day."
- Weak: "Standing desk." Better: "Which standing desk is best for someone short who works in a small bedroom."
- Weak: "Why Acme Dental is the best." Better: "What should I look for in a dentist for a nervous adult who has not been in years."
Notice what the better versions share. They contain a person, a situation and a doubt. They sound like something said, not typed. And none of them contains the name of the business being tested.
The diagram above sets the whole contrast out as two lists. The point it makes is that a weak set is weak long before any assistant is asked. If your questions are tuned to flatter you, no amount of careful recording will fix it.
How many questions you need
Ten is enough to learn something. Thirty is about as many as a person will keep up by hand. Beyond that, the habit tends to collapse.
Ten questions suits a first look, or a small business with a narrow offer. It will show you a rough pattern.
Fifteen to twenty suits most businesses. It lets you cover several kinds of question and a few of your main products or services.
Twenty-five to thirty suits a business with several distinct offers or several places. Anything more probably needs software or a service, as we discuss in our page on AI visibility tools.
Count the cost before settling on a number. Thirty questions across four assistants is 120 answers to ask and record each time. At a few minutes per question and assistant, that is a full afternoon. If you will not do it monthly, pick a smaller number you will.
A good compromise is a core of ten you always run and a rotating extra ten you add in on a quarterly basis. Just keep the core exactly as it is.
Freeze the set, and why
This is the rule that gets broken most often, and it matters most.
Once the first run is done, do not change the wording of any question. Not to improve it, not to fix a typo that does not change the meaning, not because a rival question would be more natural now. If you change the wording, you have a new question, and your history for that question starts again.
The reason is simple. Assistants are sensitive to wording. A small change in a question can change which names are given. If the question changes at the same time as the answer, you cannot tell which one caused the difference. A trend needs the same measuring instrument every time.
What you can do is add. If you think of a better question, add it at the end as a new item, and keep the old one running beside it. After a quarter or two, you can see whether the new one tells you more.
Write the questions in a document with a date. Treat it like a lab protocol. Our tracking spreadsheet layout has a place for exactly this: the question text sits in one column and does not get edited.
There is a quiet benefit too. A frozen set is a safeguard against your own hopes. If you could edit the questions, you might nudge them toward the ones you do well on.
Two traps to avoid when you write the set
Writing it after you have seen the answers. If you run a few questions, see which ones you do well on, and then build the "real" set, you have built a set that flatters you. Write the questions first. Decide the set before you look at a single result.
Writing questions assistants cannot answer well. Some questions are too narrow, too new, or too private for any assistant to have an answer. If the assistant clearly has nothing, you learn little. A question like "What is the price of the blue model at Acme today" is a lookup, not a recommendation question. Keep to questions where a shortlist or a recommendation is a reasonable answer.
A related issue is the difference between an assistant naming you and an assistant describing you. If you ask about a general topic and you are named, that is visibility. If you ask a question so specific to your business that only you could be the answer, you have written a question that guarantees a pass.
Our page on writing for LLMs covers how the content side works, which is a different problem from testing. LLM means large language model, the kind of system behind these assistants. Keep the testing questions and the content job separate in your head.
Get one other person to read the set
Before you freeze the set, show it to one person who knows your customers but does not work in your business. A salesperson, a front desk worker, a friendly customer.
Ask them three things.
- "Would a customer really say this?"
- "What is a question you hear that is not here?"
- "Is there one here that sounds like we wrote it to look good?"
Expect to change two or three questions. That is the point. The second reader catches the habits that you can no longer hear in your own wording, such as insider terms and flattering framings.
Also check that the set could in principle name someone other than you. If a question can only be answered with your business, you have written a statement of fact, not a test. If a question is so broad that the answer would be a general article, narrow it to a situation.
When the set is ready, date it, save it somewhere safe, and send a copy to anyone who will be helping you run it. If you ever work with a provider, this document is the thing they must agree to be measured on. For what those providers do, see our overview of AEO agencies. AEO means answer engine optimization, the work of getting named in AI answers.
The question set and RunOctopus
We should be clear. RunOctopus does not run a free question set against the assistants for you. The free site read does a different job. You paste your address at the link below, Ollie, our AI, reads your website and shows the plan it would follow, with no account and no call. It does not ask ChatGPT, Perplexity, Claude or Gemini anything.
What RunOctopus does is a done-for-you service. Ollie researches your business and then writes and publishes pages on your own website, with a team that plans, writes, reviews and publishes alongside it. It works with Shopify, WordPress, WooCommerce and Wix, with nothing to install and nothing to approve. Customers who pay also get reports on where ChatGPT, Perplexity, Claude and Gemini name their business.
If you are working on your own question set first, that is a perfectly good way to start, and the set you write here will tell you what content to prioritize in either case.
When you are ready to see what a plan for your site looks like, use the free site read.