Why case studies mislead, and why they still matter
A case study is a sales document that looks like a report. That does not make it worthless. It makes it a claim to be tested, and the testing is quick once you know the questions.
Case studies are the closest thing the buyer of SEO services has to evidence. You cannot try the service first, and you cannot see the work until you have paid for it. So the case study has to carry weight, and that gives the seller every reason to choose the most flattering numbers.
None of this requires dishonesty. A real result from a real client can still mislead if the number is chosen carefully, the base is hidden or the client was nothing like you. The skill is in reading what is shown and noticing what is left out.
If you are at the stage of evaluating providers, our guide on what a consultant should show you before you sign places case studies alongside the other evidence worth asking for, such as sample work and a plain description of the process.
Start with the metric
The first question is what is being measured. Not all numbers are equal.
Clicks from Google are the strongest simple measure. They are people who arrived at the site because of a search result, and they come straight from Google Search Console, which is free and which you can check on your own store.
Impressions count how often a page appeared in results, whether or not anyone clicked. They can jump for reasons that bring no visitors at all, for example when a page appears for obscure searches. Impressions alone are the most common way a modest result is made to look large.
Revenue is the number you care about, and the hardest to prove, because it depends on attribution and on everything else that happened that month. A case study that shows revenue should say how it was attributed.
Visibility scores are invented by tools. They are useful for tracking trends inside the tool, and are not an outcome for your business.
When a case study leads with a percentage, ask which of these it is and what it started from. Our page on ecommerce SEO reporting shows what a trustworthy report holds, and the same standard applies to a case study.
The starting point and the time period
A percentage is meaningless without its base. A store that grows from 10 clicks a month to 50 has grown by 400 percent. A store that grows from 10,000 to 12,000 has grown by 20 percent and gained far more visitors. Always ask for the raw numbers before and after.
The time period matters in two ways.
How long did it take? Results that took two years tell you something different from results that took three months. Be skeptical of any case study that does not state the period.
Was the period chosen to flatter? A comparison that begins at the lowest point, for example just after a site migration, will always look like a recovery. Seasonal stores make this worse. Comparing a December to a February tells you about the calendar. Honest comparisons use the same period a year apart, or a clearly stated window.
One more check: did anything else change? A store that launched a major sale, got a press mention or changed platforms in the same window may owe the result to something besides SEO. A good case study acknowledges this.
The six questions together
Put together, the checks become a short routine you can run on any case study in a few minutes. The diagram below lists them in order, with the trap that each one is designed to catch.
Do not treat a miss on one question as proof of bad faith. Many honest providers publish case studies that fail one or two checks because they were written quickly. Treat it as a reason to ask. A provider who answers the questions plainly, with numbers, is showing you how they will report to you later.
A provider who gets defensive about the base, the period or the source is also showing you something.
Does the business resemble yours?
A result is evidence about stores like the one it came from. A case study from a business with a huge brand, a large budget or a different model tells you little about yours.
Check the things that drive search results. Is it a store with a product catalog, or a service business? How big is the catalog? Was the brand already well known? Was the site already getting traffic, or starting from zero? Who is the customer, a consumer or a business buyer?
The most useful label a case study can carry is the business type, stated honestly. When a provider shows you results from a few kinds of business, notice which of them match yours. Strong results from three other kinds of business are not weak evidence, but they are evidence about those kinds.
Read-across has limits. A store selling something very competitive, like mass-market electronics, faces different odds than one selling a niche product that few sites cover well. A fair case study mentions this.
Our own results, labeled the way we are asking you to read them
Here are three real results from RunOctopus clients, taken from Google Search Console for the 28 days to September 29, 2026. We label each by business type, because that is the standard we suggest you hold everyone to, including us.
- Real estate business, Los Angeles (Sterman): from 44 to 210 Google clicks a month since April.
- Real estate careers site, Florida (Florida Realtor Careers): from 8 to 116 Google clicks a month since launching in June.
- Reptile supply store (All Angles Creatures, where RunOctopus started): from 0 to 4,500+ Google clicks a month in five months.
Read these as you would any case study. The metric is Google clicks, from Search Console. The bases are stated. The periods are stated. The business types are labeled, and notice that only one of the three is an online store. The first two are service and lead-generation sites, which is why we are not presenting them as ecommerce results.
They also do not tell you what will happen to your store. They tell you what happened to these three businesses, in these windows. For a view of what the search results are worth in money, see our page on ecommerce SEO ROI.
Look for the work, not the slogan
The most useful part of a case study is often the shortest: what did they actually do?
A good account names the work in plain terms. "Rewrote 40 category pages to answer the questions buyers ask before choosing a size" is specific. "Implemented a comprehensive SEO strategy" is a slogan. If you cannot tell from the text what was changed on the site, you cannot judge whether it would apply to your store.
Look for these in particular.
- Which pages changed: products, categories, new articles, technical fixes?
- How many? A few pages or hundreds?
- What did the new pages say that the old ones did not?
- Was anything built that other sites might link to?
Detail is evidence of effort. It is also what lets you ask a provider sharper questions later, such as those on our page of questions to ask an ecommerce SEO agency.
Verify what you can
Three checks cost nothing.
Visit the site. If the client is named, open the site, search for some of its products and see whether it appears. If the client is not named, ask whether you can speak to them or see a screenshot of Search Console with the property name visible.
Look at the pages. Are they better than the ones around them in search results? Would you want to read them?
Ask for the reporting. A provider who can show a real monthly report from a real client is showing you what you would receive. Compare it against the standard in the red flags page, ecommerce SEO red flags.
Be aware that some clients prefer not to be named. That is normal. A provider can protect privacy and still describe the business type, the numbers and the work.
A case study is a claim with some evidence attached. Check the metric, the base, the period, the business type, the work and the source, and you will know how far to trust it.
What case studies leave out
Every published case study is a survivor. Providers choose to show their best results, and nobody publishes the client who left after four months with nothing to show for it. That is a normal feature of marketing, and it is worth keeping in mind.
You can compensate with a few questions. Ask what share of clients see results like the ones shown. Ask what the typical result is, not the best one. Ask about a client where it did not go well, and what happened. A confident provider will answer, and the answer is often more informative than any of the published stories.
Notice, too, how many case studies are shown. One dramatic story is less convincing than several modest ones across different kinds of business. A provider with many clients and few published results is either careful about privacy or hiding the average.
Finally, watch for stories that bury the cause. If a store doubled its traffic in a year and the account credits a single clever tactic, be doubtful. Results in search come from many pieces of work compounding over months, and a case study that explains the whole thing in one sentence has simplified it.
Revenue claims and attribution
When a case study quotes revenue from organic search, it is stating something that is genuinely hard to measure, and it is worth knowing why.
Analytics tools credit a sale to the channel of the last visit, or sometimes the first, depending on settings. A shopper who found you in a search, left, came back through an email a week later and then bought may be counted for email. Another shopper who read a guide, asked an AI assistant about your brand and then typed your name into the address bar is counted as direct traffic. Neither shows up as search revenue, though search started both.
So revenue figures are estimates, and the useful question is how the estimate was made. Ask which analytics tool, which attribution setting and which date range. A provider who explains the limits is being straight with you. A provider who states a precise revenue figure with no explanation is not showing you how it was reached.
This is also why clicks from Search Console are a better base for a first check. They are not revenue, and they are not affected by attribution settings.
Questions to ask the provider about their own case studies
When a provider sends you a case study, a short set of follow-ups shows quickly how much is behind it.
- What were the monthly clicks before and after, and from which source?
- How long had the client been working with you at each data point?
- What work was done in the first month, the third month and the sixth?
- Did anything else change in that period, such as a new site, a sale or a press mention?
- How many of your clients in this business type saw a similar result?
- Can I see the report the client received?
None of these is hostile. They are what a careful buyer asks of any supplier. The answers also give you a preview of what a working relationship would feel like: whether questions get direct answers, whether numbers are specific, and whether the provider is comfortable being checked.
How RunOctopus handles this
RunOctopus is a done-for-you service. Its AI, Ollie, researches a business, then writes and publishes pages directly on its own website, with a team planning, writing, reviewing and publishing alongside Ollie. It works with Shopify, WordPress, WooCommerce and Wix, and the results above came from that work.
You install nothing, approve nothing, upload nothing and schedule nothing. Pages go live on their own. Week one brings 8 long-form articles, 6 pages and a custom interactive tool built for your niche, then 175+ pieces of content a month, written, published and linked. Reports show where ChatGPT, Perplexity, Claude and Gemini name your business.
The 90-day promise is the claim we ask you to hold us to: more visibility on Google and in AI answers within 90 days, measured against the 90 days before you joined, or every dollar back. Start with the free site read. Paste your address and Ollie reads the site and shows the plan. No account, no call.