Home/Resources/AI Shortlist Plays
AI Shortlist Plays / Measure first

Why does ChatGPT give a different answer every time we ask?

Because each answer is generated afresh, from searches that differ run to run. Treat one answer as a sample. Run each question several times per engine and track the share of runs that name you.

Effort: 1 to 3 hours a month

Who does it: Marketing analyst or coordinator

Best for: B2B firms of 20 to 500 people

Evidence: Strong (SparkToro and Gumshoe.ai, reported 30 January 2026)

01 / WHY IT MATTERS

Why it matters.

Research by SparkToro with Gumshoe.ai, reported by Search Engine Journal on 30 January 2026, ran 2,961 prompts across ChatGPT, Claude and Google’s AI in November and December 2025. Repeated runs of the same prompt returned the same brand list less than 1% of the time, and the same list in the same order less than 0.1% of the time.

The same research found that visibility percentage across many runs was more consistent than position. So a single screenshot, good or bad, tells you little. A count across runs tells you a lot.

02 / SELF-CHECK

The 10-minute self-check.

Take your main commercial question and do this now:

  • Run it three times on ChatGPT in a temporary chat, so memory does not shape the answer.
  • Run it three times on Perplexity, logged out.
  • Write down every company named in each run.
  • Count how many runs named you, and how many names appear in every run.

If the lists differ a lot, you have just seen why one run is not a verdict.

03 / STEPS

Steps.

  1. Decide a fixed number of runs per question per engine. Five is a practical minimum for a small team.
  2. Remove personalisation: temporary or logged-out sessions, no custom instructions, the same browser location each time.
  3. Record the date, engine, mode (search on or off), every name and every source for each run.
  4. Calculate the named rate: runs naming you divided by total runs, per question and engine.
  5. Compare month to month on the named rate. Ignore single-run swings.

This is the counting method behind our AI search visibility reports.

04 / OUR OWN SITE

What we saw on prolinkage.com.

05 / MEASURE

How to measure it.

  • Named rate: per question, per engine. Monthly.
  • Stable competitors: names that appear in more than half of runs. These are the firms the engines are confident about.
  • Run log completeness: every run dated and stored, so anyone can recount.
07 / SOURCES

Sources.

AI Shortlist Plays

The self-check shows what you can see yourself.

What the engines say across repeated runs, which sources they read and who they name instead is harder to see from inside. Our free AI Visibility Snapshot runs a sample of your buyers’ questions and is reviewed by a person.

More than a decade of experience, across businesses and markets.