Most AI-visibility dashboards tell you whether ChatGPT mentioned your brand. They don’t tell you why — and the why is where your budget decision lives. This playbook shows you how to measure AI visibility in ChatGPT by reading the model’s own network traffic, so you can separate the one metric that predicts inclusion from the dozen vanity metrics that don’t. It is a build, not a think-piece: nine steps, effort estimates, failure modes, and a clear go/no-go on whether to buy a vendor instead.
The method comes from work by search practitioner Suganthan Mohanadasan, who read the raw HTTP responses behind 60 ChatGPT conversations in July 2026 and found that the assistant writes brand names into its own search query before it fetches a single page. Brands named in that first query reached the final answer 68.9% of the time; brands merely retrieved from the web reached it 2.1% of the time — a roughly 33x gap. The decision is made largely before your server is ever contacted.
Why measure AI visibility in ChatGPT at the query, not the answer
When a user asks ChatGPT “best [category],” the model rewrites that into its own search strings, runs them, reads the results, and writes an answer. Those strings sit in the JSON your browser downloads to render the page, under a key currently named search_queries (OpenAI renamed it from search_model_queries in early August 2026). Reading them is not a leak — your own browser needs that data — and it is the closest thing to a ground-truth signal we have for how ChatGPT sees your category.
That matters because the two things practitioners lump together as “AI visibility” are actually two different games with different owners and budgets:
- Being in the category vocabulary — does ChatGPT name your brand in its first query, unprompted? This is set by years of being written about, reviewed, and compared across the open web. Schema,
llms.txt, and page speed do nothing here because your server is never contacted. - Winning the citation once you’re in — of the ~600 pages ChatGPT reads to write one answer, only about 3.1% get cited. Position within the domain group is the strongest predictor: 1st-ranked pages cite at 5.2%, 6th-or-later at 0.3%.
If you measure only the final answer, you can’t tell which game you’re losing — and you’ll spend on technical GEO work when the problem is that ChatGPT has never heard of you. Measuring at the query fixes that.
The 9-step ChatGPT AI visibility test harness
Build this once and re-run it monthly. A marketing data scientist can stand up the manual version in an afternoon; the automated version is a few days of work.
- Define your query set (effort: 2–3 hrs). List the 10–20 buyer questions that should surface your brand: “best [category] 2026,” recommendation phrasings (“what should I use to…”), complaint phrasings (“our [job] is a mess”), and competitor-displacement phrasings (“alternatives to [rival]”). Include the shapes that don’t trigger a web search (how-to, definitional) as controls — if ChatGPT answers from training with no search, there is no shortlist to join.
- Capture the query stream manually first (effort: 30 min). In Chrome, open DevTools → Network, ask one query, filter for
conversation, open the response, and search forqueries. Read the first string the model wrote and look for names you never typed. Do this once by hand so you understand the payload before you automate it. - Run each query five times (effort: 1–2 hrs). The shortlist is not stable. In repeat runs, one category kept five of six names; another collapsed from six vendors to a single probe. Names that appear in all five runs are your true competitive set in ChatGPT’s head; names that flicker are contested ground where a push can move the needle.
- Parse and log the first query per conversation (effort: 1 day to automate). Script the capture (a browser extension, a Playwright session, or a proxy that logs
search_queries). For each conversation, record: the user prompt, the first search string, the brands named in it, and a timestamp. The first query — before any fetch — is the clean signal; anything after retrieval is contaminated by what came back. - Classify each brand into two buckets (effort: half day). For every run, tag each brand as named in the query or only retrieved. This is the split that produced the 68.9% vs 2.1% finding. Do brand-string matching carefully — naive tokenizers split “Car and Driver” into two tokens and let junk like “SaaS” or a city name through as a brand.
- Compute your two core metrics (effort: 2 hrs). (a) Query-presence rate: across all runs, how often does your brand appear in ChatGPT’s first query? (b) Named-to-mentioned rate: when named, how often do you reach the final answer? Track the same two for your top competitors. These two numbers, not “share of voice,” are your dashboard.
- For present brands, measure conversion not retrieval (effort: half day). Build a labelled table of retrieved pages with a cited/not-cited flag. Count how often your pages get fetched versus credited. In the source data one brand was fetched 66 times and cited zero — a page problem, not a visibility problem. Fetched-a-lot, cited-rarely is the diagnosis you want to surface.
- Flag personalization contamination (effort: 2 hrs). Results are account- and location-specific — a Dubai account pulled in UAE-local vendors unprompted. Run from a clean or representative account, note the geo, and label any query whose shortlist looks locale-driven. Never present a single account’s shortlist as “what ChatGPT thinks.”
- Package it as a monthly monitor (effort: 1 day). Wire the two core metrics plus the fetched-vs-cited column into a scheduled job and a simple table. Re-run monthly: the format itself moves (the fan-out dropped from ~12 searches per answer to ~4 after the August rename), so a harness you don’t re-run silently rots.
Common failure modes and how to avoid them
Treating percentages as precise. Every rate above comes from one account and a few hundred conversations. Report query-presence as a direction (“named in 4 of 5 runs”), not a decimal. The reproducible mechanism — that ChatGPT injects brands pre-fetch — is solid; the exact numbers are directional.
Measuring one answer and calling it a trend. The shortlist changes between identical runs. One capture is noise; five runs is a signal. Bake the repeat into the harness, not into an analyst’s memory.
Spending on technical GEO when you’re absent from the query. If your brand never appears in five runs, no amount of schema or page work will move you — that spend belongs in digital PR, review-site placement, and category content that builds the training-data association. Match budget to the column you’re losing.
Ignoring the citation filter once you’re in. Being retrieved 9th in a domain group is a rounding error (0.3% cite rate). Consolidate near-duplicate pages — two tightly matched pages is the sweet spot (6.2%); past six pages from one domain, per-page conversion collapses.
Letting the parser lie to you. Brand-matching errors quietly inflate your counts. When in doubt, read the raw query strings — they are the evidence — and treat the aggregated counts as approximate.
Decision criteria: build this harness, or buy a vendor?
Build it if you have a data scientist who can script a capture, you want ground-truth query data you fully control, and you need to defend a budget reallocation with your own numbers. The manual check is free and takes two minutes per query.
Buy instead if you need multi-account, multi-geo coverage to rule out personalization (the single hardest gap in the DIY method), or you want lift measurement rather than presence measurement — for that, look at panel-based products such as DISQO’s AI Search Lift, which uses consented panels and exposed-vs-control testing to estimate campaign impact inside LLM search. And before you compare any two vendors’ numbers, map them to the IAB’s 2026 “Measuring Visibility in the AI Era” standard, which defines a shared metric vocabulary and two data-quality tiers — without it, “AI share of voice” numbers are not comparable across vendors.
Do both if you’re serious: run the harness for pre-retrieval presence (the 68.9% game), and use a panel vendor for downstream lift.
Internal linking suggestions
When this publishes, link it to Matt’s related playbooks: “How to Stand Up AI Visibility Measurement to the IAB Standard: A 10-Step Setup Playbook” (the governance layer this harness feeds into), “How to Pilot an MCP Server for Marketing Data: A 9-Step Plan with Effort Estimates” (same capture-the-stream ethos applied to behavioral analytics), “The Marketing KPI Governance Playbook” (so the two core metrics here don’t become another vanity KPI), and “First-Party Data Activation: A 9-Step Playbook” (the downstream side of owning your own measurement signal).
FAQ
How do I check if ChatGPT knows my brand?
Ask ChatGPT your “best [category] 2026” question in Chrome with DevTools open, filter the Network tab for conversation, and read the first search_queries string. If competitors appear there and you don’t, ChatGPT doesn’t associate your brand with the category — a positioning and PR problem, not a technical one. Run it five times, because the shortlist shifts between runs.
What is the difference between being named in ChatGPT’s query and being retrieved?
Named brands come from the model’s memory and are written into its search string before any page is fetched; retrieved brands are pulled from web results during the search. In the source study, named brands reached the answer 68.9% of the time versus 2.1% for retrieved-only — about 33x. Being named is the entry ticket; a separate, harsher filter then decides citation.
Can you measure AI visibility in ChatGPT without a vendor tool?
Yes. The core signal — whether ChatGPT names your brand in its own query — is readable in your browser’s DevTools in about two minutes per query, and scriptable into a monthly monitor in a few days. You only need a vendor for multi-account/geo coverage (to control for personalization) or for panel-based lift measurement.
How often should I re-run an AI visibility audit?
Monthly. ChatGPT’s shortlist changes between runs and its retrieval format changes between releases — the fan-out dropped from ~12 searches per answer to ~4 after an August 2026 rename. A harness you run once and forget will report stale numbers within weeks.
Sources: Suganthan Mohanadasan, “ChatGPT Already Knows Who’s In The Running Before It Searches” (Aug 2026); PPC Land, “Brands named in ChatGPT’s own query win mentions 33x more often”; Search Engine Journal, “ChatGPT Already Knows Who It’ll Recommend Before It Searches”; PPC Land, “IAB sets AI visibility measurement standard”; MarTech, “DISQO launches AI Search Lift”.
