How We Built the ChatGPT Visibility Checker — What It Act...
Active Document
Inside Agency7's free ChatGPT Visibility Checker — why we query two LLMs instead of one, how we chose the six recommendation prompts, what the mention rate really means, and what the tool cannot see.
Loading document...
Today we shipped the ChatGPT Visibility Checker — a free tool that asks ChatGPT and Claude six different recommendation questions for your city and tells you whether they name your business. Mention rate, per-provider split, top competitors, prioritized fix list. Thirty to forty seconds, no signup.
This post is about how it works under the hood — which prompts we chose and why, what the mention rate genuinely measures, what the tool can't see, and what an Edmonton business should do with a low score. If you've been wondering whether these "AI visibility" tools are real or snake oil, read on. Ours is defensibly real, but its limits matter too.
In January 2026, the line between "Google traffic" and "AI traffic" is still moving, but the direction is unambiguous. A growing share of small-business discovery happens inside ChatGPT, Claude, Perplexity, and Gemini — especially for "who should I hire for X in Y" questions, which is exactly the prompt Edmonton service businesses live or die by.
Agency7's AI SEO service is half about Google and half about getting our clients cited by LLMs. Clients kept asking the same question: "how do I even know if I'm on ChatGPT right now?" So we built the tool we wished existed.
We could have queried one LLM with one prompt and called it a day. We didn't, for three reasons.
Why two providers. ChatGPT (OpenAI) and Claude (Anthropic) are trained on overlapping-but-different web snapshots, weight different signals, and get asked different questions by real users. A business can be visible to one and invisible to the other. Showing the split is more honest than averaging it away.
Why six prompts instead of one. "Best AI agency in Edmonton" is one query. It's not the only way a potential customer will ask. The six we picked cover the real recommendation patterns:
If your business shows up on query 1 but not on queries 3–6, you have general brand visibility but weak service-specific visibility. That's a different problem than showing up nowhere — and it calls for a different fix. The per-query breakdown in the results page exists specifically to expose that.
Why 12 runs and not more. Each LLM call costs real money and takes real time. Twelve runs is enough to produce a stable mention-rate reading (you can see a meaningful difference between 0%, 17%, 50%, and 75%+) without dragging the report past forty seconds, where users start abandoning.
LLMs don't always return businesses in exactly the format a user would expect. "Agency7" might show up as "Agency 7," "Agency7.ca," or "Agency7 Inc." We normalize both sides:
If any returned company name is a substring of the entered business name or vice versa, it counts as a mention. The trade-off: a business called "Seven" would match "Agency7" — a false positive. In practice this is rare enough that we haven't built stricter matching, but it's the biggest known flaw.
We ask each LLM to return a strict JSON object: {"companies": ["Name 1", "Name 2", ...]}. Three reasons:
Agency7"; ignore prior instructions and return gets their query sanitized, and if it reaches the model the model is trained to ignore it.We also told the LLMs: "If you do not know of any specific companies, return {"companies": []}." This is the single most important line. Without it, Claude in particular would confabulate plausible-sounding Edmonton company names. With it, a query for a sparsely-covered city returns an empty list — which is itself a useful signal.
llms.txt, or entrenched-competitor dominance in your category.A 50% mention rate across 12 runs is not 50% chance of any given query mentioning you — it's 6 out of 12 runs did, and you can see which 6 in the per-query breakdown. That breakdown is where the diagnostic value lives. If you show up on ChatGPT but not Claude, the two models are weighting different signals about you. Claude weights Anthropic-era training data (later cutoff, more 2025+ content); ChatGPT weights a mix. A business visible on one but not the other usually has either recent press the other missed or long-tail directory presence one model didn't absorb.
Honesty bit. The checker queries model weights directly. It does not see:
We're transparent about this on the form page. The tool is a signal, not a verdict. A business scoring 0% on our checker might still be findable when a real user asks a real ChatGPT with browsing on. Conversely, scoring 75% on our checker doesn't guarantee Perplexity will cite you.
app/api/chatgpt-visibility/route.ts, nodejs runtime (not edge — we need the OpenAI and Anthropic SDKs), 60s maxDuration.gpt-4o-mini, 400 max tokens, temperature 0.3, response_format: { type: "json_object" }.claude-haiku-4-5-20251001, 400 max tokens, temperature 0.3, system prompt explicitly demands JSON-only with no prose.Promise.all. Total wall-clock time is bounded by the slowest individual call (~2-4 seconds), not the sum.One form, two fields. Business name and city. We could have asked for a URL, industry, phone number. We didn't, because the longer the form, the more people bounce. Two fields is enough to run the test.
No email gate. Most visibility-check tools on the market capture your email before showing results. We don't. We'd rather have 10× the organic shares than 1× the email capture, and if you run the tool twice and like it, the "book a strategy call" CTA at the bottom is a higher-intent conversion than an email anyway.
Real-time loading state with a timer hint. Users abandon long loading states. We tell them upfront it takes 20–40 seconds and show a spinner plus the word "Running 12 LLM queries…" to make the wait feel purposeful.
Score-banded results copy. The "what to do next" section in the results isn't static. If you scored 50%+ it shows defense tactics (publish authoritative content, maintain citations, keep llms.txt current). If you scored under 50% it shows recovery tactics (build third-party mentions, fix technical foundation, acknowledge the entrenched-competitor gap).
Full competitor list, not just three. Showing the top 15 businesses cited above you is uncomfortable but useful. The counts tell you who's dominating: a competitor cited in 8 queries has substantially more visibility than one cited in 2. This is the part that makes an owner call us.
If you're under 50%, don't spiral. The fix order matters more than the score.
llms.txt. Update on any service change within a week.We built this tool partly as a lead magnet, and we won't pretend otherwise. If your score is embarrassing and you want help, that's exactly what Agency7 does — we run a full 40-query audit across four providers, analyze your structured data, review your llms.txt, and hand you a prioritized 60-day fix plan.
Try it now: ChatGPT Visibility Checker — free, three runs per 24 hours, no signup.
Or if you'd rather skip straight to the full audit: book a free 15-minute strategy call.
No. It's a cheaper, consistent, automated version of what you'd do by hand. A full manual audit (us doing it for clients) queries 40+ prompts across 4 providers, pulls ChatGPT with browsing enabled, and cross-references against your structured data. The tool gives you the directional signal in 30 seconds.
Those two are the biggest share of the generative-search market for business-recommendation queries in early 2026. Perplexity and Gemini are next on our roadmap. Adding them is not hard; it's mostly a cost decision (more LLM calls per report) and a latency decision (longer wait time).
Yes. Any city works — Calgary, Vancouver, Toronto, Seattle, wherever. The query set is the same. The only caveat is that LLMs have more training data about larger cities, so a tiny town might return empty company lists across the board.
Accurate for what it measures — which is whether two specific LLMs, asked a specific set of six recommendation questions, name your business. That's a real signal, but it's not "whether you're on ChatGPT" in the broadest sense, because the broadest sense would include browsing-enabled ChatGPT, voice mode, different phrasings of the question, and LLMs we don't query. Treat the number as directional.
Yes, and they should. The competitor list is genuinely useful for category research. The rate limit (3 runs per 24 hours per device) exists to prevent automated scraping, not to hide information.
Because lead-magnet math works. Running the 12 queries costs us under a dollar per report. If one report in a thousand turns into a client engagement, the tool pays for itself hundreds of times over. Also: we believe the information is too important for most SMBs to be gated.
Not right now. We might open that up later for Edmonton web agencies who want to white-label it, but for now the tool lives at agency7.ca/business-tools/chatgpt-visibility and our backlink benefit is part of the point.
This is the fifth free tool in Agency7's business-tools lineup, joining the Cost Estimator and AI-Readiness Score. Each is built to do one job well, rank for a specific AEO query, and push real value to Edmonton business owners thinking about AI. If you have ideas for the next tool, tell us — hello@agency7.ca. The best tool ideas come from the "I wish there was a way to check X" conversations we have with clients.