How we actually measure AI search — and the numbers we ignore
AI search is new enough that a lot of people selling it lean on numbers that look impressive and mean nothing. Because you can't open a dashboard and watch your "AI ranking" tick up, it's easy to get sold a story instead of a result. So here's exactly how I measure whether this is working for a Big Stone Gap business — and, just as important, the vanity metrics I refuse to bill you for.
The measurement that matters is simple and testable: when a real person asks an AI assistant the questions your customers ask, does it name you? I run the actual prompts — "who does this in Big Stone Gap," "best option near Wise County," "who should I call for this in the coalfields" — across ChatGPT, Perplexity, and Google's AI answers, and I record whether your business shows up, how it's described, and whether the facts are right. That's the baseline. Then we do the structural work, and I run the same prompts again over the following weeks and watch the answer change.
Around that core test, I track a handful of things that genuinely move the needle:
- Whether AI names you at all for your category and town — presence versus absence, checked on the same set of real prompts over time.
Whether it describes you accurately — right services, right service area, right phone — because AI naming you with wrong facts costs you the call.
Whether your key facts are actually machine-readable — schema in place, entity details consistent across the web — since that's the input AI is drawing from.
Referral and direct traffic that shows up without a matching Google search, which is often the fingerprint of someone who found you through an AI answer.
Now the numbers I ignore. Nobody can hand you a trustworthy "you rank #1 in ChatGPT" figure — these tools don't publish rankings, and any tool claiming a precise AI position is guessing dressed up as data. I don't count raw "impressions" from AI-visibility software as a win on their own, because being crawled isn't the same as being recommended. And I don't chase how often your name appears in AI writing across the internet as if volume equals authority — a hundred thin mentions lose to being the one clear, correct answer to the question that ends in a sale.
The honest limit is this: you can't dictate what an AI says, so I don't promise a number I can't control. What I can show you is a before-and-after on the real questions your customers ask, plus proof the underlying facts about you are now structured and consistent. In an underserved market like the coalfields, where these tools currently have thin information to work from, moving from "not mentioned" to "named as the answer" is a concrete, checkable win — and it's the only kind of AI-search result worth paying for. If you want to see your own baseline, I'll run the prompts and show you exactly where you stand today.