How We Measure Whether AI Search Is Actually Working
AI search is new enough that a lot of the reporting around it is theater. Someone shows you a screenshot of ChatGPT mentioning your business once and calls it a win. That's not measurement — that's a lucky moment on a good day. If you're a Vinton business trying to get named when someone asks an AI assistant for a recommendation in the Roanoke Valley, you deserve to know what's genuinely moving and what's noise. Here's how we actually track it.
The core question we answer is simple: when a real person asks an AI tool a question that should surface your business, do you get named, and are the facts right? So we test the way your customers actually ask. Not "best plumber in Virginia" — that's too broad to be useful. We run the specific, local, buying-intent prompts: "who does emergency HVAC repair near Vinton," "reliable roofer in the Roanoke Valley," "where can I get X in Vinton VA." We run those across the major assistants, repeatedly, over time, because a single answer isn't a trend — a pattern of answers is.
From there, three things matter. First, presence — are you named at all when you should be? Second, accuracy — when you're named, is your phone number, service area, and specialty correct, or is the AI hallucinating stale details that will send a customer to a dead number? Third, framing — are you described the way you'd want a prospect to hear you, or as an afterthought? We log all three and watch the direction they move month over month. Improvement in that pattern is the real signal.
We also watch a quieter but honest metric: referral traffic and leads that trace back to AI-driven discovery. It's harder to attribute than a Google click, but as assistants start citing sources and driving visits, that trickle is measurable — and it's a customer who came to you already half-convinced.
- The vanity metrics we ignore: one-off mentions we can't reproduce, "AI visibility scores" from tools that don't disclose their method, and raw prompt counts that say nothing about whether a customer ever saw the answer or acted on it.
The reason accuracy sits at the center of all this is that AI assistants pull from the same foundation that powers local search — your website, your business profile, consistent information across the web. When those are clean and structured, you get named correctly more often. When they're a mess, the AI guesses, and its guesses aren't in your favor. That's why AI search work and solid local SEO reinforce each other rather than competing for budget. If you want to see how you're currently showing up when valley customers ask an assistant for a recommendation, get in touch and we'll run the real prompts and show you the actual results.