How we measure whether AI search is actually working
AI search is new enough that a lot of people selling it lean on numbers that sound impressive and mean nothing. Because almost no Marion business is tracking this yet, it's worth being blunt about what we watch and what we deliberately ignore — so you can tell real progress from a dressed-up report.
The measurement that matters is straightforward: does the AI name you when a real customer asks a real question? So the core of the work is repeatedly asking the tools the questions your Smyth County customers actually ask — the kind about where to eat, who to call for a specific repair, where to stay near town — and recording whether your business shows up, in what position, and how accurately it's described. We test across the tools that matter, because ChatGPT, Google's AI answers, and phone assistants each build their responses differently, and being named in one doesn't guarantee the others.
Getting mentioned isn't the finish line — being described correctly is. An AI that recommends you but lists the wrong hours, the wrong service area, or a service you don't offer is costing you the customer at the door. So a second thing we track is accuracy: what the AI says about you versus what's true. Closing those gaps is often where the fastest wins come from, because they're usually caused by stale or contradictory information the model pulled from somewhere on the web.
We also watch whether AI crawlers are reaching your site at all — the server logs and crawl signals that show the systems feeding these tools can find and read your pages. If they can't get in or can't parse what you do, nothing downstream works, and that's a fixable technical problem rather than a mystery.
Here's what we don't put in a report, because it looks like progress and isn't:
- Raw "AI traffic" counts — a big number that doesn't tell you whether the visits were customers or bots, and easy to inflate.
- Being mentioned for questions no one in Marion asks — showing up for an obscure phrasing proves nothing about the searches that bring real calls.
- One-time screenshots — AI answers shift week to week; a single lucky mention isn't a ranking, it's a snapshot, and we track the pattern over time instead.
- Vague "visibility scores" from third-party dashboards that can't be traced back to a real question and a real answer.
The honest version of this work is a short list of the questions that matter to your trade, tracked consistently, with a clear before-and-after on whether you're named and named correctly. That's the reporting you get. The main AI Search page explains the underlying approach.