How we measure whether AI search is actually working for you
AI search is new enough that a lot of it gets sold on vibes. So it's fair to ask the blunt question: how do you know it's working, and how do you know you're not paying for a metric that means nothing? Here's exactly what we track for a South Boston business, and what we deliberately ignore.
The measurement that matters most is simple and unglamorous — we ask the tools the questions your customers ask. On a regular schedule we prompt ChatGPT, Google's AI Overviews, and the assistants people use here with real queries: "who's the best HVAC company near South Boston?", "where should I eat downtown near The Prizery?", "reliable tree service in Halifax County." Then we record whether your business shows up, how it's described, and who's mentioned alongside you. That's the scoreboard. Everything else is supporting evidence.
The second thing we watch is accuracy, because being mentioned wrong is its own problem. If an AI names your business but lists old hours, the wrong service area, or a competitor's phone number, that's a citation issue to run down — usually stale data somewhere on the web the model is pulling from. We track not just presence but whether the details it repeats about you are correct, since a confident wrong answer costs you the same customer a missing answer does.
Third, we look at whether AI-driven visitors actually arrive and do something. Some AI tools now link out. When they do, we can see that traffic land on your site and whether those visitors call, fill the form, or ask for directions. A mention that never turns into a Halifax County customer walking in the door is a start, not a win, and we treat it that way.
Now the vanity metrics we ignore. We don't celebrate a single lucky mention that we can't reproduce — if you appear once and never again on the same question, that's noise, not a ranking. We don't chase national queries a South Boston business will never realistically win or benefit from. We don't count "impressions" on AI features you can't tie to a real person near here. And we don't confuse being mentioned in a long list of ten businesses with being the one the tool actually recommends — position and framing matter, and "you're option seven of ten" is a problem to fix, not a result to report.
- Tracked — do the AI tools name you for real local questions, consistently
- Tracked — are the details they repeat about you correct
- Tracked — does AI-referred traffic turn into calls, forms, or visits
- Ignored — one-off mentions, national queries, unattributable impressions, being buried in a list
Because so few Southside businesses are even in this race yet, the honest measure of success is narrow and answerable — over a month or two, are you showing up more often, more accurately, and higher in the answer than when we started. That's the report you'll get, in plain language, and it's what the AI Search audit checks before we begin so there's a real baseline to measure against.