How We Measure Whether AI Search Is Actually Working
AI search is new enough that a lot of people selling it lean on numbers that look impressive and mean nothing. Because there is no tidy "AI ranking report" the way there is for Google, it is easy to hide behind fuzzy metrics. So before we start, here is exactly how I judge whether the work is paying off for your Floyd business — and which numbers I refuse to celebrate.
The measurement that matters most is direct and low-tech: we actually ask the AI tools your customers use. On a set schedule, I run the real questions a visitor would type — "plan a weekend in Floyd, Virginia," "best places to stay near the Blue Ridge Parkway," "where to eat in Floyd," "tree service near Floyd" — through ChatGPT, Google's AI Overviews, and the other assistants, and I record whether your business gets named, how it is described, and who gets named alongside you. Moving from "not mentioned" to "recommended by name" is the whole game, and it is trackable.
The second thing I watch is accuracy, because being mentioned wrong is its own problem. I track whether the AI has your hours right, your services right, and your description right — and whether it has stopped repeating stale or flat-out incorrect details. For a discovery-driven town like Floyd, an AI confidently telling a traveler you close at 4 when you are open until 8 costs you a real customer, so correcting and holding that accuracy is a measured outcome, not an afterthought.
Third, I look at the traffic and leads that behave like AI referrals: visits landing directly on the right page with no search query attached, a rise in people who arrive already knowing your name and what you offer, and — the number that actually pays the bills — calls and bookings from people who say some version of "the AI recommended you." I set up your analytics so this is visible instead of vanishing into an "unknown" bucket the way it does on most sites.
Here is what I deliberately ignore, because it flatters everyone and helps no one:
- Raw "AI visibility scores" from tools that grade you against nobody real in your market
- Impression counts and reach numbers with no path to a call, booking, or foot traffic
- Getting mentioned for searches no actual Floyd customer would ever type
- One-time screenshots of a good answer, with no tracking of whether it holds up next month
The standard is simple: are the AI tools naming your Floyd business, describing it correctly, and sending you people who become customers — and is that improving over time? I report on those three things in plain language every month. If a metric cannot connect to a customer, it does not go in your report. Ask for the written plan and I will show you the exact questions we will be tracking for you.