How to Audit Your Website’s AI Search Presence in 3 Simple Steps

Most attempts to measure AI search presence stop at the interview stage. You ask the assistants about your company, record what they say, and end up with a scorecard describing an outcome. That is worth doing, and it leaves the most useful question unanswered: whether the machines could reach your site in the first place, what they received when they tried, and whether anything came back.

This audit runs from the other direction. Everything in it comes from systems you already own, produces evidence rather than impressions, and can be completed in an afternoon by anyone with log access. Three steps, in the order the dependencies fall.

Why the outcome does not diagnose the cause

A gap in AI search presence has at least four unrelated causes: the crawler never arrived, it arrived and was refused, it was served something empty, or everything worked and the content simply was not good enough to quote. Those failures look identical from the answer side and demand completely different fixes.

Getting this wrong is expensive in a specific way. Teams that assume a content problem spend two quarters rewriting pages while a firewall rule quietly refuses every request, and nothing improves, because nothing that was rewritten was ever fetched. Establishing your actual AI search presence at the infrastructure level first is what prevents that, and it is why access evidence comes before any content recommendation in the Generative Engine Optimization work we deliver.

Step one: verify who actually reached you

The whole point of an AI crawler audit is that this step produces evidence rather than impressions. Open your server logs, or your CDN’s traffic analytics if the origin sits behind one, and filter the last thirty days for the AI user agents that matter. That gives you a first list. Do not trust it.

A user agent string is self-reported text that any client can set to anything. This is not a theoretical concern in this field. In August 2025 Cloudflare publicly accused Perplexity of using stealth, undeclared crawlers to evade no-crawl directives, and subsequently delisted it as a verified bot. Whatever position you take on that dispute, the operational lesson is not in question: the name in the log is a claim, not an identity.

A user agent string is what a visitor says it is. An IP address is what the network can prove. Any audit that treats the first as evidence is measuring a claim.

There are two documented ways to verify AI crawlers, and Google publishes both for its own crawlers in a form that transfers cleanly to any bot with published infrastructure. The manual method is a reverse DNS lookup on the accessing IP address, confirming the returned hostname belongs to the expected domain, followed by a forward lookup on that hostname to confirm it resolves back to the same IP. Both directions must agree. The automated method is matching the accessing IP against published lists of address ranges in CIDR format, which the major operators maintain as JSON files.

The output of step one is a single sentence you can defend: these named crawlers, verified by network evidence, fetched this many distinct URLs in the last thirty days. Most teams have never written that sentence, and writing it usually settles an argument that has been running for months, which is why it sits at the front of the methodology we run at the start of an engagement.

Step two: verify what your stack served them

Arrival is not access, and this is where an AI crawler audit usually finds its answer. The second step reads the response side of the same log lines, and the question is whether a crawler that reached you was actually given anything.

Sort the verified crawler requests by status code. A healthy pattern is dominated by 200s with a scattering of 301s and 404s that match your own known redirects and dead pages. An unhealthy one shows 403s, 429s, or a run of challenge responses, and each of those has a different owner: a 403 is usually a firewall or bot rule, a 429 is rate limiting, and a challenge page means your protection layer decided the visitor looked automated, which of course it was.

Then check what a successful response actually contained. Fetch one of your important pages the way a crawler does, without executing scripts, and confirm your real content is present in that raw response rather than assembled afterwards in the browser. A page that looks complete to you and arrives nearly empty to a crawler produces exactly the same symptom as having no content at all, and it is one of the most common findings in the website development work we take on when a site was built without this constraint in mind.

Your CDN now makes this decision for you, in three categories

There is a development inside step two that deserves its own section because of how directly it governs AI search presence, because it moved the block from a file you control to a setting you may not have looked at.

The three-way split is genuinely useful once you understand what each category does. Search bots collect content so it can be surfaced and cited in answers. Training bots collect content to build models. Agent bots fetch pages in real time because a user asked for something. Those are three different bargains, and a business that wants to be recommended while declining to donate its archive to model training now has a way to express exactly that, which was impossible when the only available control was a single switch.

The audit action is to open the setting and write down what it currently says, then check it against what the business actually wants. Those two things disagree more often than not, usually because a default was accepted years ago by someone solving a different problem. On larger properties the setting may also differ between zones or environments, which is the kind of drift that goes unnoticed until someone compares them, and a familiar exercise inside the enterprise programs we manage.

Step three: verify what came back

The third step closes the loop by measuring AI referral traffic, and it comes with an honesty requirement most reporting skips.

Isolate referrals from the assistant domains in your analytics as their own segment rather than letting them dissolve into a general referral bucket. Then accept the limit: not every assistant sends a referrer, and traffic arriving from a native application frequently carries none at all, which means some genuine AI referral traffic lands in direct and cannot be separated out. Reporting the segment as complete overstates precision; reporting it as a floor is accurate and still useful.

The comparison worth making is between crawl volume and referral volume, because the ratio is the actual answer to whether presence is converting into anything. Heavy crawling with negligible referrals means you are being read and not recommended, which is a content and corroboration problem. Light crawling with any referrals at all means access is the constraint and the content is probably fine. That single comparison directs the next quarter of work more reliably than any dashboard, and it is the reasoning behind how we structure reporting on the analysis we publish ourselves.

Reading the pattern the three steps produce

Run all three steps and your AI search presence diagnoses itself, which is the point of doing them in order rather than sampling whichever is easiest.

No verified crawler activity at all points at reachability, and the cause is in robots rules, a CDN setting, or a firewall, in that order of likelihood. Activity with a high proportion of refusals points at bot protection tuned for scrapers that is catching everything automated. Clean access with empty responses points at rendering. Clean access, full responses, and no referrals point at the content and the wider web, which is the only one of the four that a writer can fix. Each conclusion is supported by evidence rather than inference, which matters when the fix requires someone else’s engineering time, and it is why we present findings this way in the builds and rebuilds documented in our portfolio.

Write the diagnosis down as a single sentence naming the gate that failed and the team that owns it. An audit that ends in a conclusion someone can act on is worth ten that end in a spreadsheet nobody opens again.

One pattern deserves separate mention because it is easy to misread. Verified crawling that stops abruptly on a specific date is almost never a decision by the crawler. It is a deploy, a plugin update, a security policy change, or a certificate problem, and the date in the log is the fastest route to the cause.

Cadence, and the sites that need to check more often

Unlike the answer side, the infrastructure half of an AI search presence audit does not need monthly repetition, because infrastructure does not drift on its own. It needs to be re-run at the moments when it breaks.

Those moments are predictable: any migration, any CDN or firewall configuration change, any plugin or dependency update that touches routing or security, and any rebuild of a template. Adding the verified-crawler check to the post-release checklist costs minutes and catches the class of failure that otherwise surfaces a quarter later as an unexplained decline.

Businesses running several domains, subdomains, or regional properties should audit each one separately rather than assuming the configuration is shared. Settings drift between zones, and a regional site can be blocking everything while the primary domain is wide open, which is a failure mode we specifically watch for on the location sections we maintain and on any client running more than one property.

What this audit cannot tell you

Being clear about the boundary is what keeps an AI search presence audit credible. Infrastructure evidence proves that machines reached your pages and what they received. It says nothing about whether your content deserved to be quoted.

It also cannot see the memory side of these systems. A large share of assistant answers are written without fetching anything, drawing on what a model already absorbed, and no server log will ever record a request that was never made. Nor can logs tell you what an answer said about you, accurately or otherwise. Those questions need the answer-side audit, which is a different exercise entirely and pairs naturally with this one.

Nor does any of it measure quality. A crawler fetching a page proves the page was available, not that it was worth quoting, and a site can pass every check in this audit while remaining absent from answers for entirely editorial reasons.

The sequencing between the two is the practical takeaway. Run the infrastructure audit first, because it is faster, produces harder evidence, and can invalidate a content project before anyone starts it. Then run the answer-side audit against a site you know the machines can actually read. That order has saved more wasted quarters than any single technique we use, and explaining why is a routine early conversation for the team here with a new client convinced the problem must be the writing.

Frequently Asked Questions

Sources