AI crawler user agents and what your server logs reveal

Everything else in this subject is sampling and inference. This one thing is a record of events that happened on hardware you pay for.

That is worth more than it sounds. When the rest of a category is priced on estimates, the party holding first-party data has the only defensible position in the room. Your logs already exist. Most businesses have never looked at them.

What does an AI crawler look like in a log file?

A line like every other line. Timestamp, requesting IP, the path requested, the response code, the bytes sent, and a user agent string at the end.

The AI-related agents sort into three groups, and the distinction changes what you do about them.

Training crawlers collect content to train models. They fetch broadly and infrequently and produce no citations. OpenAI's `GPTBot` is the best known.

Search index crawlers build the retrieval index a product searches when answering. OpenAI's `OAI-SearchBot` is the clearest example. These are the ones that can lead to your page being cited.

Live user fetchers are triggered in the moment, when a person asks an assistant about something and it goes to look. OpenAI's `ChatGPT-User` is this category. They often arrive one page at a time, right after a query, and they are the closest thing to a real-time signal you will find.

Google is the exception that trips people up, and it matters for what follows. `Google-Extended` is not a crawler. Google describes it as a robots.txt token with no user agent of its own. There is no Gemini crawler at all. Google grounds its AI features on content already fetched by ordinary Googlebot, which means you will never see a distinctly Gemini-shaped line in your logs no matter how carefully you look.

Operators publish their own crawler names and change them, so treat any list in an article as a starting point and check the current documentation from each company. The categories are stable. The strings are not.

Why identify by IP range instead of user agent?

Because a user agent string is a claim, and an IP range is a fact.

Anything can send a request announcing itself as any crawler. SEO tools do it. Scrapers do it. Researchers do it. If you count by user agent, you are counting self-descriptions, and a portion of them are false. This is not a hypothetical problem, it is the normal condition of the open web.

The scale of the noise is measurable. When Ahrefs analysed requests to llms.txt paths across 137,210 domains in May 2026, it found that of the traffic those files received, 96% was bots and 4% humans, and 77% of those bots were not AI tools at all. SEO audit tools were 21.7%, unidentified agents 14.9%, general crawlers 13.1% and technology profilers 11.6%. All four AI categories combined came to 19.5%, and the retrieval bots that can actually produce a citation were 1.1%. Ahrefs sells SEO software, and its own stated caveat is that fetched does not mean read, so treat every figure as a ceiling.

Read that as a warning about your own log filter. If you grep for AI-sounding strings and report the total, roughly four fifths of what you count is something else.

The major operators publish the IP ranges their crawlers use, precisely so site owners can verify. Match a request against the published range, and you know. Match it against a text string, and you know what somebody typed.

Any report of AI crawler activity that does not state its verification method is not a measurement, it is a text search. Ask. The answer takes one sentence and tells you whether the rest of the report is worth reading.

What does thirty days of logs actually look like?

Here is the shape of the exercise, with arithmetic you can rerun against your own file.

Raw volume: 412,000 requests in the month, across everything.

Filter to AI-shaped user agents: 2,000 requests.

Verify each source IP against the operators' published ranges: 1,760 verify. 240 do not. That means 12% of what called itself an AI crawler was something else wearing the name. Those 240 go in the bin before you draw any conclusion.

Sort the 1,760 by category: 1,200 training crawler requests, 400 search index crawler requests, 160 live user fetches. The middle number is the one that matters, because the search index is the path to a citation. Training crawlers dominate the count and mean the least. Ahrefs saw the same skew in its llms.txt data, where training crawlers fetched roughly five times more often than retrieval bots did.

Now check response codes on those 400 search index requests. Say 300 returned a 200, 80 returned a 403, and 20 hit a redirect chain. That is 20% of the requests from the only crawlers that can produce a citation, refused by your own firewall, and another 5% wasted on redirects. Nothing in any visibility dashboard will ever show you that.

Finally, check page coverage. Of your 48 indexed pages, the search index crawler fetched 31. Seventeen were never fetched at all.

You now have two work lists. Eighty blocked requests, which is a configuration change. Seventeen pages, which is a discovery problem. Neither required buying anything, and both are more concrete than any score you could have subscribed to.

Does blocking a crawler actually remove you from the answers?

No, and this is the most expensive misunderstanding in the subject.

The claim in circulation is that blocking `GPTBot` in robots.txt takes you out of ChatGPT. It does not, on two separate grounds. `GPTBot` governs the training crawl, not the search citations. And blocking has been measured against outcomes.

BuzzStream, working with Citation Labs' XOFU tracking tool, examined 4 million citations across 3,600 prompts covering ChatGPT, Gemini, Google AI Overviews and Google AI Mode across ten industries, published March 2026. It then checked the robots.txt of the sites being cited.

By citation volume, roughly 70% of all ChatGPT citations in that dataset came from sites blocking ChatGPT's retrieval bots. Two concrete cases from the same study: cnbc.com blocks ChatGPT-User, GPTBot and OAI-SearchBot simultaneously and still appeared 1,298 times, while yahoo.com blocks Google-Extended and appeared roughly 30,000 times.

Nobody has a conclusive explanation, and BuzzStream does not claim one. Only 15% of the cited publications predated ChatGPT's launch, which weakens the "indexed before you blocked" theory. Around 70% of the dataset also blocks CCBot, which weakens the Common Crawl backdoor theory. Robots.txt is voluntary in the first place. And some retrieval systems lift citations straight out of search results, which your robots.txt has no authority over at all. BuzzStream is a link building software company, so label it as vendor research with a disclosed method.

The practical reading for a local business: blocking is not a control, it is a request, and the measured compliance rate is low. If you were planning to block on principle, that is a legitimate position, but do not expect it to change what an assistant says about you.

The same logic kills the mirror-image claim about Google. `Google-Extended` does not remove you from AI Overviews or AI Mode. Google's own documentation says the controls for that are `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex`, which are the same blunt instruments that also strip your ordinary Search snippet. There is no way to opt out of one without paying for it in the other.

What can logs actually tell you?

More than you would expect from something this dull.

Which pages get fetched, and which never do. If your main service page has never been retrieved by a search index crawler, no amount of content strategy on that page is the problem. Discovery is.

Whether fetches are succeeding. A crawler receiving redirects, timeouts or server errors is a fixable engineering problem masquerading as a visibility problem. The common invisible failure is a CDN or security plugin silently returning 403 to OAI-SearchBot, PerplexityBot or ClaudeBot. Cloudflare, Sucuri and Wordfence configurations all turn up doing it. Nobody chose it, nobody was told, and it will not appear in any visibility dashboard. Check your logs, not your robots.txt.

Trend over time. Crawl volume rising after you publish, or falling after a site migration, is a signal you can act on the same week.

Which surfaces are paying attention to you at all. Retrieval is a precondition for citation. If nothing fetches you, nothing can cite you, and every other explanation is downstream of that.

The trend view also cures a common misreading. Because AI answers vary run to run, owners conclude something broke. Usually nothing broke, and non-determinism explains why the ranking changes every time you check. Log volume moving steadily while answers bounce around is the normal picture.

What can logs not tell you?

Whether a fetch produced a citation. Whether a citation produced a visit. Whether a visit produced a customer.

The gap between fetch and citation is wider than most people assume. In the University of St. Gallen measurement study, ChatGPT activated web search on only about 42.2% of runs, meaning 57.8% of runs returned no citations at all regardless of what had been crawled. A page can be fetched perfectly and still sit outside more than half the answers, for reasons that have nothing to do with your site.

Logs also tell you nothing about the sources that are not your website. A large share of what gets said about a local business comes from a Business Profile, review platforms and third-party pages, none of which appear in your logs. This is why a business with no website is not invisible to AI search, and also why your log file is an incomplete picture even when it is a perfect record.

They will not explain geography either. Local answers vary by where the person asking is standing, the same way you can rank in one part of town and not another. Your logs have no opinion about that.

And they say nothing about the text being summarized. If assistants are reading your review pages, they are reading your responses too, which is one practical reason a templated reply to a bad review can make it worse. That text is out there being retrieved by systems you do not control.

What to do this week

1. Ask your host or developer for thirty days of raw access logs. If nobody can produce them, fix that first, because it is a bigger problem than AI visibility. 2. Filter for known crawler activity and verify a sample by IP range against the operators' published lists. Expect a meaningful share to fail verification. 3. Check response codes for those requests. A 403 served to a search index crawler is the cheapest fix in this entire subject, and your CDN is the first place to look. 4. List the pages that were never fetched. That list is your discovery problem, in order. 5. Save the month as a baseline. Without one, next month's numbers mean nothing.

Once you can see who is fetching you, the next decision follows naturally, which is whether you want to block any of them.

Want to know who is actually crawling you?

Send me your domain and your host at eric@seod.com and I will tell you exactly how to pull your raw logs on that platform, plus a filter you can paste to isolate AI crawler traffic. If you can send a log sample, I will read it and tell you what it shows, including whether your CDN is quietly refusing anyone. No charge, no pitch.

I do this because raw logs are the one honest dataset in this category and most owners have never been shown how to open them.

Otherwise, there is more on AI search and AEO here.