Server log analysis for AI crawler traffic

Everything else in this category is sampled, modelled, or guessed. Log data is a record of what happened. No consent banner blocks it, no ad blocker strips it, no vendor sits between you and it.

That matters more than it sounds, because the alternatives are genuinely thin. Google's Search Console generative-AI performance reports, announced 3 June 2026, expose impressions, pages, countries, devices and dates, and explicitly do not expose clicks, click-through rate, position, or the prompt the user typed. AI Overview and AI Mode clicks arrive at your site as ordinary `google / organic` with no referral parameter, so there is no AI channel in GA4 to build. Against that, a log line is a fact.

What is actually in a server log?

One line per request, and each line carries the same fields.

The IP address that made the request. The timestamp. The path requested. The response code your server returned. The user agent string identifying the client.

That is it. No names, no sessions in the analytics sense, no conversions. A log is a ledger of fetches.

For crawler work, four of those fields do the job. User agent tells you who claims to be visiting, IP tells you who actually is, path tells you what they wanted, and the response code tells you whether they got it.

The response code is the one people ignore and it is often where the finding is. A crawler receiving redirects, blocked responses, or server errors on your most important pages is a problem you can fix today, and it will never appear in any analytics tool because analytics only runs when a page renders in a browser.

Set your expectations for volume before you look, or you will misread an empty file as a failure. When Ahrefs examined every request to llms.txt paths across the 137,210 domains in its web analytics product with traffic in May 2026, it found the traffic that did occur was 96% bots and 4% humans, and 77% of those bots were not AI tools at all: SEO audit tools 21.7%, unidentified agents 14.9%, general crawlers 13.1%, technology profilers 11.6%. All four AI categories together came to 19.5%, and the retrieval bots that actually produce citations were 1.1%, of which OpenAI's search crawler was 0.74%. Ahrefs sells SEO software, so label it vendor research. The shape holds across the rest of your log: most of what visits you is not what you are looking for.

Why does verification by IP matter so much?

Because a user agent string is a line of text the visitor chooses.

Anyone can send a request claiming to be any crawler. Scrapers do it routinely to get past rules that would otherwise block them. If you count by user agent alone, you will report a number that includes impostors, and you will not know what share.

Every serious provider publishes IP ranges or a reverse DNS method for verifying their crawlers. The check is mechanical: take the IP from the log line, confirm it falls inside the published range for the agent it claims to be, discard the ones that do not.

This is the difference between a measurement and a guess, and it is the specific step that gets skipped by tools selling crawler dashboards. A crawler report that has not verified by IP is reporting on a population that includes anyone who wanted to be counted.

Separate the crawler types while you are in there. Search indexing crawlers, AI training crawlers, and AI retrieval crawlers that fetch a page in response to a live user question do different jobs. The last group is the one connected to answers appearing, and lumping all three into one AI traffic number is how a real measurement gets turned into a metric that looks impressive and predicts nothing.

What does one month of log analysis actually produce?

Here is a full pass on a small site. Substitute your own counts and the arithmetic holds.

The raw file. 41,208 requests in thirty days across all agents.

Filter by user agent. 612 requests claim to be an AI crawler of some kind.

Verify by IP. 147 of those fail the published range check and get discarded, leaving 465 verified requests. The spoof rate on your own site is 147 of 612, which is 24%derived. That is the number nobody reports, and it is a quarter of the headline.

Split by crawler job. Of the 465 verified requests, 312 are training crawlers, 65 are search-index crawlers, and 88 are live retrieval crawlers. Retrieval is 88 of 465, or 18.9%derived. Only that last slice is connected to a user getting an answer with you in it.

Check the response codes. 61 of the 465 returned a 403. That is 13.1%derived of your verified AI crawler traffic being turned away, almost certainly by a security plugin or a CDN rule nobody remembers writing. This is the finding. It is invisible in robots.txt, invisible in analytics, and fixable this afternoon.

Check the paths. The 88 retrieval fetches concentrated on four pages: services, two location pages, and pricing. Pricing was fetched 34 times, which is worth knowing when you decide whether to publish pricing at all, because a page that does not exist cannot be retrieved by anything.

What you may now say out loud. Verified AI crawlers made 465 requests in thirty days, 88 of them live retrieval fetches, and 13.1%derived of the total were refused by your own infrastructure. What you may not say is that any of it produced a citation.

Why is robots.txt the wrong place to look?

Because it is a request, not a control, and it tells you nothing about what happened.

BuzzStream measured this directly, tracking 4 million citations across 3,600 prompts in ten industries covering ChatGPT, Gemini, Google AI Overviews and Google AI Mode in March 2026. Sites blocking OpenAI's live retrieval bot were still cited 70.6% of the time. Sites blocking its search-index bot, 82.4%. Sites blocking the training crawler GPTBot, 88.2%. Sites blocking Google-Extended, 92.3%. Two concrete cases from that dataset: cnbc.com blocks three OpenAI agents simultaneously and appeared 1,298 times, and yahoo.com blocks Google-Extended and appeared roughly 30,000 times. BuzzStream sells outreach software, so label it vendor research, and note that the authors offer several candidate explanations and settle on none.

So the two claims you will hear are both wrong. Blocking GPTBot does not remove you from ChatGPT, because GPTBot governs training and not search citations. And Google-Extended does not remove you from AI Overviews or AI Mode. Google's own documentation is unambiguous that Google-Extended is a robots.txt token controlling training and grounding for Gemini, not a crawler with its own user agent. There is no Gemini crawler. Google grounds Gemini on content that ordinary Googlebot already fetched. The snippet controls, `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex`, are the only levers, and they are the same blunt ones that apply to classic Search.

For a local business the recommendation is to block nothing. The common and invisible failure is the inverse of the one everyone worries about: your CDN or security plugin quietly returning a 403 to the retrieval bots you wanted. You cannot see that in robots.txt. You can see it in a log file in about four minutes.

What can log data tell me, and what can it not?

It tells you retrieval. It does not tell you citation.

You can say with confidence that a given crawler fetched a given page on a given date, how often, and whether it succeeded. You can say whether that pattern changed after you published something. You can say which sections of your site attract crawler attention and which are ignored.

You cannot say whether any of it was used in an answer, whether a user saw your name, or whether anyone visited afterwards. No log anywhere contains that. A provider fetching your page is not a promise about anything downstream.

Be careful with the volume too. Crawler counts are lumpy. A site might get almost nothing for two weeks and then a burst. Reading a week of that as a trend is exactly the small sample problem that fools people everywhere else in analytics, and crawler data is noisier than most. The KDD 2014 rule of thumb applies without modification: a figure that looks interesting or different is usually wrong, and the first thing to suspect is your own instrumentation. A crawler spike that arrives the same week you changed hosts is a hosting change.

The honest summary line is short: logs prove your content is being fetched, and they prove nothing beyond that. Say it that way in your reporting and you will be one of very few doing so.

How do I actually get and read the logs?

Ask your host. That is the first step and often the only hard one.

Managed hosting platforms vary. Some give you raw access logs by download. Some expose them only through a dashboard. Some retain them for a very short window by default, which is the setting to change before you need the data rather than after.

If you sit behind a content delivery network, the CDN log is the one that matters, because it sees requests your origin server never does. It is also where the 403s are being generated, so it is the log with the finding in it.

Once you have files, you do not need special software. A modest site's monthly log opens in any spreadsheet after a filter, and a few command line steps will give you counts by agent and by path. The analysis is genuinely simple. The access is the part that takes a week of emails.

Do it monthly, on a schedule, as part of a routine that already exists. A monthly review habit that survives busy season is the only way this gets looked at more than twice, because nothing about log analysis is urgent enough to interrupt a bad week.

If you run more than one site or location, keep them separate. Crawl patterns differ, and two locations perform differently for reasons that are usually specific rather than general.

What to do this week

Email your host and ask three questions. Do I have access to raw server or CDN logs. How long are they retained. Can I extend that retention or export them automatically.

If retention is short, change it today. Logs you did not keep are the one thing in analytics that cannot be recovered later.

Pull the last thirty days. Filter to requests whose user agent contains a known AI crawler name. Verify a sample of them against published IP ranges before you believe any of it, and record your spoof rate.

Look at response codes for those verified requests. Anything that is not a success on an important page is your finding, and a 403 is the one to chase first.

Then check which pages they fetch most, and write the month's counts down somewhere you will still have them next quarter.

Questions about your logs?

Email me at eric@seod.com with the name of your hosting provider and whether you use a CDN. I will tell you where your logs live on that platform, what the default retention is, and the one setting to change this week so you stop losing data you will want later.

I answer these myself. It is a short email and it is the kind of thing that is genuinely annoying to work out alone.

More on measurement sits in the analytics library.