AI crawlers in server logs: how to find and count their visits

To find AI crawler visits in your server logs, search the user agent field of your access log for each crawler's token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and so on), then group the matches by crawler, status code, page and day. That tells you which AI tools reach your site, which pages they read, and whether your server answers them with content or with a 403. Because a user agent is just a string anyone can send, finish by checking the source IPs against each operator's published list.

The commands below work on the default log formats of nginx and Apache. They use only grep, awk, sort and uniq, plus a short Python script for IP checks. All crawler details come from the operators' own documentation, read on 8 November 2026.

Where your access logs are and what a line looks like

Both nginx and Apache ship a "combined" log format. nginx defines it in the ngx_http_log_module docs, and Apache in mod_log_config. A line looks like this:

20.171.207.10 - - [08/Nov/2026:10:12:01 +0000] "GET /pricing HTTP/1.1" 200 5123 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

In order: client IP, two identity fields, timestamp, request line, status code, bytes sent, referrer, user agent. If you split the line on double quotes, the request is field 2, the status and size are in field 3, the referrer is field 4 and the user agent is field 6. Every awk command in this post relies on that.

Common default locations:

Older days are usually rotated and gzipped (access.log.2.gz). On Linux, zcat -f /var/log/nginx/access.log* streams current and compressed files together, so you can pipe it into any command below in place of access.log.

Two cautions. First, if you use a custom or JSON log format, adjust the field numbers. Second, if a CDN sits in front of your server, the IP in your origin log may be the CDN's, not the crawler's, unless you restore the real client IP from a header such as the one your CDN documents.

Which user agent tokens to search for

Each operator documents a token that appears in its user agent. Match on the token, not the full string, because version numbers change: OpenAI's docs currently show GPTBot/1.4, where older guides quote /1.0 or /1.1.

Sources: OpenAI, Anthropic, Perplexity, Common Crawl, Apple, Bing and Meta. For the longer list, including Amazon, DuckDuckGo and Mistral, see AI crawler user agents.

Set the list once as a shell variable so every command uses the same tokens. The matches are case-sensitive, so keep the casing shown:

AI='GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Applebot|Googlebot|bingbot|meta-externalagent|meta-externalfetcher|meta-webindexer|Amazonbot|DuckAssistBot|MistralAI-User|Bytespider'

How to count hits per AI crawler

Hits per crawler

This prints one count per crawler, taking the first token found in each user agent:

awk -F'"' -v re="$AI" 'match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log \
  | sort | uniq -c | sort -rn

A plain grep -o is tempting but overcounts: Bingbot's user agent contains bingbot twice, once in the token and once in its help URL.

Hits per day

awk -F'"' '$6 ~ /OAI-SearchBot/ { print substr($1, index($1, "[") + 1, 11) }' access.log \
  | sort | uniq -c

Swap in any token. The dates sort as text, which is fine within one month.

Which pages each crawler fetched

awk -F'"' '$6 ~ /ClaudeBot/ { split($2, r, " "); print r[2] }' access.log \
  | sort | uniq -c | sort -rn | head -20

If your most important pages are missing from the search crawlers' lists but old tag pages and parameter URLs are near the top, look at your internal links and sitemap before anything else.

robots.txt fetches

awk -F'"' -v re="$AI" '$2 ~ /^GET \/robots\.txt/ && match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log \
  | sort | uniq -c

Crawlers that follow robots.txt fetch it regularly. OpenAI says it may add a robots.txt; marker to the GPTBot and OAI-SearchBot user agent on those requests, so grep -F 'robots.txt;' access.log isolates them.

Are AI crawlers getting 403 or 429? Check the status codes

A count tells you a crawler tried. The status code tells you whether it got anything. This groups every AI hit by crawler and status:

awk -F'"' -v re="$AI" 'match($6, re) { split($3, s, " "); print substr($6, RSTART, RLENGTH), s[1] }' access.log \
  | sort | uniq -c | sort -k2,2 -k1,1rn

Sample output:

 812 GPTBot 200
  40 GPTBot 404
 233 OAI-SearchBot 403
  57 PerplexityBot 429

How to read it:

A 403 to GPTBot may be a deliberate training opt-out. A 403 to OAI-SearchBot, Claude-SearchBot or PerplexityBot is different: those search tools cannot read the page, whatever your robots.txt says. OpenAI, for example, asks sites that want to appear in ChatGPT search to allow both the OAI-SearchBot user agent and requests from its published IP ranges. Google documents that 429 and 5xx responses make its crawlers slow down; other operators document less, so treat sustained 429s to any search crawler as a problem. Rate limiting AI crawlers covers how to throttle without locking them out.

Do not panic over some 404s. In a December 2024 analysis of nextjs.org traffic, Vercel and MERJ found 34.82% of ChatGPT's fetches hit 404 pages, against 8.22% for Googlebot. That is a snapshot of one site, but it suggests AI crawlers request plenty of stale or wrong URLs.

How to check that a "GPTBot" is really GPTBot

Anyone can send GPTBot in a user agent. Scrapers can use it to get past user-agent rules, so before you allow or block anything based on these numbers, check the IPs. Most operators publish their ranges as JSON:

• OpenAI: gptbot.json, searchbot.json, chatgpt-user.json
• Anthropic: bots.json
• Perplexity: perplexitybot.json, perplexity-user.json
• Common Crawl: ccbot.json

These files change often (OpenAI regenerated chatgpt-user.json on 6 November 2026), so download them fresh rather than copying ranges into config. Extract the IPs claiming one token, download the list, and compare:

awk -F'"' '$6 ~ /GPTBot/ { split($1, a, " "); print a[1] }' access.log | sort -u > gptbot-ips.txt
curl -s https://openai.com/gptbot.json -o gptbot.json
python3 check_ips.py gptbot.json gptbot-ips.txt

check_ips.py uses only Python's standard library. It reads the prefixes array that all the files listed above share at the time of writing, with ipv4Prefix or ipv6Prefix entries (Common Crawl's list includes IPv6 ranges):

import ipaddress, json, sys

ranges_file, ips_file = sys.argv[1], sys.argv[2]
prefixes = json.load(open(ranges_file))["prefixes"]
networks = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in prefixes]

for line in open(ips_file):
    if not line.strip():
        continue
    ip = ipaddress.ip_address(line.strip())
    ok = any(ip in net for net in networks)
    print(ip, "in published list" if ok else "NOT in published list")

Googlebot, Bingbot, Applebot and CCBot (over IPv4) can also be verified with a reverse and forward DNS lookup. How to verify a real AI crawler covers each method.

How to read AI crawler activity in Cloudflare

If your site sits behind Cloudflare, your origin logs only show requests Cloudflare let through. A crawler blocked or challenged at the edge leaves no trace on your server, so a missing crawler in your logs can mean an edge rule, not a crawler that never came.

Cloudflare gives you three places to look:

1. AI Crawl Control. Available on all plans, its Analyze AI traffic view shows requests per crawler, the distribution of status codes by class, and the most popular paths, filterable by date, crawler, operator, hostname and path. Referral metrics need a paid plan. On the Free plan, AI Crawl Control identifies crawlers by user agent only, so it shares the spoofing weakness of a grep.
2. Security events. When a request was blocked or challenged, the security events log shows which Cloudflare feature acted on it, such as Bot Fight Mode or a WAF custom rule (AI Crawl Control's blocks run as WAF custom rules). The Cloudflare AI crawler settings guide lists the usual suspects.
3. Logpush. For raw per-request data, Logpush exports the HTTP requests dataset to storage. Useful fields include ClientRequestUserAgent, ClientIP, ClientRequestPath, EdgeResponseStatus, OriginResponseStatus, SecurityAction and VerifiedBotCategory. Cloudflare announced on 30 September 2026 that Logpush is available on Free, Pro, Business and Enterprise plans with usage-based pricing; check that the HTTP requests dataset is offered for your zone before you build on it.

Comparing EdgeResponseStatus with OriginResponseStatus and SecurityAction tells you whether a 403 came from Cloudflare or from your own server: an OriginResponseStatus of 0 means the request never reached your origin.

What the patterns tell you

The three kinds of crawler leave different footprints. Training, search and user-triggered crawlers explains why they exist; in logs they tend to look like this:

These are tendencies, not documented rules, but they help in a few ways:

• A user-triggered fetch means a person asked an assistant something that led to your page. The log does not tell you the question.
• Few GPTBot hits do not mean you are excluded from training. OpenAI says that if you allow both GPTBot and OAI-SearchBot, it may use one crawl for both purposes.
• No search crawler at all over a month is worth checking: robots.txt, firewall rules and CDN settings, in that order.
• Some user-triggered agents do not follow robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them, so seeing these after a Disallow is expected. Anthropic, by contrast, says all its bots, Claude-User included, honor robots.txt.

Crawler visits are only half the picture. Clicks from AI answers show up as human visits with referrers such as chatgpt.com or perplexity.ai; measuring AI crawler visits and referral traffic covers both.

A note on privacy

Access logs hold every visitor's IP address, and under laws such as the GDPR an IP address can count as personal data. Crawler lines come from the operators' IPs, but the same files hold your human visitors too. Before you share logs with a colleague, a vendor or an auditor:

1. Filter to crawler lines first: grep -E "$AI" access.log > ai-only.log.
2. Strip query strings if your URLs can carry emails, tokens or session IDs.
3. Keep the extract only as long as your privacy policy allows, and delete it when the analysis is done.

Frequently asked questions

Why don't I see Google-Extended or Applebot-Extended in my logs?

They are not crawlers. Both are robots.txt tokens that control how content already fetched by Googlebot or Applebot may be used, and Google says Google-Extended has no separate user agent. You will see Google's and Apple's regular crawlers, such as Googlebot and Applebot, instead.

My logs show no AI crawlers at all. Are they blocked?

Possibly. Check your live robots.txt, then your firewall and CDN, because an edge block removes the requests before they reach your server. Small or new sites may also simply get few visits.

Can I trust the user agent string?

Not on its own. Anyone can send GPTBot or ClaudeBot, so confirm the IP against the operator's published list before acting on the numbers.

How far back should I look?

At least 30 days. User-triggered fetches are sparse, and search crawlers may visit some pages only every few weeks, so a single day of logs can be misleading.

Next step

If your logs show 403s or 429s to search crawlers, run the free single-page check on one of the affected pages; it needs no account. If you would rather not run these commands yourself every month, Crawlito's monthly retainer, from $150, includes a verified AI-crawler visit report built from your server or Cloudflare logs.