How to measure AI referral traffic and AI crawler visits

To measure AI referral traffic, keep two separate numbers. Crawler visits are requests from bots such as OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot, and you find them in server or CDN logs, then verify them against the operator's published IP ranges. Referral visits are people who clicked a link inside ChatGPT, Perplexity, Copilot, Gemini or Claude, and you find them in analytics by referrer host and by the utm_source=chatgpt.com tag ChatGPT adds to search links. Neither number tells you how often an assistant mentions you, and both undercount.

This guide shows how to collect each one, how to segment them, and which conclusions the data does not support.

Crawler visits and referral visits are two different numbers

The two get mixed up because both are "AI traffic". They come from different sources, answer different questions and fail in different ways.

Crawler visits are a readiness signal: they show whether access works. Referral visits are an outcome signal: they show some of the human traffic that results. Report them side by side, never added together.

It also helps to split crawler visits by purpose, because a training crawler and a search crawler mean different things for you. The training, search and user-triggered crawler guide explains the three groups.

How to count AI crawler visits in your logs

Most AI crawlers fetch HTML without running JavaScript, so your analytics tag never fires and their visits do not show up in GA4. You need raw access logs from your web server, your host or your CDN.

Count requests per crawler

On an nginx or Apache server using the common "combined" log format, this counts requests by user-agent token. Match on the token, not the full string, because version numbers change (OpenAI's docs currently show OAI-SearchBot/1.4 and GPTBot/1.4 as examples).

TOKENS='GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User'

grep -oE "$TOKENS" access.log | sort | uniq -c | sort -rn

The full list of tokens and what each one does is in the AI crawler user agents list. Google-Extended and Applebot-Extended never appear in logs: they are robots.txt tokens only. Google crawls with its existing crawlers, such as Googlebot, and Apple with Applebot.

Check what status codes each crawler gets

A count of visits is less useful than knowing what the crawler received. In the combined format the status code is the first field after the quoted request line, and the user agent is the last quoted field:

awk -F'"' '$6 ~ /OAI-SearchBot/ { split($3, f, " "); print f[1] }' access.log | sort | uniq -c

A healthy result is mostly 200 and 304. Lots of 403, 429 or 503 responses usually point to a WAF rule, rate limit or bot-protection setting, which matters more than the visit count.

OpenAI may add a robots.txt; marker to the user agent when OAI-SearchBot or GPTBot fetches robots.txt, according to its crawler documentation. That lets you separate robots.txt checks from page fetches:

grep 'OAI-SearchBot' access.log | grep -vc 'robots.txt;'

The guide to finding AI crawler visits in server logs covers other log formats, hosting panels and log shipping.

Verify the IP before you count it

Anyone can send GPTBot in a user-agent header. Before a request goes into your report, check its source IP against the operator's published list. OpenAI publishes searchbot.json, gptbot.json and chatgpt-user.json. Anthropic publishes bots.json, and Perplexity publishes perplexitybot.json. At the time of writing all of these use the same shape: a prefixes array of ipv4Prefix entries.

This script fetches the current lists each time it runs, so you never hard-code ranges that go stale:

import ipaddress, json, sys, urllib.request

SOURCES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "Anthropic": "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
}

def networks(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ip-check/1.0"})
    with urllib.request.urlopen(req, timeout=10) as resp:
        data = json.load(resp)
    for prefix in data.get("prefixes", []):
        cidr = prefix.get("ipv4Prefix") or prefix.get("ipv6Prefix")
        if cidr:
            yield ipaddress.ip_network(cidr)

ranges = {name: list(networks(url)) for name, url in SOURCES.items()}

for line in sys.stdin:
    if not line.strip():
        continue
    try:
        ip = ipaddress.ip_address(line.strip())
    except ValueError:
        continue
    owners = [name for name, nets in ranges.items() if any(ip in net for net in nets)]
    print(ip, ", ".join(owners) or "not in any published range")

Save it as verify_ips.py and feed it the unique IPs that claimed to be a given bot:

grep 'GPTBot' access.log | awk '{print $1}' | sort -u | python3 verify_ips.py

Anthropic's file does not say which of its bots uses which prefix, so a match means "from Anthropic", not "ClaudeBot". Google, Bing, Apple and Common Crawl also support forward-confirmed reverse DNS. The crawler verification guide walks through each method.

If you use Cloudflare

Cloudflare's AI Crawl Control is listed as available on all plans and includes an "Analyze AI traffic" view. On the Free plan it identifies AI crawlers by user-agent string only; higher plans can use Bot Management detection. Treat Free-plan figures as claimed identity, not verified identity. The Cloudflare AI crawler settings guide explains which of those settings also block crawlers.

How to measure AI referral traffic in analytics

Human clicks from assistants show up as ordinary sessions. You segment them by two signals: the referrer host and UTM parameters.

The signals to look for

OpenAI's Publishers and Developers FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs so publishers can track traffic from ChatGPT search results. The referrer hosts in the table are the assistants' own web addresses. They are not a documented contract, so check your own referrer report for the exact hostnames you receive. Use both signals for ChatGPT: a session can carry the UTM without a referrer, or a referrer without the UTM.

Build an "AI assistants" channel in GA4

Google's custom channel group help page describes this exact setup:

1. In Admin, under Data display, open Channel groups and click Create new channel group. You need Editor access.
2. Add a channel named "AI assistants" with the condition Source matches regex.
3. Click Reorder and move the channel above Referral. GA4 puts each session in the first channel it matches, so below Referral it would never fill.
4. Save, then open Acquisition > Traffic acquisition and switch the report to the new channel group.

Google's help page offers an example regex, but it is broad: patterns such as .*gpt.* match any source containing those letters, which can pull in unrelated sites. A tighter version that anchors on the hostnames looks like this:

^(chatgpt\.com|(www\.)?perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com|claude\.ai)$

Custom channel groups apply to reports retroactively, so you can see past data once it is saved. Standard GA4 properties allow two custom channel groups. Review the regex when assistants change their domains.

Without GA4: count referrals in the server log

The combined log format stores the referrer in the second quoted field, after the request line ($4 when you split on quotes). This counts human visits by assistant origin and the ChatGPT UTM tag:

awk -F'"' '$4 ~ /chatgpt\.com|perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com|claude\.ai/ { print $4 }' access.log \
  | sed -E 's#^(https?://[^/]+).*#\1#' | sort | uniq -c | sort -rn

awk -F'"' '$2 ~ /utm_source=chatgpt\.com/' access.log | wc -l

Server-side counts include visitors who block analytics scripts, so they are often higher than GA4's. They also include any bot that sends a fake referrer, so expect some noise.

Why many AI visits arrive with no referrer

Referral numbers undercount for reasons you cannot fix from your side:

• Referrer policy. When a page sets no policy, browsers default to strict-origin-when-cross-origin, which sends only the origin (such as https://chatgpt.com/) to other sites, and a page that sets no-referrer sends nothing, as MDN's Referrer-Policy reference explains. An assistant's web app controls that setting.
• Apps. Links opened from desktop or mobile apps can arrive without a referrer and land in "Direct".
• Copied links. A person who copies a URL from an answer and pastes it into a new tab carries neither a referrer nor, if they trim it, the UTM.
• Google AI features. Clicks from AI Overviews and AI Mode come from Google itself. Google's AI features documentation says that traffic is counted in the Search Console Performance report under the "Web" search type. There is no separate AI Overviews report, so you cannot split it out. The AI Overviews requirements guide covers what Google does document.
• Citations without clicks. Bing Webmaster Tools' AI Performance report, a public preview since February 2026, counts citations across Copilot and Bing AI summaries. None of its metrics are clicks, so do not mix them with sessions.

Treat every referral figure as a minimum. A rise in "Direct" traffic to deep pages at the same time as a rise in AI referrals may be related, but you cannot prove it from analytics alone.

What not to conclude from the numbers

• Crawler visits are not citations. OAI-SearchBot fetching a page means it can read it. It does not mean ChatGPT quoted it.
• GPTBot volume says nothing about search. GPTBot collects content that may be used for training. ChatGPT search visibility is controlled by OAI-SearchBot, and OpenAI treats the two settings independently.
• User-triggered fetches are demand, not crawling. ChatGPT-User, Claude-User and Perplexity-User visit because a person asked a question, so their volume tracks what people ask about, not a crawl schedule.
• Zero referrals does not mean zero mentions. Assistants often answer without a link being clicked, and many clicks carry no referrer.
• A change after a robots.txt edit is not instant. OpenAI says its search systems take about 24 hours to adjust to robots.txt changes, and Perplexity says up to 24 hours, so compare full weeks, not days.

A simple monthly report

Keep the two halves apart and keep the method fixed from month to month.

Note any robots.txt, WAF or rendering change in the month next to the figures, so a later reader can see why a line moved.

Frequently asked questions

Can Google Analytics show AI crawler visits?

No. GA4 and similar tools record visits when a JavaScript tag runs in a browser, and most AI crawlers do not run it. Crawler visits come from server, CDN or WAF logs.

Does every ChatGPT click carry utm_source=chatgpt.com?

OpenAI says ChatGPT adds the tag to referral links from its search results, but not every visit will keep it, for example when a link is copied and trimmed. Segment on the chatgpt.com referrer as well as the UTM tag.

Can I see AI Overviews traffic separately in Search Console?

No. Google counts clicks from AI Overviews and AI Mode in the Performance report under the "Web" search type, together with other Google Search clicks.

Is a request with a GPTBot user agent always from OpenAI?

No. User-agent strings are easy to fake. Check the source IP against openai.com/gptbot.json before you count it.

Next step

Counting crawler visits only makes sense once crawlers can reach your pages. Run the free single-page check, which needs no account, to confirm that before you set up the report. If you would rather not build the crawler half yourself, Crawlito's monthly retainer (from $150) includes a verified AI-crawler visit report built from your server or Cloudflare logs.