Amazon CloudFront user agent: why crawler names vanish in logs
Your origin logs show the Amazon CloudFront user agent instead of GPTBot or Googlebot. Here is why, how to forward User-Agent, and where to read the real names.
By Crawlito · 8 min read
In short
- Unless a cache behavior forwards the header, CloudFront replaces the viewer's User-Agent with the value Amazon CloudFront before it sends the request to your origin.
- Attaching an origin request policy that includes User-Agent, such as the managed UserAgentRefererHeaders policy, restores the real value at the origin without adding it to the cache key.
- Your origin only sees cache misses, so CloudFront standard logs, whose cs(User-Agent) field records the viewer's header, are the better source for counting crawler visits.
- CloudFront appends the viewer's IP to the end of X-Forwarded-For, so trust only the address CloudFront added, and only on requests that came from CloudFront's origin-facing ranges.
- CloudFront delivers standard logs on a best-effort basis, usually within an hour but sometimes up to 24 hours late.
On this page
- Why CloudFront sends "Amazon CloudFront" as the user agent
- How to confirm CloudFront is rewriting the header
- How to forward User-Agent with an origin request policy
- Why origin logs undercount crawlers behind CloudFront
- How to read crawler names from CloudFront standard logs
- How to get the crawler's real IP from X-Forwarded-For
- Frequently asked questions
- Next step
If every line in your origin access log says Amazon CloudFront where the browser or crawler name should be, CloudFront is doing what AWS documents: unless you configure it to forward the User-Agent header, it replaces the viewer's value with Amazon CloudFront before sending the request to your origin. The fix is an origin request policy that includes User-Agent. To count crawler visits, read CloudFront's own standard logs, which record the viewer's real user agent, because your origin never sees requests served from the cache.
This post covers the CloudFront side only. For the general method of finding and counting AI crawlers in a log, see AI crawlers in server logs. For 403 errors on S3-backed static sites, see CloudFront and S3 return 403 on direct page links.
Why CloudFront sends "Amazon CloudFront" as the user agent
AWS's page on request and response behavior for custom origins says that if you do not configure CloudFront to cache based on User-Agent, it adds this header before forwarding a request:
User-Agent: Amazon CloudFrontIt adds the header whether or not the viewer sent one, and removes the viewer's own value. The origin request policy page lists User-Agent among the headers CloudFront includes in every origin request, but by default the value is CloudFront's, not the viewer's.
That one substitution causes three separate problems:
| Symptom at the origin | Cause |
|---|---|
| No GPTBot, ClaudeBot, PerplexityBot or Googlebot lines at all | Every request carries Amazon CloudFront instead |
| User agent rules in nginx, Apache or the app never fire | The rule never sees the crawler's token |
| Every request appears to come from CloudFront IP addresses | The TCP peer is a CloudFront server; the viewer's IP is in X-Forwarded-For |
The same AWS table shows other headers are affected too. Without forwarding, CloudFront removes Referer, Accept and Accept-Language, so a referrer-based report built on origin logs fails the same way.
How to confirm CloudFront is rewriting the header
Before changing anything, prove it. Add the headers you care about to your origin's log format. For nginx:
log_format cfdebug '$remote_addr xff="$http_x_forwarded_for" [$time_local] '
'"$request" $status ua="$http_user_agent" '
'cfid="$http_x_amz_cf_id"';
access_log /var/log/nginx/cfdebug.log cfdebug;Then request a URL that is not cached yet, through CloudFront, with a crawler-style user agent. Adding a throwaway query string usually forces a miss only if your cache policy includes query strings, so a fresh path is more reliable:
curl -s -o /dev/null -A "Mozilla/5.0 (compatible; TestBot/1.0)" \
"https://www.example.com/cf-ua-test-$(date +%s)"
tail -n 1 /var/log/nginx/cfdebug.logIf the line shows ua="Amazon CloudFront" and a non-empty cfid, the request came through CloudFront (AWS documents that it adds X-Amz-Cf-Id to every origin request) and the user agent was replaced. If you see ua="Mozilla/5.0 (compatible; TestBot/1.0)", the header is already forwarded and the cause of missing crawler lines is elsewhere, most often caching, covered below.
How to forward User-Agent with an origin request policy
CloudFront separates two decisions. A cache policy controls the cache key. An origin request policy controls what else is sent to the origin on a cache miss, without splitting the cache. AWS recommends against caching on User-Agent because it has so many values that the cache hit ratio would collapse, so the origin request policy is the right place for it.
AWS publishes managed origin request policies. The ones that matter here:
| Managed policy | ID | Sends User-Agent | Notes |
|---|---|---|---|
UserAgentRefererHeaders | acba4595-bd28-49b8-b9fe-13317c0390fa | Yes | Only User-Agent and Referer; no cookies or query strings |
AllViewerExceptHostHeader | b689b0a8-53d0-40ab-baf2-68738e2966ac | Yes | All viewer headers except Host, plus CloudFront headers; meant for API Gateway and Lambda function URL origins |
AllViewer | 216adef6-5c7f-47e4-b989-5492eafa07d3 | Yes | Every header, cookie and query string, including the viewer's Host |
If a behavior already uses an AllViewer policy, User-Agent already reaches the origin. For most sites that only need the crawler name, UserAgentRefererHeaders is the smallest change. An origin request policy only works alongside a cache policy on the same behavior, so keep the cache policy you have.
In CloudFormation, the change is one property on each cache behavior:
DefaultCacheBehavior:
TargetOriginId: app-origin
ViewerProtocolPolicy: redirect-to-https
CachePolicyId: !Ref ExistingCachePolicyId
OriginRequestPolicyId: acba4595-bd28-49b8-b9fe-13317c0390fa # UserAgentRefererHeadersIn the console, edit each behavior and pick the policy as its origin request policy. Do it for every behavior, not just the default one: a /blog/* or /docs/* behavior with no origin request policy still sends Amazon CloudFront.
One trap: once the origin can see the crawler's name, it is tempting to serve crawlers something different. The cache key does not include User-Agent, so a 200 response tailored to one crawler can be cached and served to everyone who requests that URL next. AWS also documents that CloudFront caches a 404 from the origin, and caches a 403 only when the origin sends Cache-Control: max-age or s-maxage. A cached error is kept for the longer of that value and the error caching minimum TTL, which is 10 seconds by default. If you block a bot by user agent at the origin, send the 403 without those headers, or block at the edge instead, as in AWS WAF Bot Control for AI crawlers.
Why origin logs undercount crawlers behind CloudFront
Forwarding User-Agent fixes the name, not the count. CloudFront only sends an origin request on a cache miss. A crawler that fetches a page already in the edge cache is served there and never reaches your server. On a site with a high hit ratio, your origin log can show a fraction of the real crawler activity, and the pages missing from it are often the popular ones.
The log field x-edge-result-type in CloudFront's own logs tells you how each request was served, with values such as Hit, RefreshHit, Miss and Error. That is the number your origin log cannot give you.
How to read crawler names from CloudFront standard logs
CloudFront standard logs record every viewer request, including the cs(User-Agent) field, which holds the viewer's User-Agent header as received at the edge. Logging comes in two versions:
- Standard logging (legacy) writes gzip files to an S3 bucket named like
prefix/DISTRIBUTIONID.YYYY-MM-DD-HH.unique-ID.gz. The bucket must have ACLs enabled. - Standard logging (v2) can deliver to S3, CloudWatch Logs or Data Firehose, lets you choose fields and output format (JSON, plain, w3c, raw, or Parquet for S3).
CloudFront does not charge for enabling standard logs; you pay the destination's storage, delivery or conversion costs. AWS says logging changes take effect within 12 hours and that delivery becomes reliable about four hours after you enable it.
The legacy format is W3C extended, tab-separated, with a #Fields: header line. The standard logging reference lists the default order: c-ip is field 5, cs-uri-stem 8, sc-status 9, cs(User-Agent) 11 and x-edge-result-type 14. Spaces and some other characters are URL-encoded, so a user agent reads Mozilla/5.0%20AppleWebKit/537.36.... Crawler tokens such as GPTBot or OAI-SearchBot survive encoding, so a pattern match still works.
Because v2 lets you pick fields, read the column positions from the #Fields: line rather than hard-coding them. The script below assumes tab-separated files with a #Fields: line, as the legacy format produces; a v2 delivery with a different output format or field delimiter needs the -F value adjusted:
AI='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|bingbot'
zcat -f cf-logs/*.gz | awk -F'\t' -v re="$AI" '
/^#Fields:/ { sub(/^#Fields: /, ""); n = split($0, f, " ")
for (i = 1; i <= n; i++) col[f[i]] = i; next }
/^#/ { next }
match($col["cs(User-Agent)"], re) {
print substr($col["cs(User-Agent)"], RSTART, RLENGTH),
$col["sc-status"], $col["x-edge-result-type"] }' \
| sort | uniq -c | sort -rnThe output groups each crawler by status and by whether the edge served it from cache:
640 OAI-SearchBot 200 Hit
85 OAI-SearchBot 200 Miss
12 ClaudeBot 403 ErrorFor larger volumes, AWS points to querying CloudFront logs with Amazon Athena. If an unfamiliar name turns up, how to identify an unknown bot user agent covers the next step.
Two limits apply. AWS says standard logs are delivered on a best-effort basis: usually within an hour, sometimes up to 24 hours late, and in rare cases an entry is never delivered. And a request whose headers exceed 20 KB or whose URL exceeds 8,192 bytes is not logged at all. Use the logs for patterns, not exact accounting.
How to get the crawler's real IP from X-Forwarded-For
A user agent can be faked, so you still need the source IP to confirm a crawler, as described in how to verify a real AI crawler. Behind CloudFront, the origin's TCP peer is a CloudFront server. AWS documents how the viewer IP reaches you:
- If the viewer sent no
X-Forwarded-For, CloudFront adds one with the viewer's IP. - If the viewer sent one, CloudFront appends the viewer's IP to the end:
X-Forwarded-For: 192.0.2.4,192.0.2.3,192.0.2.2, where192.0.2.2is the address CloudFront saw. - A load balancer between CloudFront and your server appends the CloudFront edge address after that.
Everything before CloudFront's entry was supplied by the client and can be invented. Trust the right-most address that is not one of your own proxies. With nginx's realip module, which is not built by default (check nginx -V for --with-http_realip_module), that looks like this:
# CloudFront ranges from ip-ranges.json (service CLOUDFRONT), regenerated on a schedule
include /etc/nginx/cloudfront-ranges.conf; # lines of: set_real_ip_from <cidr>;
set_real_ip_from 10.0.0.0/16; # your load balancer subnet, if any
real_ip_header X-Forwarded-For;
real_ip_recursive on;With real_ip_recursive on, nginx replaces the client address with the last address in the header that is not trusted, which is the viewer CloudFront saw. AWS publishes its ranges in ip-ranges.json; see CloudFront's IP ranges page.
Two hardening steps make that address trustworthy:
- Accept origin traffic only from CloudFront. If your origin is on AWS, allow inbound traffic from the managed prefix lists
com.amazonaws.global.cloudfront.origin-facingandcom.amazonaws.global.ipv6.cloudfront.origin-facingin its security group. Otherwise anyone can reach the origin directly with a forgedX-Forwarded-For. - Consider the CloudFront-Viewer-Address header. A custom origin request policy can add
CloudFront-Viewer-Address, which carries the viewer's IP and source port, for example198.51.100.10:46532, so you do not have to parseX-Forwarded-For. AWS notes it can go in an origin request policy but not a cache policy. It is only as trustworthy as step 1 makes it, because a client that reaches the origin directly can send the header too.
In CloudFront's own logs you do not need any of this: c-ip is the viewer's address, or the address of a proxy the viewer used, with the original in x-forwarded-for.
Frequently asked questions
Is "Amazon CloudFront" in my logs a bot?
No. It is the value CloudFront puts in the User-Agent header of its own requests to your origin when the viewer's header is not forwarded. The real visitor, person or crawler, is hidden behind it.
Will forwarding User-Agent hurt my cache hit ratio?
Not when you use an origin request policy. Headers in an origin request policy are sent to the origin on a miss but are not part of the cache key. Adding User-Agent to a cache policy is what AWS advises against.
Does CloudFront send robots.txt requests to my origin with the real user agent?
Only on a cache miss, and only if the behavior that matches /robots.txt forwards User-Agent. To see every robots.txt fetch, filter CloudFront's standard logs on cs-uri-stem instead.
Why don't the numbers in CloudFront logs and my origin logs match?
Cache hits are served at the edge and never reach the origin, and CloudFront delivers its logs on a best-effort basis with occasional delays of up to 24 hours. Compare the two only over a window that ended more than a day ago.
Next step
Once the crawler names are back, check whether those crawlers actually get your pages: run the free check on a page that sits behind CloudFront. If you would rather have someone review the distribution's behaviors, WAF rules and robots.txt together with evidence for each crawler, an audit covers them.
Filed under AI crawlers. Topics: aws, cdn, server-logs, ai-crawlers, troubleshooting