Unknown bot in server logs: how to identify the user agent
Identify an unknown bot user agent: read its product token and info URL, check operator docs, verify IPs or signatures, then look it up in RDAP and Cloudflare Radar.
By Crawlito · 9 min read
In short
- Under RFC 9110 a User-Agent header is a list of product identifiers and comments written by the client itself, so it tells you who a bot claims to be, never who it is.
- RFC 9309 says a crawler's robots.txt product token should be a substring of its User-Agent header, and its example adds a link to a page describing the crawler, so a missing token is the first warning sign.
- A bot is only as verifiable as its operator makes it: a published IP list, a reverse DNS pattern or a Web Bot Auth signature turns a claim into something you can check.
- An RDAP lookup, such as https://rdap.arin.net/registry/ip/ followed by the address, names the network that holds the IP and redirects to the right regional registry when ARIN does not hold it.
- If the operator documents nothing you can check, treat the bot as unverified: throttle or block it at the server, and do not rely on robots.txt to stop it.
On this page
- Step 1: read what the user agent string claims
- Step 2: find the operator's own documentation
- Step 3: look for a way to verify the claim
- Step 4: look up who owns the IP with RDAP
- Step 5: check Cloudflare's bot directory
- Step 6: decide whether to allow, slow or block
- What to conclude when the operator documents nothing
- Frequently asked questions
- Check what the bots you want are getting
To identify an unknown bot, read its user agent for a product token and an info URL, open that operator's own documentation, and look for something you can check: a published IP list, a reverse DNS pattern or a signed request. Then look up who owns the source IP with RDAP and see whether Cloudflare lists the bot in its directory. If the claim and the evidence agree, you know who runs it. If the operator documents nothing you can check, you cannot verify it, and you decide on behaviour alone: allow, slow down or block.
This post is the decision framework. For counting known crawlers, see AI crawlers in server logs. For the exact check per major operator (OpenAI, Anthropic, Perplexity, Google, Bing and others), see how to verify real AI crawlers. For the known names, see the AI crawler user agents list.
Step 1: read what the user agent string claims
RFC 9110 defines the User-Agent header as one or more product identifiers, each a name with an optional version, followed by optional comments in parentheses:
User-Agent = product *( RWS ( product / comment ) )
product = token ["/" product-version]By convention, product identifiers are listed in decreasing order of significance, and RFC 9110 encourages clients not to borrow other products' tokens to declare compatibility. Many crawlers still start with Mozilla/5.0 and put the part that actually identifies them inside the comment, so look past the first token. RFC 9309, the robots.txt standard, adds the rule that matters here: a crawler's product token (the name you write after User-agent:) should be a substring of the User-Agent header, and the identification string should describe the crawler's purpose. Its own example, which it describes as carrying a link to a page about the crawler's purpose, is:
User-Agent: Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html)Break any unknown string into the same parts:
| Part | In the example | What it tells you |
|---|---|---|
| Compatibility prefix | Mozilla/5.0 | Nothing about the operator |
| Product token | ExampleBot | The name to search for and to use in robots.txt |
| Version | 0.1 | Changes over time; do not match on it |
| Info URL | https://www.example.com/bot.html | Where the operator claims to document the bot |
Pull the distinct unknown strings out of a combined-format log first, so you work through each one once rather than line by line:
awk -F'"' '$6 ~ /[Bb]ot|[Cc]rawl|[Ss]pider/ { print $6 }' access.log \
| sort | uniq -c | sort -rn | head -30Three patterns change what you do next:
- A clear token and an info URL. Go to step 2.
- A library or tool name only, such as
python-requests/2.xorScrapy. That names the software, not the operator. See Python requests and Scrapy user agents in logs. - A plain browser string or
HeadlessChrome. Automation that does not name itself. See HeadlessChrome in server logs.
RFC 9110 also says a robotic user agent should send a From header with an email address, so the person responsible for it can be contacted if it causes problems. Many crawlers do not, and the common and combined log formats do not record it, but if you log it, it is one more claim to check.
Step 2: find the operator's own documentation
Open the info URL, but do not stop there: anyone can put any URL in a header. Check that the URL sits on the domain of the company the token suggests, then search that company's own site for its crawler page. A real operator page usually states:
- The exact product token and a sample user agent string.
- What the bot is for: search, training, monitoring, previews, a user-triggered fetch.
- Whether it follows robots.txt, and which token to use.
- How to verify it: an IP list, a reverse DNS domain or a signature.
- A contact address.
If the URL returns a 404, points to an unrelated site, or leads to a page that mentions none of this, record that. It does not prove the bot is malicious, but it does mean the string gives you nothing to verify.
Step 3: look for a way to verify the claim
A product token is a claim. What turns it into evidence is a verification method published by the operator. In practice there are three:
| Method | What you check | Where it comes from |
|---|---|---|
| Published IP list | The source IP falls inside a CIDR range in the operator's JSON file | The operator's crawler page |
| Reverse DNS pattern | The IP resolves to a hostname in the operator's domain, and that hostname resolves back to the same IP | The operator's crawler page |
| Web Bot Auth signature | The request carries Signature, Signature-Input and Signature-Agent headers that verify against a published key | The operator's key directory |
Google's verification guide is a good model of what complete documentation looks like: it names the three domains its crawlers resolve to (googlebot.com, google.com, googleusercontent.com), gives the hostname pattern for each crawler type, and publishes separate JSON files such as common-crawlers.json, special-crawlers.json and user-triggered-fetchers.json. The scripts for running IP list and reverse DNS checks are in how to verify real AI crawlers, and signatures are covered in Web Bot Auth.
If the operator publishes one of these and the request passes, you have identified the bot. If it publishes one and the request fails, the request is not from that operator, whatever the string says.
Step 4: look up who owns the IP with RDAP
When the operator documents nothing, or when you cannot even tell who the operator is, the source IP still tells you whose network the request came from. RDAP is the JSON-based alternative to WHOIS for registration data, and the regional internet registries run RDAP servers. ARIN's RDAP page gives the query pattern <rdap-url>/ip/<IP address>, and explains that bootstrapping sends clients HTTP redirects to the registry that holds the record.
curl -sL https://rdap.arin.net/registry/ip/66.249.66.1 | python3 -c '
import json, sys
d = json.load(sys.stdin)
print(d.get("name"), d.get("handle"), d.get("startAddress"), "-", d.get("endAddress"))
for e in d.get("entities", []):
fn = [x[3] for x in e.get("vcardArray", [None, []])[1] if x[0] == "fn"]
print(" ", e.get("roles"), fn)
'When we ran this on 15 November 2026, it returned the network GOOGLE (66.249.64.0 to 66.249.95.255) with the registrant Google LLC. A query for an address in Europe, such as 193.0.6.139, came back as a 303 redirect to rdap.db.ripe.net and then returned the RIPE NCC record. The -L flag follows that redirect for you.
Read the result for what it is:
- The network belongs to the company the token names. Consistent with the claim, but weaker than the operator's own list: a large company's networks also carry traffic that is not its crawler.
- The network belongs to a cloud or hosting provider. Common and not damning in itself, since many small crawlers run on rented servers. It does mean the IP says nothing about the operator: anyone can rent the same servers.
- The network belongs to a residential or mobile ISP. A self-declared crawler from a home connection is unusual. Treat it as unverified.
Step 5: check Cloudflare's bot directory
Cloudflare publishes a bots and agents directory on Radar at radar.cloudflare.com/bots/directory. Its verified bots documentation says a bot qualifies as verified by declaring who it is, through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and by behaving well, including obeying robots.txt and keeping request rates reasonable. Approved bots appear in that directory.
The same data is available through the Radar bots API, which needs a Cloudflare API token. At the time of writing, each entry has a name, an operator, categories, a description, user agent patterns and an access level that says whether the bot reaches sites under its own identity or forwards requests for another agent or a person. The botVerificationStatus=VERIFIED parameter limits the results to verified bots:
curl -s "https://api.cloudflare.com/client/v4/radar/bots?botVerificationStatus=VERIFIED&format=CSV&limit=100&offset=0" \
-H "Authorization: Bearer $CF_API_TOKEN" | grep -i "examplebot"Page through with offset if the bot is not in the first batch. A verified listing tells you Cloudflare has approved the operator's identification method, and the operator name gives you the right company to look up. An absence tells you less: a legitimate crawler may simply never have applied. If your site is behind Cloudflare, the cf.client.bot field in WAF rules, which marks requests from known good bots, does the check per request, as covered in how to verify real AI crawlers.
Step 6: decide whether to allow, slow or block
Put the evidence together. The decision depends on two questions: can you verify who it is, and does its behaviour cost you anything?
| What you found | Conclusion | Action |
|---|---|---|
| Documented token, verification method, request passes | Identified | Treat it by purpose: allow search and user-triggered agents you want, set robots.txt rules for the rest |
| Documented token, verification method, request fails | Impersonator | Block the combination of that name and IPs outside the published ranges |
| Documented token and robots.txt support, no verification method | Probably that operator, unverified | Use robots.txt for the operator's intent; rate limit if load is a problem |
| No documentation, polite behaviour (robots.txt fetched, low rate) | Unknown, low cost | Leave it, or add a robots.txt group for its token and watch whether it obeys |
| No documentation, heavy or odd behaviour | Unknown, costly | Rate limit or block at the server or edge |
Two checks are worth doing before you block anything by name. First, see whether the bot fetched /robots.txt at all:
grep -F "ExampleBot" access.log | awk -F'"' '{ split($2, r, " "); print r[2] }' \
| grep -c '^/robots.txt'A bot that never requests robots.txt cannot be reading your Disallow rules for its token, so a robots.txt rule is unlikely to be the fix. Second, look at its status codes and request rate over a few days, not a few minutes. A crawler that hits hundreds of URLs a minute and ignores 429 responses is a load problem whatever its identity.
Keep in mind what RFC 9309 itself says: robots.txt rules "are not a form of access authorization". They are a request. Only server, CDN or firewall rules actually stop a client.
What to conclude when the operator documents nothing
Most unknown bots end up here. No info URL, or an info URL that says nothing useful, no IP list, no DNS pattern, no signature, and no entry in Cloudflare's directory. The honest conclusion is narrow:
- You cannot verify who runs it. The name in the string is a label the client chose.
- You can describe what it does. Pages fetched, request rate, network owner from RDAP, whether it reads robots.txt.
- That is enough to act on. Decide on behaviour and cost, not on the name.
What you should not conclude is that a well-known operator is behind it because the string resembles one of theirs, or that blocking it affects any AI search tool. At the time of writing, OpenAI, Anthropic and Perplexity document the user agents of their search and user-triggered crawlers and publish IP lists for them. A bot that matches none of those is not one of those crawlers, so blocking it does not stop them, as long as your rule does not also match their names or ranges.
Frequently asked questions
Can I trust the URL inside a user agent string?
Only as a pointer. The client writes the whole header, URL included, so confirm the URL is on the operator's own domain and that the page documents a way to verify the bot.
Is a bot from a cloud provider IP always fake?
No. Many legitimate crawlers run on rented servers. A cloud IP simply means RDAP cannot confirm the operator, so you need the operator's own IP list, DNS pattern or signature.
Will a robots.txt rule stop an unknown bot?
Only if the bot chooses to follow it. RFC 9309 says robots.txt rules are not access authorization, so for a bot that never fetches robots.txt or ignores it, use server or edge rules instead.
Does blocking an unknown bot hurt my AI search visibility?
Blocking a bot you cannot verify does not block the documented search and user-triggered crawlers of OpenAI, Anthropic and Perplexity, which publish their tokens and IP lists. Just make sure your block rules do not also catch those names or ranges.
Where do I start if there are dozens of unknown bots?
Sort them by request count and work down the list. The few that cause most of the load are worth identifying; the long tail of occasional visitors usually is not.
Check what the bots you want are getting
Once you have sorted the unknown bots, confirm your rules did not also catch the ones you want. Run the free single-page check on one of your pages; it needs no account.
Filed under AI crawlers. Topics: server-logs, ai-crawlers, bot-protection, troubleshooting