YouBot user agent: how You.com's crawler identifies itself
The YouBot user agent string, its robots.txt and Crawl-delay support, how it handles 429 and 503, and three ways to verify a request really came from You.com.
By Crawlito · 8 min read
In short
- YouBot is the crawler that discovers and indexes pages for the You.com search engine, and its user agent contains the token YouBot/1.0 and the label env:prod.
- You.com says YouBot honors robots.txt, including rules for User-agent: YouBot and Crawl-delay, and caches robots.txt for 30 minutes.
- YouBot reduces its crawl rate on a 429 response and pauses crawling on a 503, according to You.com's crawler page.
- You.com documents three verification methods: Web Bot Auth signatures checked against its key directory, reverse DNS ending in search.you.com, and the IP range 68.67.112.0/24.
- You.com's key directory held one Ed25519 key when we fetched it on 15 November 2026, and the keyid in its own signature equals that key's JWK thumbprint.
On this page
YouBot is the web crawler You.com uses to discover and index pages for its search engine. It identifies itself with a Chrome-style user agent that contains YouBot/1.0, it follows robots.txt groups named YouBot including Crawl-delay, and it slows down on a 429 and pauses on a 503. To confirm a request is genuine, You.com documents three checks: a Web Bot Auth signature, a reverse DNS name under search.you.com, and the IP range 68.67.112.0/24.
Everything below comes from You.com's YouBot page and its key directory, both read on 15 November 2026. For the general methods behind each check, see how to verify real AI crawlers and Crawl-delay for AI bots. This post covers what is specific to YouBot.
What the YouBot user agent looks like
You.com publishes this user agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36X.X.X.X stands for a Google Chrome version number, so the exact string in your logs changes as that version moves. The parts You.com calls out are the compatible identifier YouBot/1.0, the documentation link https://docs.you.com/youbot and the environment label env:prod. At the time of writing, that docs link returns a 301 redirect to https://you.com/docs/youbot.
Three things follow for log analysis:
- Match on
YouBot, not on the whole string. The Chrome version changes, so an exact-match rule will break. - Do not treat it as a browser. The string carries
ChromeandSafaritokens, so a filter that counts anything withChrome/as human traffic will count YouBot visits as people. env:prodis the documented label. You.com only documents the production value. If you see a YouBot string with a different label, the page does not say what it means, so verify it like any other request.
To pull YouBot requests out of a combined-format access log:
grep 'YouBot/' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | headThis lists the client IPs that sent the YouBot token, busiest first. Those are the addresses to verify in the sections below. If you are trying to work out an unfamiliar string rather than one you already know, how to identify an unknown bot covers the general approach.
What YouBot can fetch
You.com's page lists the technical properties of the crawler, which help when you are checking whether a server or CDN setting could get in its way:
| Property | What You.com lists |
|---|---|
| Protocols | HTTP/1.1, HTTP/2, HTTPS with TLS 1.2 and above |
| Content types | HTML pages, PDF documents, plain text, XML and RSS feeds, structured data (JSON-LD, microdata, RDFa) |
| Content encodings | gzip, deflate, Brotli (br) |
HTTP/3 is not on the list, and the page does not say whether YouBot runs JavaScript, so do not rely on content that only appears after client-side rendering.
How YouBot reads robots.txt
You.com says YouBot fully respects robots.txt, including user-agent-specific rules and crawl-delay settings. Its page gives three examples. Blocking the whole site:
User-agent: YouBot
Disallow: /Blocking selected directories:
User-agent: YouBot
Disallow: /private/
Disallow: /admin/Allowing the crawler but asking it to slow down:
User-agent: YouBot
Crawl-delay: 10You.com's page does not say what unit the 10 is in. Crawl-delay is not part of RFC 9309, so the number has no standard meaning, and the page does not state a minimum or maximum value.
You.com does not say whether YouBot's parser follows RFC 9309 to the letter, but these are the rules the standard sets for how a group like this interacts with the rest of your file:
- The token is case-insensitive. The standard says crawlers MUST use case-insensitive matching to find their group, so
youbotandYouBotselect the same group. - A named group replaces
*. The*group is only a fallback for crawlers with no matching group. Once you add aYouBotgroup, a crawler that follows the standard ignores your*rules, so copy any disallows you still want into it. - Several YouBot groups are merged. If more than one group matches, their rules are combined into one.
Here is a group that keeps YouBot out of private areas, asks for a gap between requests and still lets it reach the rest of the site:
User-agent: YouBot
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Crawl-delay: 5
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /searchHow long a robots.txt change takes
You.com says YouBot caches robots.txt for 30 minutes, so a change can take up to 30 minutes to take effect. That is well inside the RFC 9309 guidance that crawlers should not use a cached copy for more than 24 hours. If YouBot is still fetching a path an hour after you disallowed it, check your own CDN cache and the robots.txt delay guide before assuming the rule is being ignored.
How YouBot reacts to status codes
You.com lists how YouBot treats each response:
| Status | What YouBot does |
|---|---|
200 OK | Crawls and indexes the page |
301 / 302 | Follows the redirect |
404 Not Found | Removes the page from its index |
429 Too Many Requests | Reduces its crawl rate |
503 Service Unavailable | Pauses crawling temporarily |
The page also says YouBot uses adaptive crawl rate limiting and spreads requests across multiple IP addresses. It does not say how far the rate drops after a 429, how long a 503 pause lasts, or whether YouBot reads a Retry-After header. Do not build plans on numbers it does not publish.
In practice that gives you a ladder, from gentlest to strongest:
Crawl-delayin the YouBot group for steady, polite pacing.429from a rate limit when YouBot exceeds what your server can take. Aim the limit at requests that are both verified YouBot and over your threshold, not at robots.txt itself.503only during real maintenance, since You.com says it pauses crawling.- Email
spider@you.comif the load continues. You.com lists reporting excessive crawl rates and requesting crawl adjustments as reasons to write, and asks for your domain, timestamps of the problem requests, a description and server logs if you have them.
Note what the table implies for 404: You.com removes those pages from its index. A page that briefly returns 404 during a deploy can drop out until YouBot crawls it again. Use 503 for planned downtime, not 404.
How to verify a YouBot request
User agents can be faked, and You.com says so on its own page. It offers three checks and recommends the first.
| Method | What you check | Where it runs |
|---|---|---|
| Web Bot Auth (recommended) | HTTP Message Signatures validated against You.com's public keys | Your CDN or your own code |
| Reverse DNS | The IP resolves to youbot-{octets}.search.you.com and back | Your server |
| IP range | The IP sits inside 68.67.112.0/24 | Firewall, WAF or log script |
Web Bot Auth signatures
You.com says YouBot uses Cloudflare's Web Bot Auth scheme. To verify, you check the HTTP Message Signatures in the request headers and validate them with the public keys in You.com's directory:
https://you.com/.well-known/http-message-signatures-directoryYou.com says the keys are JSON Web Keys using Ed25519. When we fetched the directory on 15 November 2026, it held one key and no kid field. The response was signed itself, and the created and expires values in signature-input change with every response:
HTTP/2 200
content-type: application/http-message-signatures-directory+json
cache-control: public, max-age=300
signature-input: binding0=("@authority");created=1794752389;keyid="5hbjTA0RjjhxRRquk32fSeRWfTgAdSHCZMK1afI1qvI";alg="ed25519";expires=1794752689;tag="http-message-signatures-directory"{"keys":[{"kty":"OKP","crv":"Ed25519","x":"0DJXAmXhPs6hq-7SHZFAFJZHW2-8zQNh8wkdD8UKZSM"}]}Because there is no kid, you match a request's keyid by computing the key's JWK thumbprint. The Web Bot Auth architecture draft says keyid must be a base64url JWK SHA-256 thumbprint, computed for Ed25519 keys as in RFC 8037. We ran that calculation, and the thumbprint equals the keyid the directory uses to sign itself. When we tried, the directory returned a 403 to Python's default user agent, so this version fetches it with curl:
curl -s https://you.com/.well-known/http-message-signatures-directory | python3 -c '
import base64, hashlib, json, sys
for k in json.load(sys.stdin)["keys"]:
canon = json.dumps({"crv": k["crv"], "kty": k["kty"], "x": k["x"]},
separators=(",", ":"))
tp = base64.urlsafe_b64encode(hashlib.sha256(canon.encode()).digest())
print(tp.rstrip(b"=").decode())
'On 15 November 2026 it printed 5hbjTA0RjjhxRRquk32fSeRWfTgAdSHCZMK1afI1qvI. Keys can rotate, and the response carries cache-control: public, max-age=300 (five minutes), so fetch the directory rather than hard-coding the key above. You.com's page does not say what value YouBot sends in the Signature-Agent header, so check what arrives in your own logs. If your CDN verifies Web Bot Auth for you, use its result rather than repeating the check at your origin. Web Bot Auth explained covers the signature format and the full verification steps.
Reverse DNS and the IP range
If you cannot check signatures, You.com documents a hostname pattern and an IP block. The hostname is youbot- followed by the connecting IP with hyphens instead of dots, under search.you.com. You.com's example maps 68.67.112.106 to youbot-68-67-112-106.search.you.com. A forward lookup on that name should return the same IP.
Because the pattern is exact, you can check for the precise hostname rather than a suffix. This script does both checks:
#!/usr/bin/env bash
# Usage: ./verify-youbot.sh 68.67.112.106
ip="$1"
expected="youbot-${ip//./-}.search.you.com"
in_range=$(python3 -c "import ipaddress,sys; print(ipaddress.ip_address(sys.argv[1]) in ipaddress.ip_network('68.67.112.0/24'))" "$ip")
ptr=$(dig +short -x "$ip" | head -n1 | sed 's/\.$//')
fwd=$(dig +short "$expected" A)
echo "IP in 68.67.112.0/24: $in_range"
echo "PTR: ${ptr:-none} (expected $expected)"
if [ "$ptr" = "$expected" ] && printf '%s\n' "$fwd" | grep -qxF "$ip"; then
echo "PASS: reverse and forward DNS match"
else
echo "FAIL: DNS does not match the documented YouBot pattern"
fiRun it on IPs from the log command above. A request that carries YouBot/ but fails both the range and the DNS check, with no valid signature, does not match anything You.com documents for its crawler. You.com publishes the range as a single /24 in the page text, not as a JSON file, so recheck the page from time to time rather than assuming it never changes.
Should you allow YouBot?
YouBot is a search crawler. You.com describes it as the crawler that indexes the web for its search engine, and its FAQ says it crawls publicly accessible pages to provide real-time search results for You.com users. The page does not say whether crawled content is used to train models, so do not assume either way.
That places it with the other search crawlers in training vs search vs user crawlers: blocking it keeps your pages out of You.com's index, and allowing it only removes the technical barrier. Whether You.com shows or cites a page depends on things outside your robots.txt.
If you want YouBot in but at a gentler pace, a Crawl-delay in its own group, plus a 429 rate limit for verified YouBot traffic above your threshold, keeps it crawling without loading your server.
Frequently asked questions
What robots.txt token does YouBot use?
Use User-agent: YouBot. You.com's examples use that exact token. RFC 9309 requires case-insensitive matching, so youbot should select the same group, but copying You.com's spelling avoids the question.
Does YouBot honor Crawl-delay?
Yes. You.com says YouBot honors crawl-delay directives and gives Crawl-delay: 10 as an example, though it does not document the unit or any limits.
How long does YouBot cache robots.txt?
You.com says 30 minutes, so a change can take up to half an hour to take effect for YouBot.
What IP addresses does YouBot use?
You.com says legitimate YouBot requests come from 68.67.112.0/24, with reverse DNS names of the form youbot-{octets-with-hyphens}.search.you.com.
How do I ask You.com to recrawl or slow down?
Email spider@you.com. You.com lists recrawl requests, crawl adjustments and excessive crawl rates among the reasons to contact it.
Check what YouBot can reach
A YouBot group, a CDN rule and a rate limit can each block a crawler you meant to allow. Run the free check on an important page to see whether the main AI crawlers can fetch it, and use the scripts above to confirm the YouBot traffic in your logs is real.
Filed under AI crawlers. Topics: ai-crawlers, robots-txt, web-bot-auth, server-logs