Skip to content

Bytespider: what is known and how to block it

Bytespider has no official ByteDance documentation we could find. Here is what is known, what is not, and how to block it in robots.txt, nginx, Apache or Cloudflare.

By Crawlito · 8 min read

In short

  • Bytespider is attributed to ByteDance, and Cloudflare lists it in its AI Crawl Control reference as an AI crawler from ByteDance, but we could not find ByteDance's own documentation of its purpose, IP ranges or robots.txt behaviour.
  • Third-party sources disagree on whether Bytespider obeys robots.txt, so a Disallow line is worth adding but should not be your only control.
  • robots.txt is not access control: RFC 9309 says its rules are not a form of access authorization, so a crawler that does not honour them is stopped only by your server, firewall or CDN.
  • A user-agent rule refuses any request that sends the Bytespider token, but it cannot stop traffic that sends a different user agent, and ByteDance publishes no IP list we could find to check requests against.
  • After you block, confirm it with a curl test that sends the Bytespider user agent and with your access logs, not by waiting to see if the traffic goes away.
On this page

Bytespider is a crawler that third parties attribute to ByteDance, the company behind TikTok. We could not find official ByteDance documentation for it, so its exact purpose, its IP ranges and whether it obeys robots.txt are not confirmed by its operator. Sources that track bots disagree on compliance. If you want it gone, add a robots.txt rule for the polite case and a server or CDN rule that refuses the Bytespider user agent for everything else.

This post separates what is documented from what is only reported, then gives working rules for robots.txt, nginx, Apache and Cloudflare, plus a way to confirm they work.

What is Bytespider, and what is not documented?

Here is what can be traced to a source:

  • Cloudflare lists it. Its AI Crawl Control reference shows Bytespider with ByteDance as the operator, the category "AI Crawler" and the user agent token Bytespider.
  • Cloudflare has described its scale. In a July 2024 post, Cloudflare named Bytespider among the top four AI crawlers by request volume on its network. It said the crawler is reportedly used to gather training data for ByteDance's large language models, including those behind its Doubao assistant. The word "reportedly" is Cloudflare's hedge, not ours.
  • Bot directories list a user agent string. Known Agents gives an example that contains the token Bytespider and a ByteDance feedback address. Match on the token, not the full string, because the rest can change.

Here is what we could not find in any primary source: a ByteDance page that states what Bytespider collects and why, a published list of its IP ranges, a reverse DNS method for verifying it, or a statement of how it treats robots.txt. Known Agents cites a Toutiao webmaster page as the official website, but we could not retrieve any crawler documentation from it at the time of writing. If ByteDance publishes one later, trust it over any directory, including this post.

That gap matters for how you handle it. Without published IP ranges you cannot tell a real Bytespider request from an imitation, which is why the later sections lean on the user agent token and on logs.

Does Bytespider obey robots.txt?

It depends on whom you ask, and none of the answers come from ByteDance in a form we could verify.

Known Agents lists Bytespider as expected to follow robots.txt and reports a compliance rate from its own measurements. Other trackers and blog posts say it ignores robots.txt. We do not repeat any compliance percentage here, because we cannot check how it was measured. Our AI crawler user agents list takes the same position: its robots.txt behaviour is unconfirmed.

Two points are on firmer ground. First, robots.txt is voluntary by design. RFC 9309 states that its rules "are not a form of access authorization". A crawler reads your file and chooses whether to comply. Second, you can test your own site: if you see Bytespider requests for a path after you added a Disallow for it, that crawler is not honouring your file, whatever a directory says. How to find AI crawler visits in server logs shows how to pull those requests out.

How to block Bytespider in robots.txt

Add a group that names the token and disallows everything:

User-agent: Bytespider
Disallow: /

Under RFC 9309, a crawler matches its product token case-insensitively and follows the group that names it. When no group names it, it falls back to the * group, and if there is none, no rules apply. So this group affects only Bytespider and does not touch Googlebot, Bingbot or the search crawlers you may want to keep.

Two cautions. The file has to be reachable: under the same RFC, a robots.txt that returns a 4xx status is treated as unavailable, which lets a crawler fetch anything, while a 5xx tells it to assume everything is disallowed. And this rule covers only crawlers that name themselves Bytespider. If your goal is to opt out of training while staying visible in AI search, read how to allow AI search and block AI training, because Bytespider is one entry in a longer list, not the whole job.

How to block it at nginx, Apache or Cloudflare

Because robots.txt cannot enforce anything, the dependable step is to refuse the request. These rules key on the Bytespider token in the User-Agent header, case-insensitively.

nginx

nginx's map module supports ~* for a case-insensitive regular expression, and map belongs in the http context. Variables are evaluated only when used, so the map costs nothing for requests that never reach the check.

nginx
map $http_user_agent $block_bytespider {   # in http {}
    default       0;
    ~*bytespider  1;
}

server {
    listen 80;
    server_name example.com;

    location = /robots.txt {
        try_files $uri =404;   # keep robots.txt readable
    }

    location / {
        if ($block_bytespider) {
            return 403;
        }
        # ... your proxy_pass or try_files
    }
}

nginx treats a variable as false when it is empty or 0, so only requests whose user agent contains the token get the 403. The documented non-standard code 444 closes the connection without sending a response, which some operators prefer for abusive traffic. Reload with nginx -t && nginx -s reload.

Apache

In Apache 2.4, SetEnvIfNoCase sets a variable when the User-Agent matches a regular expression, ignoring case. Apache's access control guide explains that a not rule cannot allow or deny a request by itself, and shows it inside RequireAll next to a positive rule (its example uses Require not ip). The same pattern applies to Require not env, which tests the variable that SetEnvIfNoCase sets (see the Require env documentation):

apache
SetEnvIfNoCase User-Agent "Bytespider" block_bytespider

<Directory "/var/www/html">
    <RequireAll>
        Require all granted
        Require not env block_bytespider
    </RequireAll>
</Directory>

The same lines can go in .htaccess if your host allows the needed overrides. Apache's own page cautions that user-agent filtering is unreliable because clients can send any value, which leads to the next section.

Cloudflare

You have two routes. In the dashboard, go to AI Crawl Control > Security > Crawlers, find Bytespider and select Block. Cloudflare's documentation says this is enforced with a WAF custom rule on your zone, and that the response can be 403 or 402. Custom block responses need a paid plan.

Or write the custom rule yourself. Cloudflare's string operators are case-sensitive unless stated otherwise, so lowercase the field first:

lower(http.user_agent) contains "bytespider"

Set the action to Block. Rules run in order and a terminating action such as Block stops evaluation, so place it above any rule that would otherwise allow the request. Cloudflare and AI crawlers explains where these settings sit and what else can interfere.

Why user agent rules miss spoofed bots

A user agent is text the client writes about itself. Every rule above refuses requests that say "Bytespider" and nothing else. That has two consequences.

The block is not an identity check. Anyone can send the token, so the rule also refuses a person testing with curl or a scraper that copied the name. That is harmless: you wanted that traffic gone either way.

The block misses traffic that does not send the token. A crawler can change its user agent, and a scraper that imitates a browser looks like a visitor. If the load keeps coming after your rule is live, the load was not, or was not only, from a client that announces itself as Bytespider. Cloudflare's own page says that on the free plan, AI Crawl Control identifies crawlers from their user agent strings, and that paid plans add more thorough detection.

For the operators that publish IP lists or reverse DNS names, you can verify a claim by source address, as how to verify real AI crawlers describes. We found no such method for Bytespider, so that route is closed. What remains is behaviour: look at which IPs send the most requests and which paths they hit, and apply per-IP limits there. Rate limiting AI crawlers covers the nginx and CDN patterns. Do not block a whole network range because one address misbehaved, since you may be refusing real visitors.

How to confirm the block works from your logs

Test before you trust. Send one request with the Bytespider token and one without, and compare:

bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)" \
  https://example.com/

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/

The first should print 403 (or 402 if you chose that on Cloudflare) and the second 200. Your request comes from your own IP, so this proves the user agent rule works, not that real Bytespider traffic is caught.

For that, read the logs. In the common combined log format, the status code is field 9:

bash
grep -i bytespider access.log | awk '{print $9}' | sort | uniq -c

Before the block you will see mostly 200. After it, you should see 403 and no new 200 lines. A crawler that keeps asking will still appear in your logs with refused requests, which is expected: the block answers the request, it does not stop the client from sending it. If you use Cloudflare, check the Events tab of Security Analytics for the matching action, and check your origin logs to confirm the requests stopped arriving there. If 200 responses to Bytespider continue, a rule is in the wrong order, a cache is answering before your rule runs, or the config was not reloaded.

Keep an eye on the same log for a few weeks. The user agent token is the only stable thing you have to match on, and it is documented only by third parties, so a change would show up as traffic that no longer carries it.

Frequently asked questions

Is Bytespider a Google or TikTok crawler?

It is attributed to ByteDance, which owns TikTok, and Cloudflare lists ByteDance as its operator. It has no connection to Google. We could not find ByteDance's own documentation, so which ByteDance products use the data is not confirmed.

Will blocking Bytespider hurt my Google rankings?

Googlebot is a different crawler with its own user agent, and a rule matching only Bytespider does not touch it. Check that your rule contains only that token, and that no broader pattern such as "bot" slipped in. We cannot say how any block affects visibility on ByteDance's own products, because that is not documented.

Should I rely on robots.txt alone?

No. Add the robots.txt group, since compliant crawlers read it, but also enforce the block on your server or CDN. RFC 9309 says robots.txt rules are not access authorization, and sources disagree on whether Bytespider follows them.

Can I verify a real Bytespider request by IP address?

Not with anything we could find from ByteDance. Unlike OpenAI or Google, which publish IP lists or DNS names, ByteDance has no verification method we could locate. Treat the user agent as a claim and judge traffic by behaviour.

Check what your site tells crawlers

The free check reads one page the way AI crawlers do, with no account needed. It shows the robots.txt rules for named AI crawlers, so you can confirm your Bytespider group is read as you intended. It runs from Crawlito's servers, not ByteDance's, so your own logs remain the evidence for whether Bytespider is blocked.

Filed under AI crawlers. Topics: bytespider, ai-crawlers, robots-txt, bot-protection, waf