Applebot-Extended and Applebot: Apple's crawlers and AI
Applebot-Extended does not crawl: it is a robots.txt token that opts content out of Apple's model training. How it differs from Applebot, with examples.
By Crawlito · 8 min read
In short
Applebot is Apple's web crawler: its data powers search in Spotlight, Siri and Safari and may also be used to train Apple's foundation models.
Applebot-Extended never crawls and never appears in your logs; it is a robots.txt token that tells Apple not to use Applebot's data to train its foundation models.
Disallowing Applebot-Extended does not remove pages from Apple search, and Apple says Applebot-Extended rules are not considered in Search ranking.
Keeping content out of AI-generated answers in Siri and Search is a separate control: the nosnippet meta tag, or isAccessibleForFree set to false for paywalled pages.
If robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules, so a Googlebot block can silently block Apple too.
Applebot is the crawler that fetches your pages for Apple's search features in Spotlight, Siri and Safari, and its data may also be used to train Apple's foundation models. Applebot-Extended is not a crawler at all: it is a robots.txt token that decides whether data Applebot has already crawled may be used for that training. Disallow Applebot-Extended and your pages stay eligible for Apple search; disallow Applebot and Apple stops crawling them.
Everything below comes from Apple's About Applebot support page, which shows a published date of 4 September 2026 and was read on 8 November 2026.
What Applebot and Applebot-Extended each do
Apple documents one general crawler, one usage token and a special-purpose podcast crawler. The first two are easy to confuse because they share a name.
Name
What it is
Makes requests?
Controls
Applebot
Apple's search crawler, which may render pages in a browser
Yes
Whether Apple can crawl your pages at all, for search and everything else
Applebot-Extended
A robots.txt product token
No
Whether Applebot's data may be used to train Apple's general-purpose foundation models
iTMS
A special crawler for Apple Podcasts
Yes
Only URLs tied to registered podcast content; Apple says it does not follow robots.txt
Apple says Applebot data powers "the search technology integrated into many user experiences" in its ecosystem, including Spotlight, Siri and Safari. The same data may be used to train the foundation models behind Apple Intelligence, Services and Developer Tools, and to give AI models up-to-date context when they generate answers in Siri and Search, which can include links to sources.
So in the usual split between training, search and user-triggered crawlers, Applebot is a search crawler whose data can also feed training, and Applebot-Extended is the training opt-out. Google-Extended is a similar robots.txt-only token, although Google's also covers grounding for Gemini; see Google-Extended explained.
How Applebot-Extended works in robots.txt
Apple's own example is short:
User-agent: Applebot-Extended
Disallow: /private/
Apple is direct about what this does and does not do. Applebot-Extended "does not crawl webpages", pages that disallow it "can still be included in search results", and it "is only used to determine how to use the data crawled by the Applebot user agent." Apple also states that "Site rules for Applebot-Extended are not considered in ranking for Search."
Because it never crawls, you will never see Applebot-Extended in your access logs. Apple's crawl requests for ordinary pages come from Applebot (iTMS only fetches registered podcast URLs). Apple does not say that an Applebot-Extended rule changes how Applebot crawls: the token governs how crawled data is used, not whether it is fetched.
Opt the whole site out of training, keep Apple search
This keeps Applebot crawling for search and signals that none of the content should be used for model training. The /admin/ rule is repeated in the Applebot group, for the reason explained in the section on the * group below:
Apple does not document how Applebot-Extended treats the * group when no group names it, so name it explicitly rather than relying on a * rule. The same pattern for other vendors is in how to allow AI search but block AI training.
Opt out one section only
Rules are path-based, so you can keep most of the site available for training and exclude, say, paid research:
Apple says allowing Applebot is what lets your content appear in search results in Spotlight, Siri and Safari, so this gives that up. The softer controls do not: Apple notes that even with Applebot-Extended disallowed and content tagged nosnippet, Applebot may still crawl and your content remains discoverable in those products.
Two robots.txt details that catch people out
Applebot falls back to Googlebot rules
Apple says: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." Consider a file written only with Google in mind:
You might read this as "block Google, allow everyone else". Applebot reads it as a block, because it follows the Googlebot group. The fix is a group that names Applebot:
Apple does not say that Applebot-Extended falls back to Google-Extended, so a Google-Extended rule is not a reliable way to opt out of Apple training. Add an Applebot-Extended line if that is what you want.
Apple's example applies the * group too
In Apple's robots.txt example, a group for Applebot disallows /private/ and the * group disallows /not-allowed/, and Apple says Applebot skips both. That reads differently from RFC 9309, where a crawler that finds a group naming it obeys that group and falls back to * only when no group matches. Bing, for example, says Bingbot ignores the * group when a Bingbot group exists. More on this in how robots.txt groups work.
The safe approach works under either reading: put every rule you want Applebot to obey inside its own group, and repeat the general rules there.
Apple says "Applebot does not follow crawl-delay." Instead, its crawl rate adjusts automatically when a site slows down or returns errors, and Apple caches crawled content to reduce unnecessary crawling. Apple does not name specific status codes and offers no robots.txt setting for crawl rate; if Applebot load is a problem, its page gives applebot@apple.com as the contact.
How to keep content out of Apple's AI answers
Training and AI answers are separate uses with separate controls. Applebot-Extended only covers training. For the answers Siri and Search generate from web content, Apple points to page-level directives instead.
You want to
Use
Effect according to Apple
Stop Apple crawling
User-agent: Applebot + Disallow
No crawling, so no search, answers or training from new crawls
Opt out of model training
User-agent: Applebot-Extended + Disallow
Pages stay in search; content not used to train foundation models
Keep a page out of AI answers and descriptions
<meta name="robots" content="nosnippet">
No description or web answer; data not used as context for AI-generated output
Mark a paywalled page
"isAccessibleForFree": false in JSON-LD
Page can appear in search; content not used as context for AI-generated output
Keep a page out of search
<meta name="robots" content="noindex">
Not indexed, not shown in Spotlight or Siri Suggestions
Apple supports both the generic robots meta name and an Apple-specific one:
For PDFs, images and other files where you cannot add a meta tag, Apple supports the X-Robots-Tag header:
http
X-Robots-Tag: applebot: nosnippet
The paywall signal is page-level only. Apple says section-level markup using hasPart is not supported, which differs from Google's paywall markup. Apple's example:
nosnippet has a cost: Apple says suggestions to visit a nosnippet URL include only the page title. For how these directives behave across other AI tools, see noindex, nosnippet and X-Robots-Tag for AI search.
How to verify real Applebot traffic
Anyone can send a user agent containing Applebot. Apple documents two ways to confirm a request is genuine.
Reverse and forward DNS. Real Applebot IPs resolve to hosts under applebot.apple.com, and that hostname resolves back to the same IP:
Apple's example output for the first command is 179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com. and for the second 17-58-101-179.applebot.apple.com has address 17.58.101.179.
Both steps matter. A reverse lookup alone can be faked by whoever controls the IP's reverse zone; the forward lookup confirms it.
Published IP ranges. Apple lists Applebot's CIDR prefixes in applebot.json, a JSON file with a creationTime field and a prefixes array of ipv4Prefix entries. At the time of writing its creationTime was 15 September 2026. Fetch it on a schedule rather than copying ranges by hand. This checks one IP from your logs and prints True or False:
bash
IP=203.0.113.7 # replace with an IP from your logs
curl -sL https://search.developer.apple.com/applebot.json \
| python3 -c "import json,sys,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(sys.stdin); print(any(ip in ipaddress.ip_network(p['ipv4Prefix']) for p in d['prefixes'] if 'ipv4Prefix' in p))" "$IP"
Apple's page links the file over plain HTTP, which redirects to HTTPS, hence -L. Do not test with the address from Apple's DNS example: 17.58.101.179 was not in the file when we checked, so the two methods can disagree for an individual IP. The same check for other vendors is in how to verify a real AI crawler.
What the user agent looks like
Apple's desktop example for search crawling:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
There is also a mobile variant built on an iPhone Safari string. Apple says it occasionally updates the advertised browser version, so match on the Applebot token, not the full string.
How to check what Applebot sees on your site
Read your live robots.txt: curl -s https://example.com/robots.txt. Look for groups naming Applebot, Applebot-Extended and Googlebot, and remember the Googlebot fallback.
Check whether a platform added rules for you. With Bot Preference Sync turned on and at least one AI crawler policy set to block or disallow, Cloudflare's managed robots.txt adds a Disallow: / group for Applebot-Extended. Squarespace's "Block known artificial intelligence crawlers" setting, off by default, also lists Applebot-Extended. Neither list names Applebot itself, though edge bot rules are a separate thing to check.
Find Applebot in your logs and check status codes:
Verify a few of the top IPs with the DNS check above. Many 403 or 503 responses to verified Apple IPs point at a firewall rule, a bot rule or an overloaded server. The $9 field assumes the common combined log format.
Do not block JavaScript or CSS in robots.txt. Apple says Applebot "may render the content of your website within a browser" and that blocked JS, CSS or XHR resources can stop it rendering properly. Even so, the safest pages have their main content in the server HTML, for reasons covered in do AI crawlers run JavaScript?
Frequently asked questions
Does disallowing Applebot-Extended remove my site from Siri or Spotlight?
No. Apple says pages that disallow Applebot-Extended can still be included in search results, and that Applebot-Extended rules are not considered in Search ranking.
Why don't I see Applebot-Extended in my server logs?
Because it never makes requests. Apple says Applebot-Extended does not crawl webpages; it only decides how data crawled by Applebot may be used.
Does blocking Applebot-Extended keep my content out of AI answers in Siri?
Not according to Apple's documentation. Applebot-Extended covers training of Apple's foundation models. For AI-generated answers, Apple points to the nosnippet directive, or isAccessibleForFree: false for paywalled pages.
Does Applebot follow my Googlebot rules?
Only when your robots.txt does not mention Applebot. In that case Apple says Applebot follows the Googlebot instructions, so add an Applebot group if you want different rules.
Does Applebot honor crawl-delay?
No. Apple says Applebot does not follow crawl-delay and instead slows down automatically when a site responds slowly or returns errors.
Next step
To see what your live robots.txt says to Applebot, Applebot-Extended and the other named AI crawlers, run the free single-page check. It needs no account; if you later want every template on the site covered, the full audit goes up to 500 pages.
Filed under AI crawlers. Topics: · applebot, robots-txt, ai-crawlers, meta-robots
How nosnippet, max-snippet, data-nosnippet and noindex control AI Overviews and AI Mode, what other AI tools document, and how to set X-Robots-Tag for PDFs.
Google-Extended is a robots.txt token, not a crawler. It covers Gemini training and grounding, not Google Search rankings or AI Overviews. Here is how to use it.