AdsBot-Google robots.txt: why User-agent: * does not apply
AdsBot-Google and Mediapartners-Google skip the User-agent: * group in robots.txt. How to name them, verify them by IP, and keep ad landing page checks working.
By Crawlito · 7 min read
In short
- Google says AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google and APIs-Google ignore the global User-agent: * group and follow only rules that name them.
- Google-Safety, which Google uses for abuse-specific crawling such as malware discovery, ignores robots.txt rules and is not affected by crawling preferences.
- Google Ads disapproves ads under its Destination not crawlable policy when robots.txt disallows AdsBot or cannot be read, and AdSense says a blocked crawler means Google cannot serve ads.
- Special-case crawlers come from the IP ranges in special-crawlers.json, and their reverse DNS names follow rate-limited-proxy----.google.com, not googlebot.com.
- A firewall allowlist built only from Googlebot's common-crawlers.json can block AdsBot even when robots.txt allows it.
On this page
AdsBot-Google and Mediapartners-Google do not read the User-agent: * group in your robots.txt. Google documents them as special-case crawlers that follow only rules addressed to them by name, so a Disallow you wrote for everyone does not stop them, and they keep visiting pages you thought were closed. If you want to block them, name them. If your ads are disapproved as not crawlable, Google's help points to the opposite problem: a named rule, an unreadable robots.txt or a firewall that stops AdsBot from reaching the landing page.
This post covers the Google ads crawlers specifically: which tokens they use, what blocking them does to Google Ads and AdSense, and how to verify and allow them at the edge. The general rule that a named group replaces * is explained in robots.txt user-agent groups. Here the twist is that these crawlers skip * even when no group names them.
Which Google crawlers ignore User-agent: *
Google splits its crawlers into common crawlers (Googlebot and friends), special-case crawlers and user-triggered fetchers. Its special-case crawlers page says these serve products that have an agreement with the crawled site, may ignore robots.txt rules, and operate from a different IP range than the common crawlers. At the time of writing it lists five active ones:
| Crawler | robots.txt token | Behaviour with * | Used for |
|---|---|---|---|
| AdsBot | AdsBot-Google | Ignores * | Google Ads web page ad quality checks |
| AdsBot Mobile Web | AdsBot-Google-Mobile | Ignores * | Google Ads web page ad quality checks |
| Mediapartners-Google | Mediapartners-Google | Ignores * | AdSense and Google Ad Manager ad serving |
| APIs-Google | APIs-Google | Ignores * | Delivering push notification messages from Google APIs |
| Google-Safety | none | Ignores robots.txt entirely | Abuse crawling, such as malware discovery for links posted on Google properties |
Google's wording for AdsBot is that it "ignores the global robots.txt user agent (*) with the ad publisher's permission". Google describes every special-case crawler as serving a product where there is an agreement between the crawled site and the product about the crawl process. The page does not describe a separate opt-in step beyond using the product.
Google-Safety is the outlier. Google says it ignores robots.txt and is unaffected by crawling preferences, so no robots.txt line will change its behaviour. Since it looks for malware and abuse, there is little reason to want it blocked.
The user agent strings you will see in logs
AdsBot-Google (+http://www.google.com/adsbot.html)
Mediapartners-GoogleThe mobile AdsBot uses an Android Chrome-style string that ends in (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html), and mobile Mediapartners-Google requests end in (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html). Neither token contains the word Googlebot, so a log filter for Googlebot alone will miss them. Our list of Googlebot user agent strings covers the common crawlers for comparison.
How to block AdsBot-Google and Mediapartners-Google
Because they skip *, this file does not keep either crawler out of /members/:
User-agent: *
Disallow: /members/To block them, give them their own group. Google lists AdsBot-Google and AdsBot-Google-Mobile as separate tokens, so name both:
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
Disallow: /members/
User-agent: Mediapartners-Google
User-agent: Google-Display-Ads-Bot
Disallow: /members/The second group follows the AdSense crawler article, which says that pages blocked only under User-agent: * are still crawled if they show ads, and that to keep the AdSense crawler out you need to address both Mediapartners-Google and Google-Display-Ads-Bot. AdSense uses Google-Display-Ads-Bot to verify your site when you add it to AdSense. The same article says the AdSense crawler only requests URLs where your ad tags are implemented, and that it also requests pages that redirect, because it must reach the original URL to see the redirect.
Before you add Disallow: / for any of them, read the next section. Blocking these crawlers is allowed, but it has direct consequences for ads.
What blocking them does to your ads
Google Ads: Destination not crawlable
Google Ads checks landing pages with AdsBot. Its Destination not crawlable policy says destinations and their content must be crawlable by AdsBot so Google can confirm users land on a page relevant to the ad. Ads that fail are disapproved. The policy names these causes:
- robots.txt disallows access, so AdsBot cannot crawl the site
- robots.txt is unreachable or times out, so AdsBot cannot read it
- site settings that allow less crawling than the number of ads requires, including a large batch of new ads at once
Google's fix list includes making sure robots.txt does not block most of the site, splitting large submissions into smaller batches over several days, and editing the ad so it is reviewed again, which Google says usually happens within 24 to 48 hours. The policy also says a violation does not lead to immediate suspension: Google gives at least 7 days' warning.
For Dynamic Search Ads, Google's uncrawlable landing pages guide shows the error as "Disapproved: Page cannot be crawled" and adds checks that are easy to miss:
- A subdomain has its own robots.txt, and it must also allow AdsBot-Google.
- Search the page source for
AdsBot-Google. A robots meta tag naming it withnoindexblocks the page. - Landing pages must be public, not behind a sign-in.
- Pages must reach their destination in fewer than 10 redirects.
- A 403, 404, 500 or DNS error means the server is blocking access, and Google notes that firewalls or other security settings, with Cloudflare as its example, can block its crawlers even when the page loads for you.
AdSense and Ad Manager
The AdSense robots.txt access article is short: if robots.txt blocks the AdSense crawler, Google cannot serve ads on your site. To restore access, remove the User-agent: Mediapartners-Google and Disallow: / lines. AdSense notes that changes are only picked up when the crawler visits again, and the crawler article says page changes may take up to one or two weeks to show in its index. It also says fixing AdSense crawl issues has no effect on your placement in Google Search results.
How to verify a request really comes from AdsBot
User agents are easy to fake, so a rule that trusts the string AdsBot-Google lets anyone through. Google's verification page gives each crawler type its own reverse DNS pattern and IP file:
| Crawler type | Reverse DNS pattern | IP range file |
|---|---|---|
| Common crawlers (Googlebot) | crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com | common-crawlers.json |
| Special-case crawlers (AdsBot, Mediapartners-Google) | rate-limited-proxy-***-***-***-***.google.com | special-crawlers.json |
| User-triggered fetchers | ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com | user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json |
The manual check is the same for all of them: reverse lookup, confirm the domain, then forward lookup and confirm it returns the original IP. The address below is a documentation placeholder; use one from your logs.
host 203.0.113.10
# expect a name ending in .google.com, e.g. rate-limited-proxy-...google.com
host rate-limited-proxy-203-0-113-10.google.com
# must return 203.0.113.10For automated checks, use the published file. It lists IPv4 and IPv6 prefixes in CIDR notation and carries a creationTime field, so you can see when Google last regenerated it:
curl -s https://developers.google.com/static/crawling/ipranges/special-crawlers.json \
| jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'The practical trap: a firewall rule for "verified Google" built only from common-crawlers.json or from a googlebot.com reverse DNS check will not match AdsBot, because it resolves to google.com and comes from the special-crawlers ranges. The result is a landing page that Googlebot can reach and AdsBot cannot. Our guides to verifying real crawlers and AI crawler IP ranges for firewall allowlists cover the same method for other operators.
How to check your setup
Work through these for each landing page or ad-bearing section:
- Read the live robots.txt on the exact host the ad points to, including subdomains:
curl -s https://shop.example.com/robots.txt. - Search it for the four tokens:
AdsBot-Google,AdsBot-Google-Mobile,Mediapartners-GoogleandGoogle-Display-Ads-Bot. If none appears, AdsBot and Mediapartners-Google have no robots.txt rules that apply to them, whatever your*group says. - If a token appears, apply the longest-match rule within that group to your landing page path, as described in the groups guide.
- Check the response robots.txt itself returns. A timeout or server error on
/robots.txtis one of the causes Google Ads names. - Search your access logs for the tokens and look at the status codes. A run of 403 or 429 responses to verified special-crawler IPs points at a WAF or rate limit, not robots.txt.
- Confirm the IPs against
special-crawlers.jsonbefore drawing conclusions from any log line.
How OAI-AdsBot compares
OpenAI has its own ads crawler, OAI-AdsBot, which OpenAI says validates the safety of pages submitted as ads on ChatGPT, only visits those pages, and has its own IP list at https://openai.com/adsbot.json. The documented difference is robots.txt: Google states that AdsBot ignores * and must be named, while OpenAI's crawler page does not say how OAI-AdsBot treats robots.txt at the time of writing. Both companies document their ads crawler separately from the crawlers they use for search. ChatGPT ads vs organic sources covers the OpenAI side in detail.
Frequently asked questions
Does Disallow under User-agent: * block AdsBot-Google?
No. Google says AdsBot-Google and AdsBot-Google-Mobile ignore the global * group. Only a group that names them applies.
Does blocking AdsBot affect my Google Search rankings?
Google lists Google Ads as the product affected by AdsBot rules, and Googlebot is the crawler for organic results. For AdSense, Google separately says fixing crawl issues has no impact on placement in Google Search results.
Can I block Google-Safety in robots.txt?
No. Google says Google-Safety ignores robots.txt rules and is not affected by crawling preferences. It is used for abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.
Why does AdsBot fail my firewall's Googlebot check?
Special-case crawlers resolve to rate-limited-proxy-***-***-***-***.google.com and use the ranges in special-crawlers.json. A rule that only trusts googlebot.com names or common-crawlers.json will not recognise them.
Should I allow these crawlers on pages without ads?
Mediapartners-Google only requests URLs where your ad tags run, and AdsBot checks ad destinations. If a page has no ads and is not a landing page, these crawlers have no documented reason to visit it.
Check the rest of your crawler access
Ads crawlers are one row in a larger table of who can reach your pages. Run the free check on a landing page to see whether the main search and AI crawlers can fetch it, or ask for an audit if you want robots.txt and edge rules reviewed across the site with evidence for each fix.
Filed under AI crawlers. Topics: google, robots-txt, ai-crawlers, waf