Bingbot and AI search: why Bing's crawler still matters

Bingbot matters for AI search because Microsoft's own AI answers are built on Bing. Microsoft says Copilot and Copilot Chat ground responses in current web information "via the Bing search service", and it documents no separate Copilot crawler. A page Bingbot cannot crawl, or that Bing has not indexed, has no documented route into those answers. Blocking Bingbot by accident, often through a firewall rule or a robots.txt group you forgot about, is easy to miss if you never look at Bing.

This post covers what Microsoft actually documents about the link between Bing and Copilot, how to check that Bingbot can reach you, how to verify a real Bingbot, and what Bing Webmaster Tools and IndexNow add. All sources were read on 8 November 2026.

How Microsoft Copilot uses the Bing index

Microsoft documents two connections between its AI products and Bing:

• Microsoft Copilot and Copilot Chat. Microsoft's enterprise data protection page says that when these products need current web information, the prompt is turned into a short query of a few words and sent to the Bing search service, with user and tenant identifiers removed. Microsoft adds that these queries are not shared with advertisers or used to train foundation models.
• Copilot Studio agents. In Copilot Studio, the Web Search setting ("Use information from the web") searches "all public websites indexed by Bing" through Grounding with Bing Search. The Public website knowledge source also runs the query on Bing, then returns results only from the sites the agent's builder listed.

The second point is easy to miss. A business building a support or sales agent in Copilot Studio and pointing it at your public site still depends on Bing having indexed that site.

Be careful about stretching this further. Microsoft's pages document Copilot and Copilot Studio. Do not assume another assistant uses Bing unless its operator says so in its own documentation. OpenAI, Anthropic and Perplexity document their own crawlers; GPTBot, OAI-SearchBot and ChatGPT-User explains OpenAI's.

There is no separate Copilot crawler

Microsoft does not document a Copilot-specific crawler or robots.txt token. Bing's list of the crawlers it uses names these:

None of the Microsoft pages read for this post says that robots.txt rules control Copilot separately from Bing. So the practical rule is simple: if you want Copilot to be able to use your pages, Bingbot needs to reach them. Some robots.txt templates include names like BingAI; that name does not appear in Bing's crawler list, so do not rely on a rule for it.

How to check that Bingbot can reach your site

Work through these in order. Each one is a common cause of Bing silently dropping pages.

1. Read your live robots.txt. Fetch the file the world sees, not the one in your repository:

   bash

   curl -s https://www.example.com/robots.txt

   Look for a User-agent: bingbot group, and for Disallow: / under User-agent: *, which blocks Bingbot when no Bingbot group exists.

2. Check for the named-group trap. Bing's robots.txt guide says that when Bingbot finds a group addressed to itself, it ignores the directives in the * group. That is also how RFC 9309 works. The result catches people out:

   User-agent: *
   Disallow: /admin/
   Disallow: /cart/
   
   User-agent: bingbot
   Crawl-delay: 5

   Here Bingbot follows only its own group, so /admin/ and /cart/ are open to it. Repeat the general rules inside the Bingbot group:

   User-agent: bingbot
   Crawl-delay: 5
   Disallow: /admin/
   Disallow: /cart/
   
   User-agent: *
   Disallow: /admin/
   Disallow: /cart/

   The robots.txt groups guide covers this rule for every crawler.

3. Request a page with Bingbot's user agent. Bing documents this string with Chrome/W.X.Y.Z, where the version tracks the Microsoft Edge version Bing renders with; 131.0.0.0 below is only a placeholder:

   bash

   curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/131.0.0.0 Safari/537.36" https://www.example.com/

   A 403 or a challenge page where a normal browser gets a 200 usually means a rule keyed on the user agent is in the way. A 200 from your own machine is not proof, because firewalls often judge by IP address too.

4. Look at your logs. Count Bingbot requests and their status codes:

   bash

   grep -i "bingbot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

   This assumes the combined log format, where the status code is the ninth field, and it counts anything claiming to be Bingbot. Many 403 or 429 responses to IPs you have verified as Bingbot (see the next section) point at a WAF or rate-limit rule. A complete absence of Bingbot over weeks is worth investigating too.

5. Check your CDN or WAF. Bot protection that challenges unfamiliar traffic can catch search crawlers. If you use Cloudflare, the Cloudflare AI crawler settings guide lists the settings to review.

How to verify a real Bingbot

Anyone can send bingbot in a user-agent header. Bing gives three ways to confirm a request is genuine.

Reverse and forward DNS

Bing's verification guide says to run a reverse DNS lookup on the IP, check that the hostname ends in search.msn.com, then run a forward lookup on that hostname and confirm it returns the same IP:

host 157.55.39.10
host <hostname-from-the-first-lookup>

The first command must return a name ending in .search.msn.com, and the second must return 157.55.39.10 again. A hostname that ends in anything else, or a forward lookup that returns a different IP, means the request is not from Bingbot.

The published IP list

Bing publishes Bingbot's IPv4 ranges at bingbot.json, in the same prefixes and ipv4Prefix format other operators use. Bing says the list can change at any time and should be refreshed daily, so fetch it on a schedule rather than copying ranges by hand. This checks one IP:

IP=157.55.39.10
curl -s https://www.bing.com/toolbox/bingbot.json \
  | python3 -c "import json,sys,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(sys.stdin); print(any(ip in ipaddress.ip_network(p['ipv4Prefix']) for p in d['prefixes'] if 'ipv4Prefix' in p))" "$IP"

The Verify Bingbot tool

Bing also offers a Verify Bingbot tool, both publicly and inside Bing Webmaster Tools, where you paste an IP and get an answer. The same approach for other vendors is in how to verify a real AI crawler.

How to slow Bingbot down without blocking it

If Bingbot puts real load on your server, slow it rather than shutting it out. Bingbot supports Crawl-delay; Bing's own example is:

User-agent: bingbot
Crawl-delay: 1

Bing Webmaster Tools also has Crawl Control, which sets Bingbot's crawl rate for each hour of the day. Bing says a Crawl-delay in robots.txt always takes precedence over the Crawl Control setting, so if both are set, the robots.txt value wins. Remember the named-group rule above when you add the line, and note that Google does not support Crawl-delay at all. More on how each crawler treats it in Crawl-delay and rate limits for AI crawlers.

Bing says search engines keep a cached copy of robots.txt for at least a few hours, so allow that long before judging a change.

What Bing Webmaster Tools tells you about AI answers

Bing Webmaster Tools is worth setting up even if Bing sends you little direct search traffic, because it is where Microsoft reports how often your site is cited in its AI answers.

On 10 February 2026, Bing launched AI Performance as a public preview. It shows how often your site is cited across Microsoft Copilot, AI-generated summaries in Bing and select partner integrations. The report includes:

• Total Citations and Average Cited Pages
• Grounding Queries, a sample of the key phrases the AI used to retrieve your content
• Page-level citation counts and trends over time

None of these metrics is click-based, and the grounding queries are a sample. The report covers Microsoft surfaces only, not ChatGPT, Perplexity or Google. On 16 June 2026 Bing added four preview features: Intents, Topics, Citation Share and Compare. Treat all of it as a preview that may change. For referral traffic and crawler visits outside Microsoft, see how to measure AI crawler traffic.

How IndexNow helps Bing find changes faster

IndexNow lets you tell search engines that a URL was added, updated or deleted, instead of waiting for the next crawl. It works in three steps.

1. Create a key. It must be 8 to 128 characters using only a-z, A-Z, 0-9 and dashes. For example: openssl rand -hex 16.

2. Host the key as a UTF-8 text file named {key}.txt at your site root, containing the key itself. You can host it elsewhere on the same host and point to it with keyLocation, but then it only covers URLs under that path.

3. Submit changed URLs. One URL with a GET request. The docs say the url value must be URL-encoded, which --data-urlencode handles:

   bash

   curl -G "https://api.indexnow.org/indexnow" \
     --data-urlencode "url=https://www.example.com/pricing" \
     --data-urlencode "key=YOUR_KEY"

   Or up to 10,000 URLs in one JSON POST:

   bash

   curl -X POST https://api.indexnow.org/indexnow \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{
       "host": "www.example.com",
       "key": "YOUR_KEY",
       "urlList": [
         "https://www.example.com/pricing",
         "https://www.example.com/blog/new-post"
       ]
     }'

The response code tells you what happened:

A 200 only means the engine received the URL, not that it crawled or indexed it.

Participating engines share submissions with each other, so one submission to any endpoint is enough. At the time of writing, the IndexNow engine list includes Bing, Yandex, Seznam, Naver, Yep, Internet Archive and Amazonbot. Google is not on it.

Keep expectations modest. The IndexNow FAQ says submission does not guarantee immediate indexing, each engine sets its own daily limits, and you should only submit URLs whose content changed. IndexNow does not replace a sitemap, either: keep a valid XML sitemap listed in robots.txt and submitted in Bing Webmaster Tools.

Can you keep content out of Copilot answers but stay in Bing?

Bing documented page-level controls for this on 22 September 2023, when its chat product was called Bing Chat. In its announcement, Bing said:

• noarchive: content is not included in Bing Chat answers or linked from them, and is not used to train Microsoft's generative AI foundation models.
• nocache: content may appear in chat answers, but only the URL, title and snippet are shown, and only those may be used in training.
• Pages with either tag still appear in Bing search results. If both are present, Bing treats the page as nocache.

Bing's post gives two forms. The first applies to every crawler that reads the robots meta tag, so other search engines will also see noarchive:

<meta name="robots" content="noarchive, nocache">

The second keeps noarchive general and sends nocache to Bing only:

<meta name="robots" content="noarchive">
<meta name="bingbot" content="nocache">

That post predates the Copilot name, and no current Microsoft page read for this post restates these rules for Copilot, so check Bing's documentation before relying on them. Blocking Bingbot in robots.txt is the blunt option: it stops Bing search crawling too, and because Copilot grounds its web answers in Bing, it very likely cuts Copilot off from those pages as well.

Frequently asked questions

Does blocking Bingbot affect Microsoft Copilot?

Microsoft does not say so in those words, but it is very likely. Microsoft says Copilot grounds its web answers via the Bing search service and documents no separate Copilot crawler, so a page Bingbot cannot crawl has no documented route into Copilot's web answers.

Is there a robots.txt token for Copilot?

No. Bing's crawler list names Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview, and none of them is Copilot-specific. Rules for names like BingAI do not match any documented Microsoft crawler.

Does Bingbot obey Crawl-delay?

Yes. Bing's own robots.txt guide gives a Crawl-delay example for Bingbot, and Bing says the robots.txt value takes precedence over the Crawl Control setting in Bing Webmaster Tools.

Does IndexNow work for Google?

No. Google is not listed as an IndexNow participant. Submissions reach Bing and the other engines on the IndexNow list, and they still do not guarantee indexing.

Can I see when Copilot cites my pages?

Partly. Bing Webmaster Tools' AI Performance report, a public preview since February 2026, counts citations across Copilot, Bing AI summaries and select partners. It does not report clicks.

Next step

If you are not sure whether your robots.txt, firewall and CDN treat Bingbot the way you intend, run the free single-page check; it needs no account. For a whole site, a full audit covers up to 500 pages and comes with evidence and prioritized fixes.