Microsoft Clarity bot activity: see AI crawler requests
How Microsoft Clarity bot activity shows AI crawler requests from your CDN logs: setup, what each card means, provider costs, limits and how to disconnect.
By Crawlito · 9 min read
In short
- Clarity's Bot Activity dashboard only fills after you connect a server or CDN log source: Cloudflare, Amazon CloudFront, Fastly, Akamai, Azure Front Door, or the latest Clarity WordPress plugin.
- Microsoft does not charge for AI Visibility or Bot Activity, but the log pipeline can cost money at your provider, for example Cloudflare Logpush or Amazon Data Firehose.
- The dashboard is read-only: Microsoft says it does not block bots or manage access, so changes still happen in robots.txt, your CDN or your firewall.
- Microsoft states that a bot request does not mean your content was retrieved, grounded, cited or shown in an AI answer.
- Disconnecting in Clarity alone does not stop the logs; you also have to delete the Logpush job, logging endpoint or Firehose stream, or deactivate the DataStream, at the provider.
On this page
Microsoft Clarity's Bot Activity dashboard, under AI Visibility, shows which AI crawlers and other verified bots request your pages, what share of all requests they make, and which paths they hit most. It does not work from the normal Clarity tracking script. You connect a server or CDN log source (Cloudflare, Amazon CloudFront, Fastly, Akamai, Azure Front Door, or WordPress through the Clarity plugin), and Clarity reads the request logs. Clarity itself is free for this, but your provider may bill you for sending the logs.
This guide covers what the dashboard reports, how to switch it on, what it costs at each provider, what it cannot tell you, and how to switch it off cleanly. If you already have raw logs and want to count crawlers yourself, the server log guide covers that route.
Why Clarity needs your CDN logs
Clarity is best known for session recordings and heatmaps, which rely on a JavaScript tag running in a visitor's browser. Most AI crawlers fetch HTML without running that tag, so they never show up in tag-based analytics. The two kinds of AI traffic guide explains why crawler visits and human referral visits have to be measured separately.
To see crawlers, Clarity has to read the request logs that your server or CDN already produces. Microsoft's disconnection guide describes the mechanism plainly: each integration forwards request-level traffic logs from your platform or CDN to Clarity, which uses them to separate bots and AI crawlers from human activity.
That makes Bot Activity useful for one group in particular: marketers and site owners whose site sits behind a supported CDN but who have no shell access, no log tooling and no wish to run grep every month.
What the Bot Activity dashboard shows
Microsoft's Bot Activity overview lists four main metrics. Later Clarity blog posts describe more cards. Here is what each one means, according to Microsoft's own descriptions:
| Card | What it reports |
|---|---|
| Bot operator | Bot requests broken down by operator and by individual bot. By default it focuses on AI-related bots, and a toggle switches to all identified bots |
| AI request share | The percentage of all requests, human and automated, that come from AI bots |
| Bot activity | Bot requests grouped by the bot's primary purpose, such as search, data services, developer tools or AI assistants |
| Path requests | The pages and resources bots request most, including HTML pages, images, JSON endpoints and XML files |
A May 2026 update added AI bot request totals, the share of site pages that bots access, a request status view that groups bot requests as successful, redirected or unsuccessful, a view that flags bots requesting paths disallowed in robots.txt, domain and path filters, and a separate Content Type card (content type moved out of the Path requests table). In August 2026 Clarity added an AI Scrape-to-Referral Ratio card that compares AI scrape activity with referral visits, to show whether AI platforms send visitors back or only fetch content. Clarity does not publish the formula; it says the ratio is only calculated across domains that are mapped correctly between your CDN data and your Clarity project.
The request status view is the most practical card for AI search readiness. A search crawler that mostly gets unsuccessful responses cannot read your pages, however often it tries.
Clarity's own pages use slightly different names for the same area: "Bot Activity" in the documentation, and "AI Bot Activity" or "Bot Analytics" in blog posts. At the time of writing the docs place it under Dashboards > AI Visibility > Bot Activity.
Robots.txt violations: what the card means
A violations view first appeared in the May 2026 update, and a June 2026 post describes robots.txt violation detection in more detail. Clarity checks bot requests against your site's robots.txt directives and splits the traffic into compliant and non-compliant requests. The Violations card shows violations as a percentage of total bot requests, with a trend line, filters by operator, bot name and activity type, and the URLs, paths and content types the violating bots requested. The post describes monitoring only: it does not say Clarity blocks anything.
Read the card with two cautions:
- Some fetchers are not meant to follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because both fetch a page when a person asks for it. A request from them to a disallowed path is not the same as a training crawler ignoring your rules.
- The card reports, it does not enforce. robots.txt is advisory. If a bot keeps fetching a path you disallowed, the fix is a block at your CDN or firewall, not a stricter robots.txt line.
If the violations concern a crawler you never meant to block, check your robots.txt first. A broad rule like the one below catches every crawler that reads it, including AI search crawlers you may want:
User-agent: *
Disallow: /How to set up Bot Activity
Microsoft's overview gives the path at the time of writing:
- Sign in to Clarity and open your project.
- Go to Settings > AI Visibility.
- Connect a supported server or CDN. The onboarding flow lists the supported providers and upcoming integrations.
- When setup is complete, open Dashboards > AI Visibility > Bot Activity.
Clarity's blog says a project admin makes the connection. On WordPress, the integration is enabled by default when you use the latest Clarity WordPress plugin, and older plugin versions must be updated first. Clarity's FAQ says insights appear once the connection is set up and data begins flowing. Microsoft does not publish a fixed delay.
What each connection creates on your side, as Microsoft's disconnection guide describes it:
| Provider | What the integration sets up |
|---|---|
| Cloudflare | A Logpush job for HTTP request logs |
| Amazon CloudFront | An Amazon Data Firehose delivery stream fed by CloudFront standard access logs, plus an S3 bucket for failed deliveries |
| Fastly | An HTTPS logging endpoint on your Fastly service |
| Akamai | A DataStream log delivery to an HTTPS endpoint |
| Azure Front Door | A diagnostic setting that sends logs to Event Hubs, plus Clarity resources that forward them |
| WordPress | Request data sent from the Clarity plugin itself |
Clarity's May 2026 update says Cloudflare support now extends to non-enterprise plans: Free, Pro and Business. If your site is on CloudFront and you also keep origin logs, the CloudFront user agent guide explains how what reaches your origin can differ from what the edge saw.
What it costs at each provider
Microsoft's cost considerations page says Clarity does not charge for AI Visibility or the Bot Activity dashboard. The costs come from your server, CDN or cloud provider. It lists these cost sources:
| Provider | Possible costs Microsoft lists |
|---|---|
| WordPress | No direct cost from Clarity, typically negligible; depends on your hosting plan and traffic |
| Cloudflare | Your plan tier, Logpush availability and usage, volume of logs delivered |
| Amazon CloudFront | CloudFront requests, Data Firehose ingestion, delivery to HTTP endpoints, S3 storage for failed delivery logs |
| Fastly | Request volume, log delivery to HTTPS endpoints, data transfer |
| Akamai | No additional cost beyond your existing service plan |
| Azure Front Door | Event Hubs ingestion and throughput units, Container Apps compute, diagnostic log export, a storage account for checkpoints |
One detail has moved since Microsoft wrote that page in May 2026. It says Logpush is typically a paid-plan feature. Cloudflare announced on 30 September 2026 that Logpush is available on Free, Pro, Business and Enterprise with usage-based pricing: 25 GB of external exports per account per month included, then $0.10 per GB. Clarity is an external destination, so a busy site can pass that allowance. Check the current price at each provider before you connect, and set a billing alert if your provider offers one.
What Bot Activity cannot tell you
Microsoft is unusually direct about the limits. The overview says bot activity represents requests made to your site and does not indicate that content was retrieved, grounded, cited or surfaced in AI-generated responses. Treat the dashboard as an access signal, not as an outcome.
Other limits to keep in mind:
- It does not block anything. Clarity's launch post says the feature does not block bots or manage access directly. Changes still happen in robots.txt, your CDN or your firewall.
- Verification is not documented. Microsoft calls the bots "verified" and says the data comes from real server-side logs, but it does not publish how it confirms that a request claiming to be GPTBot really came from OpenAI. If a decision depends on it, check a sample of IPs against the operator's published list, as in the crawler verification guide.
- You only see what the connected source sees. Clarity reads the logs you forward, so a subdomain served elsewhere, or a second CDN, is not included unless you connect it too. Microsoft's scrape-to-referral post itself mentions cases where CDN coverage and Clarity coverage do not fully align.
- Citations are a separate report. Clarity's FAQ treats AI Citations as a different view from Bot Activity, and Bing Webmaster Tools has its own AI Performance report. Keep those numbers apart from request counts.
If you report on AI search to a client or a manager, put Bot Activity in the access section of the report, next to status codes, as in the AI search reporting dashboard.
How to disconnect Bot Activity
This step is easy to get wrong. Microsoft's disconnection guide says removing the integration in the Clarity dashboard only removes the Bot Activity dashboard. It does not stop data from flowing into Clarity by itself, and Microsoft's cost page says that to avoid provider costs you must disconnect at the server or CDN level. You also have to remove the integration at the provider:
| Provider | What to remove |
|---|---|
| Cloudflare | In Analytics & Logs > Logpush, delete the Logpush job created for Clarity |
| Amazon CloudFront | Remove the Firehose destination from the distribution's logging settings, delete the delivery stream, and optionally the S3 bucket for failed deliveries |
| Fastly | Clone the active version, delete the HTTPS logging endpoint for Clarity, save, then activate the new version |
| Akamai | In DataStream, deactivate the stream |
| Azure Front Door | Delete the forward-to-eventhub diagnostic setting, then remove the four Clarity AI Visibility resources |
| WordPress | Deactivate and delete the Microsoft Clarity plugin |
Two Fastly and WordPress details matter. Fastly uses versioned configurations, so the change does nothing until you activate the new version. On WordPress, Microsoft says the only way to stop Bot Activity data at the time of writing is to remove the plugin, which stops all Clarity data collection and removes the Clarity project linked to the site. Historical data already in Clarity is not affected by disconnecting.
Frequently asked questions
Is Microsoft Clarity bot activity free?
Clarity does not charge for AI Visibility or Bot Activity. The log pipeline behind it can cost money at your CDN or cloud provider, depending on plan, traffic and region.
Can I use Bot Activity without a CDN?
Only on WordPress, through the latest Clarity plugin. Other sites need one of the supported integrations: Cloudflare, Amazon CloudFront, Fastly, Akamai or Azure Front Door.
Does a high AI request share mean AI tools cite my site?
No. Microsoft says bot requests do not indicate that content was retrieved, grounded or cited. Use citation reports such as Bing Webmaster Tools' AI Performance report for that question.
Can Clarity block AI crawlers that ignore robots.txt?
No. Clarity reports robots.txt violations but does not block or manage access. Use your CDN or firewall to block a crawler.
Why does disconnecting in Clarity not stop the data?
The logs are pushed from your provider, so the job, stream or endpoint keeps running until you delete it there. Microsoft's disconnection guide lists the step for each provider.
Next step
Bot Activity tells you which crawlers came and whether they got a successful response. To see whether a specific page is readable for the main AI crawlers right now, run the free single-page check, which needs no account.
Filed under Strategy and audits. Topics: analytics, ai-crawlers, server-logs, measurement