About CrawlitoBot

When it visits

Only when a person asks Crawlito to check a site. A free check fetches one page. A full-site audit is ordered by someone who confirms they own the site or are allowed to audit it, and covers up to 500 pages.

It does not crawl the web on its own, build a search index, or collect content to train AI models.

What it fetches

robots.txt first, before anything else. Then the requested pages as plain HTML without running JavaScript, plus llms.txt and the XML sitemap when they exist.

To compare what people see with what crawlers receive, a second pass renders the same pages in a headless Chromium browser. That browser does not send the CrawlitoBot user agent. It is told to skip the paths you disallow for CrawlitoBot, and anything it returns from a disallowed page is discarded: never stored, scored or included in a report.

During an audit, requests are spaced at least one second apart per host, or longer if you set a Crawl-delay for CrawlitoBot. Each host's robots.txt is read before anything else on that host, including www and non-www versions of your domain.

How robots.txt applies

Rules that name CrawlitoBot are always followed. If your robots.txt disallows the requested page, Crawlito stops before fetching it. Pages you disallow are skipped during an audit, and redirects into disallowed pages are not followed.

Rules for all bots (User-agent: *) do not stop a check. Every visit is requested by a person looking at that specific site, and finding out whether a site blocks crawlers is the point of the check.

Opt out or slow it down

To stop CrawlitoBot from fetching any page of your site, add this to robots.txt. Crawlito then reads only robots.txt and stops; the rendering browser is never started:

User-agent: CrawlitoBot
Disallow: /

To keep checks but space requests further apart (in seconds, up to 30):

User-agent: CrawlitoBot
Crawl-delay: 5

Changes apply to the next check, because robots.txt is read fresh at the start of every visit.

Questions

See how Crawlito handles data on the privacy page, or contact Crawlito through the audit form.