Framer AI crawlers: robots.txt, sitemap and markdown
Can ChatGPT, Perplexity and Google read a Framer site? What Framer's generated robots.txt and sitemap do, how to replace them, and how to test markdown.
By Crawlito · 7 min read
In short
- Framer says every page is pre-rendered to HTML on its servers, so crawlers that do not run JavaScript still receive the page text, title, meta description and any JSON-LD.
- Framer generates robots.txt and sitemap.xml for every published site, and says the generated robots.txt allows all AI and search crawlers by default.
- Framer's help center says you can replace the generated robots.txt by uploading your own file under Site Settings > Hosting > Files, on Pro and Enterprise plans.
- Framer serves a markdown version of optimized pages at the same URL: add the md parameter, or send an Accept: text/markdown header.
- Framer says a page marked noindex is respected, but it does not document whether such a page is removed from the generated sitemap, so check yours.
On this page
- What does Framer generate for AI crawlers by default?
- Does Framer pre-render HTML for crawlers that skip JavaScript?
- What does the generated robots.txt contain, and can you change it?
- What does page-level noindex do for AI search?
- How do the sitemap and llms.txt work on Framer?
- How do the markdown versions work?
- How can you test a Framer page with curl?
- Frequently asked questions
- Check your own Framer page
Framer says AI crawlers can read a standard Framer site. It says every page is pre-rendered to HTML on its servers, so the text is in the first response even for crawlers that do not run JavaScript, and its generated robots.txt allows all AI and search crawlers by default. You can replace that robots.txt with your own file on a Pro or Enterprise plan, and Framer can also serve a markdown version of each optimized page.
This post follows Framer's help center at the time of writing (November 2026). Menu names and plan limits change, so match on the setting name if your dashboard looks different. Nothing here guarantees that an assistant will cite your site; it covers what Framer documents about whether crawlers can reach and read it.
What does Framer generate for AI crawlers by default?
Framer's article Make your site readable by AI agents lists four defaults for every published site:
| Default | What Framer says |
|---|---|
| Pre-rendered HTML | Every page is rendered to HTML on Framer's servers before it is served |
| robots.txt | Generated automatically; all AI and search crawlers are allowed by default |
| sitemap.xml | Generated automatically and lists all pages |
| Markdown | Served at the same URL for optimized pages |
Framer also names crawlers it says it tests with: Googlebot, GoogleOther, Google-Extended, Bingbot, GPTBot, PerplexityBot, YandexBot and AhrefsBot. The article does not say that list is exactly what your robots.txt covers, so read your own file instead of assuming.
Does Framer pre-render HTML for crawlers that skip JavaScript?
Framer sites are built with React, which usually raises the question of an empty page shell. Framer's help article says no: pages are pre-rendered, and crawlers that do not run JavaScript still receive the full page text, semantic HTML, the title and meta description, Open Graph and Twitter tags, canonical URLs and any JSON-LD.
That matters because several AI crawlers have been measured not running JavaScript. In December 2024 Vercel and MERJ reported that the major AI crawlers they observed did not execute it. For background, see do AI crawlers run JavaScript and raw vs rendered HTML.
Verify it rather than trust it. Fetch the raw HTML and search for a sentence you can see on the page:
curl -s https://example.com/ | grep -c "a sentence from your page"A count of 0 means that text is not in the server response. That can happen if a component loads its content after the page renders, so test the pages that matter, not only the home page.
What does the generated robots.txt contain, and can you change it?
Open https://yourdomain.com/robots.txt to see the live file. Framer's article on accessing your robots.txt file says Framer generates one for every published site. It does not document the exact default contents, and it does not say whether the default includes a Sitemap: line. Framer's own site, framer.com, serves a robots.txt with a sitemap line, but it is a large marketing site with its own rules, so that is not evidence of what your site gets.
On whether it can be edited, you will find conflicting answers. Some third-party guides say it cannot be. Framer's help center says it can, with an upload:
- Open Site Settings > Hosting > Files (an older help article describes the Files tab on the domain dashboard).
- Click Add and upload a file named
robots.txt. - Set the path to
/. Framer says to enter only the folder path, not the file name. - Publish the site.
Framer says a custom robots.txt replaces the generated one, and that static files, including this, are available on Pro and Enterprise plans. The help center does not describe another way to edit robots.txt on other plans.
Because the upload replaces the generated file, whatever you leave out is gone. If you upload one, include a Sitemap: line yourself.
To keep ChatGPT search access while opting out of OpenAI's training, the pattern is the same on any platform. OpenAI says OAI-SearchBot and GPTBot are independent, and that search changes can take about 24 hours after a robots.txt update:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlRemember that a crawler obeys the group that names it and ignores the * group, as RFC 9309 describes. Our guides to robots.txt for AI crawlers and GPTBot, OAI-SearchBot and ChatGPT-User cover the syntax and the bots.
Framer's AI article also says you can restrict specific agents "through your site settings". It does not say where that setting is, so treat a robots.txt upload as the documented route and confirm any other control by checking the live file afterwards.
What does page-level noindex do for AI search?
To hide one page, Framer's article on preventing pages from being indexed says to open Page Settings, turn off the Search engines option and republish. Framer then adds <meta name="robots" content="noindex"> to that page and leaves robots.txt alone.
Framer explains the reasoning: Google says robots.txt is not a mechanism for keeping a page out of Google, and recommends noindex instead. For the tag to work, the crawler has to fetch the page, so do not also disallow the same path in robots.txt.
Check that it landed, and that you did not leave it on by accident after a launch:
curl -s https://example.com/page | grep -io '<meta[^>]*robots[^>]*>'How AI search tools treat a noindex tag varies by operator; see noindex, nosnippet and X-Robots-Tag.
How do the sitemap and llms.txt work on Framer?
Sitemap. Framer's article on accessing your sitemap says a sitemap is created automatically for every published site, lists your pages with their last update, and stays current as you publish. Open /sitemap.xml on your domain to see it. The article does not cover custom sitemaps, CMS pages or noindex pages. After you set a page to noindex, open the sitemap and check whether the URL is still listed; a sitemap should list only the pages you want indexed. See XML sitemaps for AI discovery.
llms.txt. Framer supports llms.txt, which its article describes as a proposed standard. In Framer's well-known files article, you upload it through the same Files setting and give it the root path so it is served at /llms.txt. Before you spend time on it, note that OpenAI's crawler documentation does not say its crawlers read llms.txt. Fix access and rendering first. Our llms.txt guide covers who reads the file.
How do the markdown versions work?
Framer serves a markdown version of each optimized page on the same URL. Two ways to get it, both from the Framer help article:
- Add an
mdquery parameter with no value, such as?md. - Send a request header of
Accept: text/markdown.
Framer adds some limits. Markdown is generated only for optimized pages, and a page that is not optimized has no markdown version. It may also be unavailable if your site is being rate-limited. Each markdown page ends with a ## Navigation section of site-wide links. With multi-site rewrites, Framer converts when it sees the parameter or the header, unless your external host already serves markdown, in which case Framer serves that.
Framer says most AI agents request markdown with the header and names Claude, ChatGPT and Cursor as tools that can use it, but it does not name specific crawlers or show evidence. OpenAI's crawler documentation does not say its crawlers request markdown. Think of it as a clean, cheap-to-read copy for agents that ask for it, not a replacement for working HTML.
How can you test a Framer page with curl?
Run these against your own domain. The first shows what the page returns as markdown, the second tests the header, and the third checks the response type:
curl -s "https://example.com/about?md" | head -20
curl -s -H "Accept: text/markdown" https://example.com/about | head -20
curl -sI "https://example.com/about?md" | grep -i content-typeOn 2026-11-11 we ran the first and third against framer.com/pricing. Both the ?md request and the header request returned content-type: text/markdown; charset=utf-8, and the body began with front matter holding the title, description and publish date. That is one page on Framer's own site, so check your own pages: if you get HTML back, the page may not be optimized, or the site may be rate-limited.
Then check the other layers in order:
curl -s https://example.com/robots.txtshows the live rules.curl -sI https://example.com/robots.txt | head -1should return a 200.curl -s https://example.com/sitemap.xml | headshould list your pages.
Frequently asked questions
Can ChatGPT read a Framer site?
It can if nothing blocks it. Framer pre-renders pages to HTML and says its default robots.txt allows all AI and search crawlers. ChatGPT search access depends on OpenAI's OAI-SearchBot being allowed, so check that your robots.txt has no rule that blocks it.
Can I edit robots.txt on Framer?
Framer's help center says you can replace the generated file by uploading your own under Site Settings > Hosting > Files, on Pro and Enterprise plans. It does not document an editor for other plans.
Does Framer add llms.txt automatically?
Framer's article says it supports llms.txt and describes setup as an upload. It does not say that one is created for you, so check /llms.txt on your domain.
Will the markdown version improve my AI visibility?
Framer does not claim that. It describes markdown as a version of the page for agents that ask for it, and nothing we found shows that any particular AI search tool requests it or uses it to decide what to cite.
Check your own Framer page
The free check reads one page the way AI crawlers do, with no account needed: the robots.txt rules for each named AI crawler, the robots meta tag and whether the main text is in the raw HTML. If you manage a whole site, an audit runs the checks across up to 500 pages with evidence and prioritized fixes.
Filed under Platform guides. Topics: framer, ai-crawlers, robots-txt, sitemaps, structured-data