Duda robots.txt and llms.txt: replacing the default files
How Duda robots.txt works: the default file and llms.txt every published site gets, how to upload a custom robots.txt for AI crawlers, and how to revert it.
By Crawlito · 8 min read
In short
- Every published Duda site gets a default sitemap.xml, robots.txt and llms.txt; the defaults cannot be edited, only replaced by a file you upload.
- You replace the default robots.txt in Settings > URL Redirect by adding a File redirect with the Source URL robots.txt and the type 2xx OK, then republishing.
- Duda does not validate custom files, so a broken robots.txt is served as is; deleting the redirect and republishing restores the default.
- Duda's default robots.txt includes a Content-Signal line with ai-train=yes, which disappears when you upload your own file unless you copy it in.
- Duda's security systems check requests before robots.txt is read, so a custom robots.txt cannot unblock a crawler that gets a 403.
On this page
Every published Duda site serves a default robots.txt, sitemap.xml and llms.txt that Duda generates for you. You cannot edit the default robots.txt, but you can replace it: upload your own file as a File redirect in Settings > URL Redirect, with the Source URL robots.txt, then republish. Deleting that redirect and republishing brings the default back. Duda does not check the file you upload, so the safest workflow is to write it carefully, publish, then fetch it from the live domain.
This post follows Duda's Help Center at the time of writing (November 2026). It covers what the default files contain, how to swap in a robots.txt with AI crawler rules, what the generated llms.txt lists, and how an agency can check a batch of client sites. For the general format of llms.txt, see the llms.txt guide.
What files Duda generates for a published site
Duda's Site Configuration Files article says all published Duda sites include a default sitemap.xml, robots.txt and llms.txt. A few details matter for AI crawlers:
| File | Generated by Duda | Editable | Replaceable |
|---|---|---|---|
/robots.txt | Yes, on publish | No | Yes, by uploading a file |
/sitemap.xml | Yes, on publish | No | Yes, by uploading a file |
/llms.txt | Yes, updated on publish | No | Not documented |
- Only published sites have these files. An unpublished draft has no robots.txt to test.
- The sitemap follows your noindex settings. Pages set to not index in their SEO settings are left out, and if the whole site is set to no index, the sitemap is empty.
- The default robots.txt carries a Content-Signal line. Duda notes that Google Search Console may flag this part of the file:
Content-Signal: search=yes, ai-input=yes, ai-train=yesDuda says the line causes no issues and is only flagged because it is a newer addition. The values say yes to search, to use as AI input and to AI training.
Duda does not print the full default file, but you can see it on any Duda-hosted site that has not replaced it. When we fetched https://blog.duda.co/robots.txt in November 2026, it returned:
# Sitemap is also available on /sitemap.xml
Sitemap: https://blog.duda.co/sitemap.xml
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yesThere are no Disallow lines and no named crawler groups, so every crawler that obeys robots.txt is allowed everywhere. The Content-Signal line sits in the wildcard group. What that line means, and which crawlers act on it, is covered in Content-Signal in robots.txt. The point for Duda users is practical: if you replace the default file, this line is gone unless you copy it into your own file, with the values you actually want.
Look at the live file before you change anything:
curl -s https://www.example.com/robots.txtHow to replace the default robots.txt on Duda
Custom files are available in the Classic editor and Editor 2.0. Duda says they are not available for Duda Vibe projects yet. The upload lives in URL redirects, which is not where most people look for it. Duda's steps, at the time of writing:
- In the side panel, click More, then Settings.
- Click URL Redirect, then Add New Redirect.
- In the Source URL field, enter
robots.txt. Duda says it has to match the file name exactly to override the default. Any other value becomes the file's own path on the live site, so the default robots.txt stays in place. - Select the File option and click + Upload file to choose your robots.txt.
- Set the Redirect type to 2xx OK (Redirect) and click Add.
- Publish or republish the site.
Duda's Site SEO/AEO Settings article also lists a Custom sitemap, robots.txt & other files option under Optimization Center > Additional Settings, and points to the same Site Configuration Files feature.
Supported file types are .xml, .txt and .json. Duda also states two limits up front: helping with the content of custom files is outside its support scope, and the platform does not validate custom files, so a corrupt file is still served.
How to revert to Duda's generated robots.txt
Go back to More > Settings > URL Redirect, find the robots.txt entry, click Delete, then publish or republish. After the publish, fetch /robots.txt again and check that the default content, including the Content-Signal line, is back.
A robots.txt for AI crawlers on Duda
Duda's How to Block AI Crawlers article gives this example, which blocks GPTBot and Google-Extended:
# Sitemap is also available on /sitemap.xml
Sitemap: http://www.example.com/sitemap.xml
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *Two things to adjust before you use it. First, write the Sitemap line with your real domain and https, so it points at the version of the site you actually serve. Second, the example only covers two training tokens. It does not mention OpenAI's search crawler, Anthropic, Perplexity or Common Crawl at all, so those crawlers fall through to the empty User-agent: * group, which allows everything.
Duda's article also says crawlers read the file from top to bottom, so it advises against putting the wildcard group first. Under RFC 9309, a crawler picks the group whose user-agent line matches its own token and only falls back to * when nothing matches, so group order does not change the result. Keeping Duda's layout does no harm; just know that the order is not what protects you. The details are in robots.txt groups explained.
A file that keeps AI search and user-triggered fetches open while opting out of training looks like this:
Sitemap: https://www.example.com/sitemap.xml
# Training crawlers and opt-out tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# AI search and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /The reasoning behind each group is in allow AI search, block AI training. The Content-Signal line here sits in the wildcard group, as in Duda's default. A crawler that has its own named group follows that group instead, so if you want a signal to apply to it, repeat the line in its group. If you would rather keep Duda's default signals, copy its Content-Signal line unchanged. Either way, it is your file now, and Duda no longer adds anything to it.
What robots.txt cannot do on Duda
Duda's article on site traffic limits says its security systems protect published sites from unusual automated traffic, which can include third-party crawlers and SEO tools, and that a blocked tool usually reports a 403. Major search engines such as Google are not blocked. Changing robots.txt does not unblock a tool, because the security check happens before robots.txt is read, and there is no setting to allow an IP address. To ask for a tool to be reviewed, contact Duda support with the tool's name, its IP addresses and user agent, and the URL and time of a blocked request. Duda says not every tool can be allowed.
Duda does not publish which AI crawlers its security layer allows, so treat a 403 to an AI crawler as a support ticket, not a robots.txt problem.
What Duda's llms.txt lists
Duda announced in June 2025 that it was rolling out llms.txt on all published sites. The Help Center now says Duda creates and updates the file for every published site, in Markdown, at /llms.txt. It contains a list of the site's pages with descriptions and URLs, covering:
- Regular pages and dynamic pages
- Blog posts
- Translated pages
- Native store category pages and product pages
The announcement adds that each entry uses the page's meta description, or the meta title when there is no description, and that drafts and noindex pages are left out. The practical consequence is that the quality of your llms.txt is the quality of your meta descriptions. A site where every page shares the same description gets an llms.txt full of identical lines.
One detail Duda does not document: when we fetched /llms.txt from two Duda-hosted sites (blog.duda.co and academy.duda.co) in November 2026, both responses carried an X-Robots-Tag: noindex header. That keeps the file itself out of search results; it does not stop anyone fetching it. Check your own with curl -sI https://www.example.com/llms.txt.
Duda's documentation covers replacing the default sitemap.xml and robots.txt, and says any .txt file can be uploaded. It does not say whether a file uploaded as llms.txt overrides the generated one. If you try it, fetch /llms.txt after publishing to see which version is served, and expect to maintain it by hand from then on.
Keep expectations modest. As the llms.txt guide explains, no major AI search crawler documents reading llms.txt, so the generated file is harmless but is not what decides whether ChatGPT or Perplexity can read your pages. robots.txt, the security layer and the HTML itself matter more.
How to check many Duda client sites at once
Because Duda serves a custom file without validating it, a bad upload fails silently. Things to watch for: a file saved in a rich-text format instead of plain text, an HTML page uploaded by mistake, a Source URL typed as something other than robots.txt so the default is still served, or a new file that dropped the Sitemap line. A short loop over your client domains catches most of these:
#!/usr/bin/env bash
# domains.txt: one hostname per line, e.g. www.client-one.com
tmp=$(mktemp)
while read -r host; do
: > "$tmp"
robots=$(curl -sL -o "$tmp" -w '%{http_code} %{content_type}' "https://$host/robots.txt")
llms=$(curl -sL -o /dev/null -w '%{http_code}' "https://$host/llms.txt")
html=$(grep -ci '<html' "$tmp")
sitemap=$(grep -ci '^sitemap:' "$tmp")
signal=$(grep -i '^content-signal:' "$tmp" | head -1)
gptbot=$(grep -ci '^user-agent: *gptbot' "$tmp")
echo "$host | robots $robots | llms $llms | html:$html sitemap:$sitemap gptbot:$gptbot | $signal"
done < domains.txt
rm -f "$tmp"Read the output row by row:
| Result | What it usually means |
|---|---|
| robots status not 200 | Site not published, or the domain is not connected |
html: above 0 | An HTML page was uploaded, or served, instead of a text file |
sitemap:0 | A custom file without the Sitemap line |
Content-Signal with ai-train=yes and gptbot:0 | Still on Duda's default file |
| llms status not 200 | Site unpublished, or something on the domain is in front of Duda |
Then test the rules themselves, not just the file. The methods in how to test robots.txt for AI crawlers show whether a given token is allowed for a given path, and common robots.txt mistakes lists the errors that look fine at a glance.
For evidence that crawlers are actually arriving, Duda's Traffic page under Performance shows an LLMs visits chart split into search crawlers and AI bots, with data from February 2026 onward. Duda does not document which user agents it counts in each bucket.
Frequently asked questions
Can I edit the robots.txt on Duda?
Not directly. Duda's default robots.txt cannot be edited, but you can replace it by uploading your own file as a File redirect with the Source URL robots.txt and republishing.
Does Duda block AI crawlers by default?
The default robots.txt documented by Duda contains a Content-Signal line that says yes to search, AI input and AI training, and Duda gives no default rule blocking AI crawlers. Its security systems can still return 403 to automated traffic they consider unusual, and robots.txt cannot change that.
How do I go back to Duda's default robots.txt?
In More > Settings > URL Redirect, delete the robots.txt entry and republish. The generated file, including its Content-Signal line, is served again.
Can I turn off or edit Duda's llms.txt?
Duda's documentation does not describe a setting to turn it off or edit it. Duda says the file is generated each time you publish, from your live pages and their meta descriptions, so editing those descriptions and republishing changes what it says.
Check a Duda site after you publish
A custom robots.txt is one upload away from blocking a crawler you wanted, or allowing one you did not. After each change, run the free check on a client's key page to see whether the main AI crawlers can fetch it. If you manage a portfolio of Duda sites and want robots.txt, the security layer and page content reviewed with evidence, an audit covers them together.
Filed under Platform guides. Topics: duda, robots-txt, llms-txt, ai-crawlers, agencies