Skip to content

Hugo, Jekyll and Eleventy: robots.txt, sitemap and baseURL

Static site generators make AI crawler access easy. How to set robots.txt, sitemaps, canonicals and baseURL in Hugo, Jekyll and Eleventy, and what still breaks.

By Crawlito · 6 min read

In short

  • Static site generators write every page as an HTML file, so crawlers that do not run JavaScript can read the content without any extra work.
  • Hugo only generates robots.txt when enableRobotsTXT is true, and its default file contains just User-agent: * with no Sitemap line.
  • The jekyll-sitemap plugin builds sitemap.xml and, if your site has no robots.txt, adds one containing only a Sitemap line.
  • Eleventy generates neither a sitemap nor a robots.txt by default, so both have to be added as templates or copied files.
  • Most static-site problems come from the site URL setting: a wrong baseURL or url puts the wrong host into canonicals, sitemaps and the robots.txt Sitemap line.
On this page

Static site generators such as Hugo, Jekyll and Eleventy are the easy case for AI crawlers. They build every page into an HTML file before deployment, so the content is in the response that crawlers download, whether or not they run JavaScript. What still goes wrong is configuration: a missing or default robots.txt, no sitemap, and a site URL setting (baseURL in Hugo, url and baseurl in Jekyll, your own data file in Eleventy) that puts the wrong host into canonical tags and sitemaps.

Each generator handles these differently, and their defaults are not the same. Here is what each one does out of the box, how to fix the gaps, and how to check the result on the live site.

Why static sites are the easy case

Many AI crawlers read the HTML your server returns and do not render JavaScript, as explained in do AI crawlers run JavaScript. A static generator produces exactly that HTML. There is no server-side code that can time out, no client-side router, and no empty shell waiting for a script.

That leaves a short list of things to get right:

  1. robots.txt exists, is served from the root of the host and says what you mean.
  2. A sitemap lists your real, final URLs.
  3. Canonical tags and the sitemap use the production host and protocol.
  4. Drafts and test pages are not published.
  5. The host returns real 404 status codes for missing pages.

Hugo: robots.txt, sitemap and baseURL

robots.txt

Hugo does not write a robots.txt unless you ask. Hugo's robots.txt documentation says to set enableRobotsTXT to true in your configuration. Without a custom template, the generated file contains only User-agent: *, which allows everything but does not mention your sitemap.

Override it with a template at layouts/robots.txt:

User-agent: *
Allow: /

Sitemap: {{ "sitemap.xml" | absURL }}

absURL builds the address from your baseURL, so the Sitemap line follows the configuration. You can also put a plain robots.txt in the static/ folder, which Hugo copies as is, but then the sitemap URL is hard-coded.

Sitemap

Hugo builds a sitemap without any setup. Its sitemap documentation says a single-language site gets sitemap.xml at the root of public, and a multilingual site gets one per language plus a sitemap index at the root. To leave a page out, add this to its front matter:

yaml
sitemap:
  disable: true

baseURL and canonicals

baseURL is the absolute address of the published site, including protocol, host and trailing slash. Hugo uses it to build .Permalink, which is what themes normally put in the canonical tag:

html
<link rel="canonical" href="{{ .Permalink }}">

If a theme does not output a canonical tag, add that line to your base template. Check baseURL in every environment you build for. A production build made with a staging or preview baseURL publishes canonicals and sitemap entries that point at the wrong host.

Jekyll: url, baseurl and the standard plugins

url and baseurl

Jekyll uses two settings in _config.yml: url is the scheme and host, and baseurl is the subpath, which is empty for a site at the root of its domain. The absolute_url filter combines them. On GitHub Pages project sites, where the site lives at /repository-name/, a wrong baseurl is the classic source of broken links, as described in GitHub Pages Jekyll baseurl broken links.

yaml
url: "https://www.example.com"
baseurl: ""
plugins:
  - jekyll-sitemap
  - jekyll-seo-tag

Sitemap and robots.txt

The jekyll-sitemap plugin generates sitemap.xml from url, and excludes any page with sitemap: false in its front matter. Its source code also adds a robots.txt when your site does not have one, and that file contains a single line:

Sitemap: https://www.example.com/sitemap.xml

A robots.txt with no rules allows every crawler, so this is fine if that is what you want. If you need groups for AI crawlers, create your own robots.txt in the site root. The plugin then leaves it alone, so add the Sitemap line yourself.

On GitHub Pages, list plugins under plugins in _config.yml. The plugin's README says that listing it only in the Gemfile does not work there.

Canonical tags

The jekyll-seo-tag plugin's template outputs a canonical link built with absolute_url from the page URL, and lets a page override it with canonical_url in front matter. So the canonical is only as correct as your url and baseurl.

Eleventy: you add both files yourself

Eleventy generates neither a robots.txt nor a sitemap by default. Both are easy to add as templates.

Keep the site address in a global data file, for example _data/site.json:

json
{ "url": "https://www.example.com" }

Then a robots.txt template, saved as robots.njk:

---
permalink: /robots.txt
eleventyExcludeFromCollections: true
---
User-agent: *
Allow: /

Sitemap: {{ site.url }}/sitemap.xml

And a sitemap template, saved as sitemap.njk. Eleventy puts every template in collections.all, and the collections documentation names sitemaps as a typical use of eleventyExcludeFromCollections, so the sitemap does not list itself:

xml
---
permalink: /sitemap.xml
eleventyExcludeFromCollections: true
---
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
{%- for item in collections.all %}
  <url><loc>{{ site.url }}{{ item.url }}</loc></url>
{%- endfor %}
</urlset>

Use a loop variable other than page, because Eleventy already uses page for the current template's data. If you prefer a fixed robots.txt, copy it unchanged with eleventyConfig.addPassthroughCopy("robots.txt") in your config. Add eleventyExcludeFromCollections: true to any page you do not want listed, such as a thank-you page.

Adding AI crawler rules to a generated robots.txt

Whichever generator you use, AI crawler rules are ordinary robots.txt groups in the template or file above. A common policy is to allow search crawlers and opt out of training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot are not named, so they use the * group and stay allowed. Keep the rules in the template, not in a host dashboard, so they are versioned with the rest of the site and survive a change of host.

What each generator does by default

HugoJekyllEleventy
robots.txtOnly with enableRobotsTXT; default is User-agent: *With jekyll-sitemap, a file with only a Sitemap line if none existsNone
SitemapBuilt injekyll-sitemap pluginNone, add a template
Site URL settingbaseURLurl and baseurlYour own data file
Canonical tagFrom the theme, usually .Permalinkjekyll-seo-tagYour own template
Exclude a page from the sitemapsitemap: disable: truesitemap: falseeleventyExcludeFromCollections: true

What still goes wrong on static sites

  • The host overrides your files. Some hosts and CDNs serve their own robots.txt, add bot protection or rewrite headers. The file in your repository is not necessarily the one crawlers get.
  • Drafts and test pages are published. A build without the production flag, or a stray test page, ends up in the sitemap.
  • No real 404. A missing page that returns the home page with 200 looks like duplicate content to search engines. Check that your host serves 404.html with a 404 status.
  • Trailing slashes and index.html. The same page can answer at /about, /about/ and /about/index.html. Pick one form, redirect the others and use it in canonicals and the sitemap. Canonical tags and duplicate URLs covers the details.
  • A Disallow left from a staging build. A robots.txt that blocks everything on a preview domain sometimes ships to production. Staging noindex leaks describes how that happens.

How to check the live site

Test the deployed site, not the build folder:

bash
curl -s https://www.example.com/robots.txt
curl -s https://www.example.com/sitemap.xml | head -20
curl -s https://www.example.com/about/ | grep -i 'rel="canonical"'
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/does-not-exist

Look for localhost, a preview domain or http:// where you expect https://, and for a 200 on the last line. Then check that the robots.txt rules match your policy for AI crawlers, using robots.txt for AI crawlers as the reference.

Frequently asked questions

Does Hugo create a robots.txt automatically?

Only if enableRobotsTXT is set to true. The default file contains just User-agent: *. Add a layouts/robots.txt template for anything more, including a Sitemap line.

Why does my Jekyll robots.txt only contain a Sitemap line?

That file comes from the jekyll-sitemap plugin, which adds it when your site has no robots.txt of its own. Create your own robots.txt in the site root to replace it.

Do I need a sitemap plugin for Eleventy?

No plugin is required. A Nunjucks or Liquid template that loops over collections.all and writes to /sitemap.xml is enough.

Do AI crawlers need anything special from a static site?

No special file is required. They need to be allowed in robots.txt and at your host or CDN, and to find your URLs through links or a sitemap. The HTML a static generator produces is already readable.

Next step

Run the four curl commands above after your next deploy. For a quick outside view of one page, the free check tests crawler access with no account.

Filed under Platform guides. Topics: robots-txt, sitemaps, canonical, ai-crawlers, html