Skip to content

Drupal Composer overwrites robots.txt after a release

Keep custom crawler rules through Drupal Composer updates by tracing the scaffold mapping, merging a maintained rules file, and checking the deployed URL.

By Crawlito · 5 min read

In short

  • Drupal Composer Scaffold can regenerate the public robots.txt from project configuration and package assets.
  • A file mapping can append a maintained project fragment to the upstream scaffold.
  • Verify the generated artifact and the live root response after a clean build.
On this page

If Drupal restores its default robots.txt after a Composer update, find which scaffold mapping generates the public file. Store your custom rules in project-owned configuration or an asset and let the normal build reproduce them. Editing the deployed copy by hand can work until the next installation generates it again. Drupal’s Composer Scaffold guide

Identify the document root that serves robots.txt

Start with a request for /robots.txt on the exact public hostname. Save the status, response type, and body. Then identify the artifact or container directory from which that host serves static files. Common Drupal projects use web/ or docroot/ as the web root. A file named robots.txt next to the top-level Composer configuration might have no relationship to the URL a crawler requests.

Compare three copies: the project-owned source, the generated file in the release artifact, and the live response. If your edit is present in the repository but absent in the artifact, inspect the Composer configuration and deployment build steps. If the artifact is right but the live response differs, investigate the serving root, static route, environment override, or edge cache. Repeating a Composer update in response to an edge-generated robots.txt will leave the real cause untouched.

Use an active crawler directive and a distinctive harmless comment to make comparisons clear. A timestamp alone can change with each build even when all meaningful directives are identical. Include the hostname in your notes; robots.txt applies at the origin where it is fetched. A production host and preview host can intentionally publish opposite policies. The guide to staging index exclusions describes how to check that boundary without copying a test environment's restrictions into production.

Locate the scaffold mapping that owns the file

Inspect the project's root composer.json for extra.drupal-scaffold, its locations, and its file mapping. Follow the referenced asset and check whether another allowed package contributes scaffold configuration. The upstream scaffolding plugin distributes common public files alongside Drupal core, and the project can specify how its own files should be applied. Read the effective build configuration, especially if different hosting environments build from different Composer roots.

Tools that show a local working-tree diff are handy here. In a disposable checkout of the same release, run the project's normal Composer installation and see whether the generated file changes. Keep the lockfile and the same installation flags used by your deployment. Running a broad dependency update as a diagnostic introduces other changes that can make the original failure difficult to reproduce.

A managed hosting platform or CDN can also supply a robots.txt response outside Drupal. If changing the generated artifact has no effect on a targeted preview deployment, inspect the routing behavior before editing the underlying content again. An asset with the right rules in a release bundle provides a clue about the next layer to inspect; it does not prove what is served publicly.

Append a maintained project fragment

Drupal documents an append file mapping for adding rules to the upstream robots.txt. The example below is a fragment to merge with the existing Composer configuration. Keep the project's already configured document root and other mappings intact. Place the referenced rules file under project version control so a clean build has all its inputs.

json
{
  "extra": {
    "drupal-scaffold": {
      "locations": { "web-root": "web/" },
      "file-mapping": {
        "[web-root]/robots.txt": {
          "append": "assets/robots-additions.txt"
        }
      }
    }
  }
}

Choose the rules themselves according to your intended crawler policy. Adding a new user-agent group can change which group a matching crawler follows. For example, a specific group for a named bot can take precedence over general rules, so preserve relevant path restrictions for that bot when your intended policy requires them. Adding a Disallow line at the end of a file without reviewing the preceding group can assign it to the wrong crawler. The robots.txt groups guide explains the matching behavior.

If the organization needs to control the entire file, Drupal also documents mapping a complete project-owned asset. That option puts future review of upstream defaults on the project. Compare core's scaffold changes during upgrades so you can make an informed decision about new paths. A small appended fragment is easier to maintain when the general upstream defaults already serve the site.

Prove a clean build and a real deployment agree

Build from a clean checkout or the same release container used in deployment. Inspect the resulting robots.txt in the configured web root and check that the relevant user-agent group contains the expected paths. Read several lines before and after the added fragment. A missing newline at the join can alter how a directive is interpreted even though a text search finds the expected agent name.

bash
curl -sS -D robots-headers.txt -o robots-live.txt https://example.com/robots.txt

After deployment, compare the public response with the artifact. Account for intended edge-managed additions if your platform uses them. Test at least one public path that should be allowed and one pattern intentionally excluded for each group you changed. The robots.txt testing guide describes how to evaluate matching rules. Directly fetching the protected page with curl is a separate test; a server can return a page even when a cooperative robot is asked to leave it alone.

Finally, repeat the build with no local edits. The public policy should be reproducible on a fresh machine with only the checked-in inputs. If someone has to manually restore the file after each deployment, the actual source of truth is still outside the release process. Keep a short record of the mapping and its intentional exceptions with the code that owns them so the next Drupal update can be reviewed accurately.

Review the outgoing rules as a change in crawler policy

A default scaffold update can introduce new administrative paths while your own fragment defines a narrower user-agent group. Compare the fully generated old and new files with the policy owner during an upgrade. For each changed group, choose a publicly accessible content route, a non-content endpoint covered by the default policy, and any deliberately excluded path. Interpret each path against the applicable group and keep a brief note of the intended result.

For example, an editor might ask to make product manuals available to a search crawler while continuing to discourage that crawler from fetching a repeating internal listing. If the custom group repeats only the manuals permission and omits the exclusion, a superficially successful change can enlarge the permitted crawl surface. Test the listing as well as the manual. Public access control remains the application's responsibility regardless of robots rules.

When the configuration is reviewed, compare the generated artifact across two clean builds using the same dependency lockfile. A reproducible file gives the team a meaningful baseline for the next core upgrade. If the files differ, examine environment-specific scaffold steps before attributing the difference to a crawler ignoring your rules.

Frequently asked questions

Do I edit Drupal core's copy of robots.txt?

Use the root project's scaffold configuration for customizations. An edit inside installed core assets is vulnerable to dependency installation or upgrade.

Why does staging show the right rules but production shows defaults?

Compare their effective web roots, scaffold inputs, artifacts, and edge responses. The two environments may get their public robots.txt from different places.

Does fixing the file instantly change crawler behavior?

You can verify the new public response immediately. Individual crawlers may use a previously fetched policy until they refresh it.

Check a public Drupal route

Use the free page check on an intended public page after the release. Pair its access findings with the exact robots.txt response from the same host.

Filed under Platform guides. Topics: drupal, robots-txt, deployment, ai-crawlers