Language redirects send crawlers to the wrong page
Reproduce unwanted language redirects with fresh requests and explicit Accept-Language headers. Check stable locale URLs, cookies, and the final article.
By Crawlito · 5 min read
In short
- Google says its crawler normally sends requests without an Accept-Language header.
- A stable locale URL should have a documented behavior for a fresh visitor.
- A language selector needs to preserve the relevant page when an equivalent translation exists.
On this page
- Write down the intended behavior of a direct locale URL
- Compare clean requests with explicit preferences
- Follow the rule that overrides the requested language
- Keep alternate-language and canonical signals coherent
- Test the full route after deployment
- Test a cache shared across language preferences
- Frequently asked questions
- Check the exact locale URL
If a crawler keeps reaching the default language, test the affected locale URL without cookies or an Accept-Language header. Compare that response with explicit language preferences and inspect every redirect. Google warns that locale-adaptive responses can leave some language versions undiscovered, and says its crawler normally does not send Accept-Language. Google’s locale-adaptive page guidance
Write down the intended behavior of a direct locale URL
Choose a specific article that exists in at least two languages. Record the public URL for each and identify a distinctive sentence from each translation. Ask what should happen when someone opens the French URL while using an English browser. The direct link might be intended to preserve the requested language, with an optional suggestion to switch.
Distinguish that article link from the root homepage. A homepage may offer a language chooser or a documented default while language-specific paths preserve their content. Applying the same automatic redirect to every request can make a translated article impossible to share reliably. A customer opening a support link needs the instructions associated with that link.
Record which layer chooses the language. The CDN, reverse proxy, application middleware, and CMS can all make independent decisions. A remembered cookie might override the URL in one component while another chooses from an IP-derived region. Without an agreed priority, the request can bounce between contradictory rules.
Compare clean requests with explicit preferences
Fetch the same locale address without cookies, then with two different language headers. Save the initial response as well as the followed chain. These are controlled diagnostics of your own site, not evidence of every crawler's behavior.
curl -sS -D no-language-headers.txt -o no-language.html https://example.com/fr/guides/install
curl -sS -H 'Accept-Language: en-US,en;q=0.8' -D english-preference.txt -o english-preference.html https://example.com/fr/guides/install
curl -sS -H 'Accept-Language: fr-FR,fr;q=0.8' -D french-preference.txt -o french-preference.html https://example.com/fr/guides/installRead the status, any Location header, and the body markers. A French URL returning English text with a success status is different from a redirect to the English article. Both can contradict the intended behavior, but they have different causes and consequences for canonical and alternate-language signals.
Repeat in a fresh browser and inspect the language selector. Check whether choosing French creates a cookie, changes the URL, or only swaps text after JavaScript loads. The multilingual site guide provides a broader comparison of visible content and direct visits across locales.
Follow the rule that overrides the requested language
If the redirect changes with the request header, inspect the language middleware and its priority order. If it changes only from another network region, examine the edge or location rule. If it appears only after the first visit, inspect the preference cookie and its scope. Keep these variables separate during reproduction.
| Observation | Next place to inspect |
|---|---|
| Locale path always redirects to the default home | Broad route rule or missing translation mapping |
| Language changes after a cookie is set | Saved preference overriding the explicit path |
| Request loops between two locale URLs | Conflicting edge and application priorities |
| Correct HTML becomes another language after loading | Client-side locale initialization |
| Only image or API requests redirect | Language rule applied beyond page navigation |
A rule may be correct for the site root but too broad for article paths or machine-readable endpoints. Test robots.txt, the sitemap, and a static asset as controls. A language redirect that sends a sitemap request to a translated homepage creates an additional discovery failure unrelated to the article content.
Repair the owning rule using the site's intended priority. A common design lets an explicit locale URL establish the requested language and uses preferences to suggest a switch. The appropriate design depends on the publication, but its behavior should be predictable and testable.
Keep alternate-language and canonical signals coherent
Inspect the canonical and hreflang entries on each translation after the redirect repair. They should describe the pages that actually serve the corresponding content. A French alternate pointing to a URL that redirects to an English homepage does not provide a usable French counterpart.
Check reciprocal references among true equivalents and the site's fallback behavior for untranslated pages. A language switch for an article with no equivalent might take the visitor to a section index, but the interface should explain that change. Avoid claiming that an unrelated page is an alternate translation just to fill a missing mapping. The hreflang audit guide explains how to build these page groups.
Keep regional commerce behavior distinct from translated editorial content. A store may need country-specific availability or currency, while a technical guide may be the same language across several countries. A single region cookie should not silently change a document into another language unless that is the site's explicit, communicated design.
Test the full route after deployment
Use a small matrix covering direct locale URLs, the root chooser, a saved preference, and a missing translation. For each, record the final URL, actual language, canonical, and selector destination. Test both the first request and a second visit after choosing a language. This catches a fix that works only before the preference cookie exists.
Include a real shared support link in the test set. Open it from outside the site in a new session, read the expected paragraph, and follow the related next step. A successful homepage language switch alone cannot establish that deep links work. The redirect debugging guide helps preserve the exact navigation chain.
Compare the public response with the deployed configuration and allow for the platform's normal cache propagation. A cached redirect can survive an application fix, especially when it was given a long lifetime. Inspect the response you receive before attributing the continued behavior to the browser or crawler.
Save the matrix with the site's release checks. When another language is added, extend the expected mappings and test a genuinely missing equivalent. The result is a reproducible locale policy that helps both visitors and clients consuming public documents.
Test a cache shared across language preferences
Suppose the origin serves different languages at one URL according to a request header, while a shared cache stores only one representation for that path. The first request can populate the cache with a language that later visitors did not request. Compare a cache miss and subsequent requests with different language preferences, using the platform's approved diagnostic method.
Record the selected representation, response headers, and whether the request reached the origin. Have the hosting owner check the cache key and the site's intended variation behavior. A stable locale path can make the desired representation explicit, but its serving rules still need to preserve that choice.
Repeat the test after a language selection cookie is set and after it is cleared. A configuration that varies by header but ignores the cookie used by the application can still mix representations. The acceptance test is that each supported direct locale request receives the expected article consistently under the documented policy, including after another visitor has warmed the cache.
Frequently asked questions
Can I rely on the browser language to expose every translation?
Google documents limitations of locale-adaptive pages. Stable, linked locale URLs give you explicit destinations to publish and test.
Should I remove the language selector?
Keep a usable selector. Check that it leads to the equivalent page where available and makes any fallback understandable.
Why is my own browser always correct?
A saved preference or previous navigation can hide the first-visit behavior. Repeat the direct request without that state.
Check the exact locale URL
Run the free page check on the language-specific article. Compare its final destination and readable text with the clean-request matrix you recorded.
Filed under Access and robots.txt. Topics: international-seo, hreflang, redirects