Skip to content

HTTP 431: a public page fails only for returning visitors

Trace 431 errors to oversized headers or cookies, compare fresh and returning sessions, and repair the source without relying only on a clean crawler test.

By Crawlito · 5 min read

In short

  • A 431 response can result from one large request header or the combined header size.
  • A clean checker request may miss a failure that affects returning browsers.
  • Repair cookie growth or the confirmed header source before changing server limits.
On this page

If a public page returns 431 only for returning visitors, compare their request headers with a fresh session. Oversized cookies are one possible cause, while a clean crawler check may still pass because it sends less state. Identify the header that exceeds the receiving component's limit, repair its growth or scope, and verify the normal returning-visitor journey.

Clearing cookies can help isolate the cause, but it is not a durable fix for a site that recreates the same oversized state.

Compare the failing and clean sessions

Record the URL, browser state, response status, and time of failure. Open the same URL in a fresh test profile and compare the result.

MDN's 431 reference explains that the server can reject one oversized header or the combined request headers. Its example includes a large Cookie header.

Do not assume cookies are responsible merely because a fresh session works. Authentication, experiment headers, and proxy-added fields can also change the request.

In developer tools, inspect header names and sizes without copying private values into shared notes. You usually need the field responsible and its approximate size, not the session token itself.

Save the failing state before clearing site data. Otherwise, the immediate symptom may disappear while the cause remains unknown.

Identify which layer rejects the request

Check edge, proxy, and application logs around the recorded time. A request rejected by an early layer may never appear in the application log.

Different components can have different limits. Increasing an application setting will not fix a CDN or reverse proxy that rejects the request first.

Use a diagnostic table:

LayerEvidence to inspect
BrowserOutgoing header names and sizes
Edge serviceRequest identifier and rejection reason
Reverse proxyHeader-size or parsing error
ApplicationWhether the request arrived
Cookie writerWhich feature added the growing value

Keep the ordinary public path intact during tests. Bypassing the proxy can help isolate a layer, but it does not establish that visitors can reach the page through production.

The HTTP status guide helps separate response meaning from application content.

If cookies are responsible, identify the feature that writes them. Look for values that accumulate search history, recently viewed items, experiment data, or serialized application state without a bound.

Check domain and path scope. A cookie intended for one app section may be sent to every public article when its scope is broad. Several subdomains may also contribute state under a shared domain.

Compare the cookie set before and after each step in a short reproduction sequence. For example, view a few items, change a preference, and revisit the article. The point is to identify the writer, not to generate a huge production request.

Do not put private user records into client cookies just because they are convenient to read. Review the data's appropriate storage and access design.

A repair may involve limiting the stored list, removing redundant values, narrowing a legitimate cookie's scope, or moving state to an appropriate server-side store. Choose the change that matches the actual feature.

Treat server limits as a capacity decision

A server limit can be too restrictive for a legitimate supported request, but raising it is not the first answer to uncontrolled growth.

Measure the normal request shape and the failing case. Review the receiving component's documented limits and the impact of any change. Keep the adjustment consistent across the actual delivery path.

Do not copy a large global header limit from an unrelated tutorial. A setting that avoids one rejection can consume resources or leave another layer as the new failure point.

If a legitimate authentication flow needs a larger bounded header, document that contract and test it. If a recently viewed list grows on every visit, repair the list.

Keep the incident scope clear: public page access should work for supported returning visitors. It should not depend on asking every visitor to reset their browser regularly.

Provide a controlled recovery path

For affected users, clearing the relevant site state may restore access temporarily. Explain any consequence, such as signing out or losing local preferences, before recommending that action.

Where appropriate, the application can expire an obsolete cookie using the correct domain and path. Test that the removal targets the intended cookie and does not silently erase unrelated state.

Be careful when the failure happens before the application can send a recovery response. The fix may need to be deployed at the layer that can still respond or through the existing supported session flow.

A hypothetical recovery record can distinguish immediate and durable outcomes:

Immediate recovery: the obsolete oversized cookie is removed.
Source repair: the feature no longer writes an unbounded value.
Verification: the same normal journey does not recreate the 431.

Keep these steps separate so a temporary recovery is not reported as the complete repair.

Verify both returning visitors and public checks

Repeat the exact journey that produced the error in a test environment or controlled account. Confirm that the header remains within the supported contract and the article loads.

Then test a clean public GET:

bash
curl -sS -L -D headers.txt -o page.html \
  'https://www.example.com/guides/setup'

The clean request is useful but covers a different state. Report both results rather than choosing whichever looks better.

Check the article's next-step form or app link as well. A page can load after a cookie change while the downstream session is broken.

The conversion measurement guide helps verify real outcomes after restoring access. A successful landing-page response is not the same as a completed user task.

Keep browser failures separate from crawler findings

A crawler without the affected cookies may receive the correct page throughout the incident. That does not make the returning-user failure unimportant.

Conversely, a proxy can add large headers to automated traffic too. Use actual request evidence instead of assuming all 431 errors belong to browsers.

For an audit, state the tested population: a clean public request passed, while a returning browser with the affected state failed. The repair addressed the state writer and was verified through the reproduction journey.

Use the raw and rendered HTML guide when a later rendering issue remains. Do not merge a header rejection and a missing-content problem into one vague score.

Add a bounded-state check when the responsible feature changes. A small realistic journey is more useful than a test that only opens the homepage in a fresh browser.

When documenting the cookie inventory, retain names, scopes, and approximate sizes while redacting values. The same cookie name can exist under different paths or domain scopes, so a list of names alone may miss the source of repeated data. Inspect the request's actual Cookie header alongside the browser's storage view.

For a returning-visitor regression check, use a controlled profile with representative old application state. A new private window is a useful control, but it cannot demonstrate that existing visitors have recovered. Record whether recovery required a scoped cleanup and whether the application starts growing the same data again after normal navigation.

Frequently asked questions

Does clearing cookies permanently fix a 431?

Only if the underlying cause is gone. A feature that recreates oversized state can bring the error back.

Why does a public checker pass while I cannot open the page?

Its request may not include your browser's cookies or other state. Compare the actual headers and session conditions.

Should I increase every header limit?

No. Identify the rejecting layer and the legitimate request contract, then fix the confirmed cause.

Use the free Crawlito check to establish the clean public response, and pair it with the returning-browser test.

Filed under Access and robots.txt. Topics: http-status, troubleshooting, cookies