HTTP 406 for a crawler: inspect Accept headers and rules
Diagnose a 406 response by comparing content negotiation and security-rule evidence, then verify the intended public representation with a controlled request.
By Crawlito · 5 min read
In short
- A 406 response concerns the server's ability to supply an acceptable representation.
- The status alone does not identify whether an application or security component generated it.
- Change one request header at a time and preserve the working and failing responses.
On this page
If a public page returns 406 to a checker or crawler, compare its request headers with a working request and identify which component emitted the response. The standard meaning concerns an unacceptable representation, but the status alone does not establish your site's implementation. Repair the confirmed negotiation or rule mismatch rather than adding a blanket crawler exception.
A browser can send a different Accept header from a command-line client. That difference is a useful lead, not proof that either client is behaving incorrectly.
Save a plain request and a browser comparison
Fetch the failing URL and retain the response body:
curl -sS -D plain-headers.txt -o plain-body.txt \
'https://www.example.com/guides/setup'
curl -sS -D html-headers.txt -o html-body.txt \
-H 'Accept: text/html' \
'https://www.example.com/guides/setup'These requests differ only in the explicit Accept value. Compare their status, content type, final route, and body. If both fail, that particular header change has not isolated the cause.
MDN's 406 reference describes content negotiation when the server cannot provide a representation acceptable under the request's negotiation headers.
Do not copy a full authenticated browser request into a public ticket. Cookies and authorization headers may contain sensitive state. Reproduce the public failure with the smallest relevant request and keep any private diagnostic material in its proper environment.
Identify the response owner
Inspect response headers, body branding, request IDs, and the corresponding logs. Determine whether the application, web server, proxy, or security product generated the 406.
A custom rule can choose a status code that does not fully describe its reason. You need the matching rule or application decision to know what to change.
Compare the recorded request time with origin logs. If the request never reached the application, a template or controller change is unlikely to fix it.
The WAF response identification guide helps locate the owner, but do not infer a product solely from a generic error page.
Write a specific finding such as “the HTML route rejects a broad Accept value” or “the request matches this managed rule.” Those findings support different repairs. A report that only says “406 means bot blocked” does not.
Test negotiation inputs one at a time
If the application owns the response, inspect its supported media types and negotiation logic. A route intended for HTML should have a deliberate behavior for common public requests.
Check whether the same URL also supports JSON or Markdown. If it does, define the supported representations and how the server chooses among them. Do not return a success status with an unrelated body just to make the checker pass.
Repeat a controlled comparison with the exact header associated with the failure. Keep language and encoding tests separate so you can identify which input changes the result.
The Markdown negotiation guide explains an existing multi-representation use case. A repair should preserve the intended HTML and Markdown paths rather than forcing every client to receive one format.
Record the expected output for each supported request. A small table of media type, status, and body marker is easier to review than a long list of trial-and-error commands.
Repair a confirmed security-rule mismatch narrowly
If a security component emitted the response, use its rule ID and matching evidence. Review why the public request matched before creating an exception.
Scope any adjustment to the intended route and request characteristics supported by the product. Do not allow every request carrying a familiar crawler user agent. User-agent text alone is not identity verification.
Keep legitimate protections for private or state-changing endpoints. A public article GET should not require removing unrelated application controls.
If a request body or query value triggered the rule, confirm whether that input belongs in the public page request at all. Sometimes the correct fix is a malformed generated link rather than a firewall exception.
Use the verified crawler guide when the decision depends on a real operator's identity. A local curl test can reveal response differences but does not originate from the operator's network.
Verify the actual representation
After the change, repeat the original failing request and the normal browser request. Confirm the intended page title and body in both responses.
Check content type and encoding as well. A 200 response containing a JSON error or login page is not a repaired article.
Include a supported alternative representation if the route intentionally negotiates content. For example, HTML should remain HTML and a documented Markdown request should receive the intended Markdown response.
A useful acceptance record is:
| Request | Expected evidence |
|---|---|
| Original public request | Correct article, no unexplained 406 |
| Browser-like HTML request | Same intended public content |
| Supported alternate format | Correct representation |
| Unrelated protected route | Existing intended restriction |
Do not expand the test into high-volume production probing. A few reproducible cases are enough to verify the narrow change.
Check generated links and client defaults
If a site integration caused the failing request, inspect how it builds headers and URLs. An API client may accidentally be reused for HTML checks with a media type intended for JSON endpoints.
Keep the public page and API contracts distinct. A content checker should request a representation the page actually offers, while the server should have a deliberate response for unsupported choices.
Review redirects too. A request may start on an HTML route and end on an API endpoint whose acceptable formats differ. Fix the destination instead of changing headers until the accidental route responds.
For a publishing workflow, save one successful request example with the expected body marker. This gives future maintainers a concrete contract when they replace the HTTP client or gateway.
Document any required representation header only if it is an intentional public interface. Do not rely on an undocumented special header as the sole way to read an ordinary article.
Report the boundary that changed
State which request failed, which component returned 406, and which setting or route was corrected. Keep that separate from claims about indexing or AI citations.
A successful response confirms current delivery under the tested conditions. It does not establish that a particular crawler has revisited or that a search system selected the page.
If the error was intermittent, retain the request IDs and conditions that distinguish the failing case. A single successful browser visit should not erase evidence of a recurring public path.
After the next release of the responsible component, repeat the same small comparison. Negotiation defaults and rule updates can change independently of the article content.
For a client report, include the intended article sentence in the after evidence. That proves the request reached meaningful content rather than merely changing status.
Also distinguish an absent negotiation header from a header with an explicit restrictive value. Save the actual request sent by each client; do not assume a library and a browser use the same defaults. A fix that succeeds only with one unusually permissive test header can conceal a broken default route. Repeat the ordinary client request after repairing the confirmed cause.
Frequently asked questions
Does 406 always come from a firewall?
No. Applications and servers can return it during negotiation. Identify the actual response owner.
Should I send Accept: / to every endpoint?
Use the interface's documented representation contract. A broad header does not repair a route that is incorrectly configured.
Does a successful browser request prove a crawler gets the same result?
No. Compare the actual request conditions and keep local tests separate from verified operator requests.
Run the repaired public URL through the free Crawlito check to confirm the page can be inspected through its ordinary public path.
Filed under Access and robots.txt. Topics: http-status, troubleshooting, rendering