Skip to content

HEAD vs GET requests: avoiding false crawler audit results

Learn why a successful HEAD request cannot prove crawlers receive your content, and build a small GET-based check with headers and saved HTML.

By Crawlito · 5 min read

In short

  • HEAD returns response metadata without the page body, so it cannot establish that article text is present.
  • Use a GET request to capture headers and content together when investigating crawler access.
  • A simulated user agent tests a request condition but does not reproduce a verified crawler's network identity.
On this page

A HEAD request is useful for checking response metadata, but a successful HEAD response does not prove that a crawler can read your page. Use GET to retrieve the body, inspect the actual text, and preserve the headers from that same request. This matters when an audit reports a healthy URL even though visitors or automated fetchers receive a challenge, empty template, or error message.

The distinction is small enough to miss in a terminal command. curl -I asks for HEAD. A plain curl request normally uses GET. If your report says “content accessible” while the evidence contains no body, the test does not support the conclusion.

Understand what each request can establish

MDN's HEAD reference describes HEAD as requesting the headers associated with a GET response without its content. That makes HEAD convenient for metadata checks, but it removes exactly the evidence needed for a content-readability audit.

Use the two methods for different questions:

QuestionUseful requestWhat to inspect
What status and headers does HEAD return?HEADStatus, redirect, content metadata
Does the response contain the article?GETBody text and headers together
Does a redirect lead to the intended page?GET with controlled redirect followingFinal address and final body
Does the application require browser execution?GET followed by a browser comparisonRaw text versus rendered text

A server or intermediary can also handle the methods differently. A HEAD failure therefore deserves investigation before calling the page inaccessible. Conversely, HEAD success cannot overrule a GET response containing the wrong content. Record both observations accurately rather than choosing the more reassuring one.

The wider meaning of response codes is covered in HTTP status codes and crawlers. Here the focus is the evidence collected by the test, not a particular search engine's treatment of every status.

Capture a useful GET response

For a public page you own or are authorized to test, save headers and body to separate files:

bash
curl --silent --show-error --location --max-redirs 5 \
  --connect-timeout 10 --max-time 30 \
  --dump-header page-headers.txt \
  --output page.html \
  --write-out 'Final URL: %{url_effective}\nStatus: %{http_code}\n' \
  'https://www.example.com/guides/setup'

The redirect and timeout values are illustrative test boundaries, not crawler specifications. Adjust them for your own diagnostic environment. A timeout means this request did not complete within your chosen limit; it does not prove that every crawler has the same limit.

Inspect page.html for the page heading and a distinctive sentence from the main content. Then look for unexpected text such as a sign-in prompt or challenge. A 200 response containing “Please enable cookies” is different from a 200 response containing the guide.

With redirect following enabled, the header file may contain several response blocks. Identify the final response and preserve the earlier blocks if the path matters. Do not report the first status as though it described the final document.

Compare methods without changing other variables

Run a HEAD request for the same exact URL and save it separately:

bash
curl --silent --show-error --head --location --max-redirs 5 \
  --connect-timeout 10 --max-time 30 \
  'https://www.example.com/guides/setup'

Keep scheme, hostname, path, query parameters, and authentication state consistent. Comparing HEAD on example.com with GET on www.example.com introduces another variable. So does testing an authenticated browser page against an anonymous terminal request.

If the results differ, work through the delivery path: CDN method handling, proxy rules, application routing, and middleware. Look for a policy that explicitly rejects HEAD or handles it through a simplified route. Do not immediately loosen bot protection across the whole site. Fix the specific method inconsistency if it is unintended.

Use a simple record with four fields: request method, requested URL, final response status, and observed content. The body field for HEAD should say “not requested,” not “empty content.” That wording prevents a method characteristic from being misclassified as a site defect.

Know what a user-agent test does not reproduce

You can add a documented crawler user-agent string to a diagnostic request, but the request still originates from your own network. It does not gain the crawler operator's verified IP address, infrastructure, cookies, or fetching behavior. A security rule that verifies network identity may legitimately treat your imitation differently.

Use user-agent comparisons to discover application branches. For example, if one request gets the article and another gets an empty template, you have a reproducible difference worth investigating. The defensible conclusion is “these two request configurations receive different bodies,” not “the real crawler is blocked everywhere.”

Our curl crawler testing guide covers request examples, and crawler verification explains why identity needs separate evidence. Combine synthetic tests with verified access logs when available.

Avoid putting real session cookies or authorization headers into commands copied into shared tickets. Prefer an anonymous test for public content. If authentication is essential to the reproduction, use an approved private diagnostic channel and redact sensitive values before sharing results.

Turn the result into a precise developer ticket

Consider a hypothetical audit where HEAD returns 200, GET returns 200, but GET contains only a JavaScript loader. The ticket should describe missing main content in raw HTML. It should not say the HTTP endpoint is unavailable. The application responds; the content required by the audit is absent from that representation.

Another hypothetical case has HEAD returning 405 while GET returns the complete article. Here the ticket concerns unsupported HEAD handling only if that violates a requirement of your monitoring or infrastructure. Do not claim that the page has failed a GET-based content check.

Include the command, timestamp with time zone, final URL, relevant headers, and a short body excerpt. Define the expected result in terms of observable content. “The anonymous GET response contains the setup heading and prerequisites” is a usable acceptance criterion. “Make the page AI friendly” is not.

After a fix, repeat the original request before adding variations. Then check one neighboring page using the same template. That small regression check helps establish whether the change corrected a shared rendering path or only one route. Preserve failures as evidence instead of overwriting every file with the latest successful output.

Frequently asked questions

Is curl -I enough for an SEO or AI readiness audit?

It is enough for a limited HEAD metadata check. It is insufficient for confirming that the main text, links, and structured data are delivered in the page body.

Does a GET request execute JavaScript?

A normal curl GET does not execute the page's JavaScript. Compare its output with a browser when you need to identify browser-generated content.

Should HEAD and GET always have identical output?

They do not have identical bodies because HEAD does not return the body. Compare their relevant metadata and investigate differences in routing or status without confusing that with a content comparison.

Can a 200 response still fail a content check?

Yes. Read the body to establish whether it contains the intended document, a challenge, or a mostly empty template.

Use the free check as a starting point, then attach a saved GET response when handing a specific content problem to a developer.

Filed under Access and robots.txt. Topics: http-status, curl, ai-crawlers