DNS and TLS errors: crawler failures before HTML arrives
Diagnose DNS, certificate, and connection failures that stop page retrieval before robots.txt or HTML can be checked, with a practical evidence checklist.
By Crawlito · 5 min read
In short
- DNS resolution and HTTPS connection failures happen before a crawler can inspect your page content.
- Check the exact hostname, including www and redirect destinations, instead of testing only the homepage.
- Do not disable certificate verification to turn a failed public HTTPS check into a passing result.
On this page
DNS and TLS problems can prevent crawlers from receiving any HTML at all. Diagnose them before editing robots.txt, schema, or page copy: confirm that the hostname resolves, the HTTPS connection succeeds, and the final destination serves the intended public page. A failure before an HTTP response is different from a page returning 403 or 404.
A browser that works on one laptop is useful evidence, but it is not a complete network test. Cached DNS answers, different connection paths, or a redirect to another hostname can explain inconsistent results. Keep the investigation focused on the failing hostname and the exact stage where retrieval stops.
Separate connection failures from HTTP responses
Start by capturing both diagnostic output and a bounded request:
curl --verbose --location --max-redirs 5 \
--connect-timeout 10 --max-time 30 \
--output page.html \
'https://www.example.com/guides/start' \
2>connection.txtUse this for an anonymous public URL. Verbose diagnostics can contain request and response details, so inspect and redact them before attaching the file to a shared issue. Avoid testing URLs that carry private share tokens or account parameters.
Classify the outcome rather than treating every failure as “blocked”:
| Observation | Investigation area |
|---|---|
| Hostname cannot be resolved | DNS records and resolver behavior |
| Connection cannot be established | Reachability, routing, firewall, listening service |
| Certificate validation fails | Hostname coverage, validity, certificate chain |
| HTTP 403 or challenge content appears | Application or edge access policy |
| HTTP 200 with missing article text | Response content or rendering |
Google's HTTP status documentation describes responses after an HTTP exchange takes place. If no such response arrives, recording an invented HTTP code obscures the problem. In your audit, use “no HTTP response captured” and preserve the actual client error.
Check every hostname in the public route
List the address visitors enter, the canonical hostname, and any redirect destination. A site may serve example.com correctly but have a certificate problem on www.example.com. If the first address cannot establish HTTPS, a redirect configured behind that connection cannot rescue the initial request.
In a hypothetical launch, the old domain redirects to a new domain, but the certificate on the old domain expires. Testing only the new domain gives a misleading green result. Old links still begin with the broken HTTPS address. The repair must include the old hostname for as long as those public redirects are needed.
Check DNS through tools available in your environment. For example, dig can inspect address records:
dig example.com A
dig example.com AAAA
dig www.example.com A
dig www.example.com AAAAThese commands report what the queried resolver returns. They do not independently establish that every resolver has the same data or that each returned address serves a healthy site. Save the resolver, timestamp, and answer when investigating propagation or inconsistent environments.
If a recent change is involved, compare the intended records with the provider's authoritative configuration. Do not repeatedly change DNS values while waiting for a previously made change to propagate. That makes the timeline harder to interpret and may introduce an additional fault.
Validate HTTPS without bypassing verification
Certificate checking is part of a successful public HTTPS test. Using curl's insecure option can help isolate a narrow diagnostic question, but the resulting request must not be reported as proof that ordinary HTTPS access is healthy. The public acceptance check should pass with verification enabled.
Inspect the certificate presented for the exact hostname, not just a certificate file stored on the server. A load balancer or CDN may terminate TLS before the request reaches your application. Renewing a certificate at the origin does not necessarily change what public clients receive at the edge.
The HTTPS scheme and authority checks are defined in RFC 9110. For this audit, the practical question is whether the client can authenticate the requested server and complete the connection. Do not broaden that into unsupported claims about every AI crawler's implementation.
When handing the issue to hosting support, include the failing hostname, the client error, the observed certificate details if available, and the time of the test. Do not send private keys, exported credentials, or unrelated server configuration. Public certificate metadata is usually enough to identify a deployment mismatch.
Investigate IPv4 and IPv6 differences carefully
If the domain publishes both address families and the ordinary request behaves inconsistently, separate tests can help locate the difference:
curl -4 --silent --show-error --max-time 20 \
--output ipv4.html 'https://www.example.com/'
curl -6 --silent --show-error --max-time 20 \
--output ipv6.html 'https://www.example.com/'A failed IPv6 test from a network without IPv6 connectivity proves little about the destination. First establish that the test environment supports the path being tested. Likewise, a successful IPv4 request does not validate an advertised IPv6 endpoint.
If one published address serves an old deployment, record that as a delivery inconsistency. Compare page content and headers as well as connection success. Two healthy connections can still reach different versions of a site. Coordinate the repair with the infrastructure owner before deleting records or changing production routing.
Treat these tests as targeted diagnosis, not a reason to scan address ranges or discover hidden origins. Public DNS answers and authorized infrastructure records are enough for an ordinary website audit.
Recheck the original journey after the repair
Repeat the exact failing URL with certificate verification enabled. Follow its redirects and confirm the final body contains the expected public content. Then test a second relevant entry point, such as the bare domain or an older published URL that redirects to the canonical site.
Keep the network and application outcomes separate in the report. “HTTPS succeeds” closes the connection issue. It does not close a subsequent noindex, challenge, or empty-HTML issue. Continue with status and redirect checks, then compare raw and rendered HTML.
For recurring failures, monitor a small set of public hostnames and expected content markers. Record certificate renewal ownership, DNS change ownership, and the deployment component that serves the public certificate. An alert without an owner can remain unresolved even when the technical symptom is obvious.
Finally, distinguish a repaired fetch from renewed search visibility. The former is directly testable now; the latter depends on downstream systems and timing. Avoid repeatedly changing working network configuration merely because a search result has not changed immediately.
Frequently asked questions
Can robots.txt fix a certificate error?
No. The client needs a working connection before it can fetch robots.txt over HTTPS. Repair the connection first, then inspect crawl policy.
Does a browser success prove every crawler can connect?
No. It confirms one successful request path. Investigate the reported failing path and use verified logs where available rather than generalizing from one device.
Should I remove IPv6 if one curl test fails?
Not without diagnosing the test environment and advertised endpoint. A local network without IPv6 can produce a failure even when the site's IPv6 service is healthy.
What should the audit mark when no status code arrives?
Mark the request as a connection or resolution failure with the actual error. Do not classify it as a 404, robots exclusion, or content-quality problem.
Once the connection works, run the URL through the free page check and continue with the AI readiness checklist.
Filed under Access and robots.txt. Topics: http-status, dns, tls, ai-crawlers