Skip to content

Slow TTFB and crawlers: response time, timeouts and fixes

How slow server response time affects Googlebot and AI crawlers, what Google documents about crawl rate and timeouts, and how to measure TTFB with curl.

By Crawlito · 6 min read

In short

  • Google says its crawl capacity limit goes up when response times, including time to first byte, stay stable, and goes down when the site slows or returns 5xx or 429.
  • Google treats network timeouts and connection resets like 5xx errors, slows crawling immediately, and can drop unreachable URLs from its index within days.
  • OpenAI, Anthropic and Perplexity do not document crawler timeouts or response-time limits at the time of writing.
  • curl's time_starttransfer value measures time to first byte from the command line; test with a cold cache and from more than one location.
  • Serving HTML from a cache, answering conditional requests with 304 and removing slow work from the request path usually fix crawler-facing slowness.
On this page

A slow server response makes crawlers fetch less of your site. Google documents this directly: when response times, including time to first byte (TTFB), rise, Googlebot lowers its crawl rate, and when requests time out, Google treats them like server errors and can drop the URLs from its index within days. AI crawler operators do not publish timeouts or response-time limits, so the safe assumption is that a slow or timed-out response simply means the crawler did not get the page.

The fix starts with measurement. Below is what Google says, what the AI operators do not say, how to measure TTFB the way a crawler experiences it, and the changes that usually help.

What Google documents about response time and crawl rate

Google's crawl budget documentation describes a crawl capacity limit, also called hostload, that its crawlers calculate for each site. It moves with your server's behaviour:

  • It goes up if the site responds consistently and response times, including latency and time to first byte, stay stable or improve.
  • It goes down if the site slows down, meaning latency rises or responses take longer.
  • It goes down if the site returns 5xx server errors or rate-limiting signals such as 429.

The same page says that if Google can load and render pages faster, it may be able to read more content from your site, and it recommends optimizing server response times and using HTTP caching, where a 304 Not Modified response saves bandwidth and server resources.

Timeouts count as server errors

Google's page on network and DNS errors is blunter. Google treats network timeouts, connection resets and DNS errors similarly to 5xx errors. With network errors, crawling starts slowing down immediately, and already indexed URLs that stay unreachable are removed from the index within days. Google does not publish how many seconds it waits before giving up, so do not design around a number.

Google's guide to reducing crawl rate adds a warning for anyone tempted to slow crawlers down on purpose with errors: a URL that returns 500, 503 or 429 for multiple days may be dropped from the index.

What AI crawler operators document

At the time of writing, the crawler pages for OpenAI, Anthropic and Perplexity describe what each crawler is for, how to allow or block it, and how to verify its IP addresses. None of them states a timeout, a response-time limit, or how the crawler adjusts its rate to a slow server.

That leaves you with what you can observe. In your logs, a slow response to an AI crawler usually shows up in one of three ways: a long request time, a status code written by a proxy such as 499 (nginx's code for a client that closed the connection) or 504, or no matching request at all because the connection failed before it was logged. User-triggered fetchers like ChatGPT-User work while a person waits for an answer, so a page that takes many seconds to start responding is the most likely to be skipped there. That is an inference from how they work, not a documented limit.

How to measure TTFB with curl

TTFB is the time from starting the request until the first byte of the response arrives. web.dev counts redirects, DNS lookup, connection and TLS setup within it, and suggests most sites aim for 0.8 seconds or less as a rough guide for users. Google does not publish a crawler threshold.

curl reports each phase with its -w option:

bash
curl -s -o /dev/null \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  -w "dns %{time_namelookup}s  connect %{time_connect}s  tls %{time_appconnect}s  ttfb %{time_starttransfer}s  total %{time_total}s  status %{http_code}\n" \
  https://www.example.com/pricing

Typical output:

dns 0.012s  connect 0.041s  tls 0.087s  ttfb 1.934s  total 2.011s  status 200

Here the network is fast and the server spent almost two seconds before sending anything. That gap between tls and ttfb is server work: rendering, database queries, calls to other APIs.

Make the test realistic:

  1. Repeat it. One request proves little. Run it ten times in a loop and look at the slowest results, not the fastest.
  2. Test a cold cache. Crawlers often request pages no human has visited recently, so your CDN may not have them cached. Check the cache header your CDN returns (cf-cache-status, x-cache or age) and test a page that shows a miss.
  3. Test with and without a crawler user agent. If the crawler user agent is much slower, a bot rule, challenge or prerendering service is in the path. Seeing your page as an AI crawler with curl shows more of these checks.
  4. Test from another region. A server in one country can look fast from your office and slow from a crawler's data centre.
bash
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{time_starttransfer}\n" https://www.example.com/pricing
done | sort -n | tail -3

How to find slow crawler responses in your logs

Your own logs show what crawlers actually experienced. In nginx, add timing fields to the log format:

nginx
log_format timed '$remote_addr [$time_local] "$request" $status '
                 '$request_time $upstream_response_time "$http_user_agent"';
access_log /var/log/nginx/access.log timed;

$request_time is the full time nginx spent on the request, and $upstream_response_time is the time your application took. Then list the slowest requests from AI crawlers:

bash
grep -E "OAI-SearchBot|ChatGPT-User|PerplexityBot|Claude-SearchBot|Googlebot" /var/log/nginx/access.log \
  | awk '{print $(NF-2), $5, $6}' | sort -rn | head -20

The field positions depend on your format, so adjust the awk columns to match. Before trusting crawler names in logs, confirm the IPs are real, as described in finding AI crawlers in server logs.

For Google specifically, Search Console's Crawl Stats report shows average response time for all resources Google fetched and a host status that flags server connectivity problems found in the past 90 days. A rising average response time that lines up with a falling number of crawl requests is the pattern Google's documentation describes.

Common causes of slow responses to crawlers, and fixes

CauseHow to spot itFix
HTML not cached at the CDNCache header shows miss or bypass on HTMLCache HTML at the edge with a short TTL, or serve static output
Serverless cold startsFirst request after idle is much slowerKeep functions warm, or prerender pages that rarely change
Slow database or API calls during renderLarge gap between connect and TTFB on every requestCache query results, move non-essential calls out of the request path
Bot protection adds a challenge or delayCrawler user agent slower than a browser user agentExempt verified crawlers, as in WAF challenge pages
Crawler bursts overload the originSlowness lines up with crawler traffic spikesRate-limit carefully and add capacity, see rate limiting AI crawlers
Full pages re-sent when nothing changedFew or no 304 responses in logsSupport If-Modified-Since and If-None-Match, see conditional requests

Two fixes cover most cases. First, take rendering out of the request: static generation or an edge cache means the server sends stored HTML instead of building it. Second, make sure the cache actually serves crawlers. A common surprise is a CDN rule that bypasses the cache for requests with a bot user agent or with cookies, so crawlers always hit the slow origin.

Be careful with one shortcut. Returning 503 or 429 to slow crawlers down protects the server for a moment, but Google says URLs that keep returning those codes for days may drop out, and AI operators do not document how they react. Use it for short incidents, not as a standing policy.

Frequently asked questions

What TTFB is good enough for crawlers?

No crawler operator publishes a threshold. web.dev suggests 0.8 seconds or less as a rough guide for users, and Google says its crawlers fetch more from sites whose response times stay stable or improve.

Do AI crawlers time out on slow pages?

They do not document a timeout. A request that takes too long or fails gives the crawler nothing to use, so treat slow and failed responses as missed fetches.

Does a slow server affect Google's AI Overviews?

Indirectly. AI Overviews draw on pages Google has crawled and indexed, and Google crawls less from a slow site and can drop URLs it cannot reach. A faster response does not decide whether a page is used in an answer.

Should I measure TTFB from my browser?

Use it as a starting point only. Your browser may hold cached connections and cookies that crawlers do not have. curl from a server in another region is closer to what a crawler sees.

Next step

Measure five important pages with the curl loop above and compare the slowest results with your crawler logs. If you want each page's response, headers and crawler access checked with evidence, the free check covers one page with no account, and an audit covers up to 500.

Filed under Rendering and content. Topics: crawl-budget, server-logs, caching, curl, google, ai-crawlers