Skip to content

python-requests, Scrapy and Go-http-client in your logs

What the python-requests user agent, Scrapy, Go-http-client and curl strings mean in your logs, which of them read robots.txt, and why they are not AI crawlers.

By Crawlito · 8 min read

In short

  • A user agent such as python-requests/2.32.3, Go-http-client/1.1 or curl/8.x is the default a library sends when the programmer did not set one, so it names the tool, not the person or company running it.
  • Scrapy's default user agent is Scrapy/VERSION (+https://scrapy.org), and projects created with scrapy startproject obey robots.txt because the generated settings.py sets ROBOTSTXT_OBEY = True.
  • Plain HTTP clients such as Requests, Go's net/http and curl fetch the URLs a script asks for and do not read robots.txt, so a Disallow rule does not stop them.
  • The AI crawlers documented by OpenAI, Anthropic, Perplexity and others identify themselves with named tokens such as GPTBot or ClaudeBot, not with bare library defaults.
On this page

A user agent like python-requests/2.32.3, Scrapy/2.13.3 (+https://scrapy.org), Go-http-client/1.1 or curl/8.5.0 in your access log is the default string an HTTP library or command-line tool sends when the person who wrote the script did not set their own. It tells you which tool made the request, not who ran it or why. None of the AI crawlers that operators document use these strings: GPTBot, ClaudeBot, PerplexityBot and the rest send named tokens. So these lines are scripts, which might be anything from your own uptime check to a scraper. Of these four tools, only Scrapy reads robots.txt, and only while the project keeps ROBOTSTXT_OBEY switched on, as new projects do.

This post covers what each string is, what each tool's documentation says about robots.txt, and how to pull them out of your logs. Deciding whether to allow, slow or block a particular one is covered in how to identify an unknown bot user agent.

What these default user agent strings are

RFC 9110 says a user agent should send a User-Agent header in each request unless it is configured not to. Libraries fill it in with their own name and version unless the code overrides it. Here is what each tool documents:

String in your logToolWhere the default is documented
python-requests/2.32.3Requests (Python)requests.utils.default_user_agent() returns python-requests/ plus the library version (source); the advanced usage guide shows it in r.request.headers
Scrapy/2.13.3 (+https://scrapy.org)Scrapy (Python)The USER_AGENT setting defaults to Scrapy/VERSION (+https://scrapy.org)
Go-http-client/1.1Go net/httpRequest.Write uses Go-http-client/1.1 if the header has no User-Agent value
curl/8.5.0curlThe man page says curl uses curl/VERSION by default
Python-urllib/3.12Python standard libraryThe urllib.request docs give the pattern, for example Python-urllib/2.6 on Python 2.6
Wget/1.21.4GNU WgetThe manual says Wget normally identifies as Wget/version

The version numbers in the first column are examples; yours will vary. Match on the part before the slash. For Go in particular, the documented Go-http-client/1.1 is not the only form: the HTTP/2 transport in Go's own source code sets Go-http-client/2.0 (golang.org/x/net/http2), so a Go client on an HTTP/2 connection can show up with that string instead.

Requests also sends Accept: */*, Accept-Encoding and Connection: keep-alive by default, and Scrapy's default request headers include Accept-Language: en. You will not normally see those in an access log, but they matter if you write firewall rules on header combinations.

Who sends these requests

The honest answer is that the string cannot tell you. A default user agent means only that the code's author did not change it. In practice, the same python-requests string comes from:

  • Your own monitoring, cron jobs, deployment hooks and internal tools.
  • Third-party services that check links, feeds, uptime or webhooks and never set a name.
  • Researchers, students and hobbyists running one-off scripts.
  • Scrapers collecting prices, content or contact details, sometimes for AI data sets.
  • Vulnerability scanners and login probes.

The reverse is also true: a scraper can change its user agent with one line of code. curl's own man page shows -A to set any string, and the urllib docs note that the headers argument is often used to "spoof" the User-Agent. Anyone who wants to look like a browser or like GPTBot can. The defaults you see are, if anything, from people who did not bother to hide.

So treat these strings as a description of the tool, then look at behaviour: which paths, how many requests, from which IPs, and whether robots.txt was fetched first.

Do these tools read robots.txt?

This is the difference that matters most, and the tools fall into two groups.

Scrapy: obeys robots.txt in new projects

Scrapy is a crawling framework, and it ships with robots.txt support. Its settings reference lists ROBOTSTXT_OBEY with a default of True and a fallback of False. The fallback is the value Scrapy uses when a project does not set it; the docs explain that the value is False "for historical reasons", but the settings.py file that scrapy startproject generates sets it to True. The same template sets DOWNLOAD_DELAY = 1 and CONCURRENT_REQUESTS_PER_DOMAIN = 1, and carries the comment "Crawl responsibly by identifying yourself (and your website) on the user-agent".

When it is on, Scrapy's RobotsTxtMiddleware holds each request while it downloads the site's robots.txt, then filters out requests that robots.txt forbids. It matches rules against ROBOTSTXT_USER_AGENT if set, otherwise the request's User-Agent header or the USER_AGENT setting. Two caveats:

  • A spider author can set ROBOTSTXT_OBEY = False, or set the dont_obey_robotstxt meta key on individual requests to skip the check.
  • With the default user agent, the rule that applies is the one for Scrapy, or your * group. A spider with a custom user agent is matched on that custom name instead.

Requests, Go, curl and urllib: never read it

Requests, Go's net/http and curl are HTTP clients, not crawlers. Their documentation describes sending the request you give them; none of them describes fetching or applying robots.txt. A Disallow line has no effect on a script built with them unless the author wrote that check in. Python does ship a parser, urllib.robotparser, but nothing calls it automatically.

Wget sits in between. Its robot exclusion page says it honors robots.txt when downloading recursively, and that users can turn this off with -e robots=off. A single wget https://example.com/page is not described as checking robots.txt.

ToolReads robots.txt by defaultCan be turned off
Scrapy (new project)Yes, ROBOTSTXT_OBEY = TrueYes, per project or per request
WgetWhen recursiveYes, -e robots=off
Requests, urllibNoNot applicable
Go net/httpNoNot applicable
curlNoNot applicable

This is why robots.txt is the wrong tool for most of this traffic. If a script built on Requests is hurting your server, a rate limit or a firewall rule is what stops it.

Are these AI crawlers?

Not the documented ones. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl and the other operators in the AI crawler user agents list each publish a named token for their crawlers, such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-User or PerplexityBot, usually inside a longer string with a link to their documentation. None documents a crawler that sends a bare python-requests or Go-http-client string.

What the logs cannot rule out is an undocumented AI scraper, or a small AI app whose developer used Requests and never set a user agent. Nothing in the string distinguishes those from any other script. If you want to know whether a request really came from an AI company, the IP is the evidence, as described in how to verify real AI crawlers. And blocking these library strings will not affect ChatGPT search, Claude or Perplexity, because their crawlers do not use them.

How to find library user agents in your logs

These commands assume the nginx or Apache combined log format, where splitting on double quotes puts the user agent in field 6. AI crawlers in server logs explains the format and where the files live.

Set the patterns once:

bash
LIBS='python-requests|Scrapy|Go-http-client|curl/|Python-urllib|Wget/'

Count hits per tool:

bash
awk -F'"' -v re="$LIBS" 'match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log \
  | sort | uniq -c | sort -rn

See which IPs send the most, with their status codes:

bash
awk -F'"' -v re="$LIBS" '$6 ~ re { split($1, a, " "); split($3, s, " "); print a[1], s[1] }' access.log \
  | sort | uniq -c | sort -rn | head -20

Check whether a Scrapy client fetched robots.txt before crawling. A well-behaved spider asks for /robots.txt from the same IP before anything else:

bash
awk -F'"' '$6 ~ /Scrapy/ { split($1, a, " "); split($2, r, " "); print a[1], r[2] }' access.log \
  | awk '{ print $1, ($2 == "/robots.txt" ? "robots.txt" : "page") }' \
  | sort | uniq -c

An IP with many page lines and no robots.txt line probably has robots.txt checking turned off, or is a different client borrowing the Scrapy name. Before you conclude that, check older, rotated log files: Scrapy asks for each site's robots.txt once, when the crawl first reaches that site, so that request may sit in an earlier file. A crawl spread over several IPs may also fetch robots.txt from only one of them.

Then read the results:

  • Your own IPs or your hosting provider's. Probably your own tools. Give them a proper user agent so you can tell them apart next time.
  • A few requests to one URL, such as a feed or an API endpoint. Usually an integration or a monitor. Leave it alone unless it causes problems.
  • Hundreds of requests across many pages from one IP. A scraper. Rate limiting is the proportionate response; rate limiting AI crawlers shows how to set limits without catching the crawlers you want.
  • Requests to /wp-login.php, /.env or admin paths. A scanner. Block it at the firewall.

How to name your own scripts so they are not mistaken for scrapers

If some of these lines turn out to be yours, change the default. Scrapy's docs recommend a user agent that identifies you with a URL or email address so site owners can contact you instead of blocking you:

python
# Requests
import requests
s = requests.Session()
s.headers.update({"User-Agent": "acme-linkcheck/1.0 (+https://acme.example/bot)"})
s.get("https://example.com/")
bash
# curl
curl -A "acme-uptime/1.0 (+https://acme.example/bot)" https://example.com/health

In Go, set req.Header.Set("User-Agent", "...") on the request before sending it. With a named agent you can write robots.txt groups and firewall exceptions for your own tools, and filter them out of reports. To fetch your own pages the way AI crawlers do, rather than as a script, see checking your page as an AI crawler with curl.

Frequently asked questions

Is python-requests in my logs an AI crawler?

Not one that any AI operator documents. The documented AI crawlers send named tokens such as GPTBot or ClaudeBot, while python-requests is the default of a general Python library used by all kinds of scripts.

Will a robots.txt Disallow stop python-requests or curl?

No. Requests, Go's net/http and curl do not read robots.txt; they fetch the URL they are given. Use a rate limit or a firewall rule if you need to stop them.

Does Scrapy respect robots.txt?

In projects created with scrapy startproject, yes, because the generated settings set ROBOTSTXT_OBEY = True. A spider author can turn it off for the whole project or for single requests.

Should I block all library user agents?

Not blindly. Your own monitoring, payment webhooks or feed readers may send them, so check the IPs and paths first. Blocking them does not affect documented AI search crawlers, which use their own names.

What does Go-http-client/1.1 mean?

It is the default User-Agent that Go's net/http package sends when the program sets none; over HTTP/2 the same client may send Go-http-client/2.0. It identifies the programming language library, not the company or service behind the request.

Check what the crawlers you want actually get

Library traffic is noise around the requests that matter. Once you have sorted it out, run the free check on an important page to see whether the named AI crawlers can fetch it and what robots.txt tells each of them.

Filed under AI crawlers. Topics: server-logs, ai-crawlers, bot-protection, rate-limiting