python-requests, Scrapy and Go-http-client in your logs
What the python-requests user agent, Scrapy, Go-http-client and curl strings mean in your logs, which of them read robots.txt, and why they are not AI crawlers.
By Crawlito · 8 min read
In short
- A user agent such as python-requests/2.32.3, Go-http-client/1.1 or curl/8.x is the default a library sends when the programmer did not set one, so it names the tool, not the person or company running it.
- Scrapy's default user agent is Scrapy/VERSION (+https://scrapy.org), and projects created with scrapy startproject obey robots.txt because the generated settings.py sets ROBOTSTXT_OBEY = True.
- Plain HTTP clients such as Requests, Go's net/http and curl fetch the URLs a script asks for and do not read robots.txt, so a Disallow rule does not stop them.
- The AI crawlers documented by OpenAI, Anthropic, Perplexity and others identify themselves with named tokens such as GPTBot or ClaudeBot, not with bare library defaults.
On this page
A user agent like python-requests/2.32.3, Scrapy/2.13.3 (+https://scrapy.org), Go-http-client/1.1 or curl/8.5.0 in your access log is the default string an HTTP library or command-line tool sends when the person who wrote the script did not set their own. It tells you which tool made the request, not who ran it or why. None of the AI crawlers that operators document use these strings: GPTBot, ClaudeBot, PerplexityBot and the rest send named tokens. So these lines are scripts, which might be anything from your own uptime check to a scraper. Of these four tools, only Scrapy reads robots.txt, and only while the project keeps ROBOTSTXT_OBEY switched on, as new projects do.
This post covers what each string is, what each tool's documentation says about robots.txt, and how to pull them out of your logs. Deciding whether to allow, slow or block a particular one is covered in how to identify an unknown bot user agent.
What these default user agent strings are
RFC 9110 says a user agent should send a User-Agent header in each request unless it is configured not to. Libraries fill it in with their own name and version unless the code overrides it. Here is what each tool documents:
| String in your log | Tool | Where the default is documented |
|---|---|---|
python-requests/2.32.3 | Requests (Python) | requests.utils.default_user_agent() returns python-requests/ plus the library version (source); the advanced usage guide shows it in r.request.headers |
Scrapy/2.13.3 (+https://scrapy.org) | Scrapy (Python) | The USER_AGENT setting defaults to Scrapy/VERSION (+https://scrapy.org) |
Go-http-client/1.1 | Go net/http | Request.Write uses Go-http-client/1.1 if the header has no User-Agent value |
curl/8.5.0 | curl | The man page says curl uses curl/VERSION by default |
Python-urllib/3.12 | Python standard library | The urllib.request docs give the pattern, for example Python-urllib/2.6 on Python 2.6 |
Wget/1.21.4 | GNU Wget | The manual says Wget normally identifies as Wget/version |
The version numbers in the first column are examples; yours will vary. Match on the part before the slash. For Go in particular, the documented Go-http-client/1.1 is not the only form: the HTTP/2 transport in Go's own source code sets Go-http-client/2.0 (golang.org/x/net/http2), so a Go client on an HTTP/2 connection can show up with that string instead.
Requests also sends Accept: */*, Accept-Encoding and Connection: keep-alive by default, and Scrapy's default request headers include Accept-Language: en. You will not normally see those in an access log, but they matter if you write firewall rules on header combinations.
Who sends these requests
The honest answer is that the string cannot tell you. A default user agent means only that the code's author did not change it. In practice, the same python-requests string comes from:
- Your own monitoring, cron jobs, deployment hooks and internal tools.
- Third-party services that check links, feeds, uptime or webhooks and never set a name.
- Researchers, students and hobbyists running one-off scripts.
- Scrapers collecting prices, content or contact details, sometimes for AI data sets.
- Vulnerability scanners and login probes.
The reverse is also true: a scraper can change its user agent with one line of code. curl's own man page shows -A to set any string, and the urllib docs note that the headers argument is often used to "spoof" the User-Agent. Anyone who wants to look like a browser or like GPTBot can. The defaults you see are, if anything, from people who did not bother to hide.
So treat these strings as a description of the tool, then look at behaviour: which paths, how many requests, from which IPs, and whether robots.txt was fetched first.
Do these tools read robots.txt?
This is the difference that matters most, and the tools fall into two groups.
Scrapy: obeys robots.txt in new projects
Scrapy is a crawling framework, and it ships with robots.txt support. Its settings reference lists ROBOTSTXT_OBEY with a default of True and a fallback of False. The fallback is the value Scrapy uses when a project does not set it; the docs explain that the value is False "for historical reasons", but the settings.py file that scrapy startproject generates sets it to True. The same template sets DOWNLOAD_DELAY = 1 and CONCURRENT_REQUESTS_PER_DOMAIN = 1, and carries the comment "Crawl responsibly by identifying yourself (and your website) on the user-agent".
When it is on, Scrapy's RobotsTxtMiddleware holds each request while it downloads the site's robots.txt, then filters out requests that robots.txt forbids. It matches rules against ROBOTSTXT_USER_AGENT if set, otherwise the request's User-Agent header or the USER_AGENT setting. Two caveats:
- A spider author can set
ROBOTSTXT_OBEY = False, or set thedont_obey_robotstxtmeta key on individual requests to skip the check. - With the default user agent, the rule that applies is the one for
Scrapy, or your*group. A spider with a custom user agent is matched on that custom name instead.
Requests, Go, curl and urllib: never read it
Requests, Go's net/http and curl are HTTP clients, not crawlers. Their documentation describes sending the request you give them; none of them describes fetching or applying robots.txt. A Disallow line has no effect on a script built with them unless the author wrote that check in. Python does ship a parser, urllib.robotparser, but nothing calls it automatically.
Wget sits in between. Its robot exclusion page says it honors robots.txt when downloading recursively, and that users can turn this off with -e robots=off. A single wget https://example.com/page is not described as checking robots.txt.
| Tool | Reads robots.txt by default | Can be turned off |
|---|---|---|
| Scrapy (new project) | Yes, ROBOTSTXT_OBEY = True | Yes, per project or per request |
| Wget | When recursive | Yes, -e robots=off |
| Requests, urllib | No | Not applicable |
Go net/http | No | Not applicable |
| curl | No | Not applicable |
This is why robots.txt is the wrong tool for most of this traffic. If a script built on Requests is hurting your server, a rate limit or a firewall rule is what stops it.
Are these AI crawlers?
Not the documented ones. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl and the other operators in the AI crawler user agents list each publish a named token for their crawlers, such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-User or PerplexityBot, usually inside a longer string with a link to their documentation. None documents a crawler that sends a bare python-requests or Go-http-client string.
What the logs cannot rule out is an undocumented AI scraper, or a small AI app whose developer used Requests and never set a user agent. Nothing in the string distinguishes those from any other script. If you want to know whether a request really came from an AI company, the IP is the evidence, as described in how to verify real AI crawlers. And blocking these library strings will not affect ChatGPT search, Claude or Perplexity, because their crawlers do not use them.
How to find library user agents in your logs
These commands assume the nginx or Apache combined log format, where splitting on double quotes puts the user agent in field 6. AI crawlers in server logs explains the format and where the files live.
Set the patterns once:
LIBS='python-requests|Scrapy|Go-http-client|curl/|Python-urllib|Wget/'Count hits per tool:
awk -F'"' -v re="$LIBS" 'match($6, re) { print substr($6, RSTART, RLENGTH) }' access.log \
| sort | uniq -c | sort -rnSee which IPs send the most, with their status codes:
awk -F'"' -v re="$LIBS" '$6 ~ re { split($1, a, " "); split($3, s, " "); print a[1], s[1] }' access.log \
| sort | uniq -c | sort -rn | head -20Check whether a Scrapy client fetched robots.txt before crawling. A well-behaved spider asks for /robots.txt from the same IP before anything else:
awk -F'"' '$6 ~ /Scrapy/ { split($1, a, " "); split($2, r, " "); print a[1], r[2] }' access.log \
| awk '{ print $1, ($2 == "/robots.txt" ? "robots.txt" : "page") }' \
| sort | uniq -cAn IP with many page lines and no robots.txt line probably has robots.txt checking turned off, or is a different client borrowing the Scrapy name. Before you conclude that, check older, rotated log files: Scrapy asks for each site's robots.txt once, when the crawl first reaches that site, so that request may sit in an earlier file. A crawl spread over several IPs may also fetch robots.txt from only one of them.
Then read the results:
- Your own IPs or your hosting provider's. Probably your own tools. Give them a proper user agent so you can tell them apart next time.
- A few requests to one URL, such as a feed or an API endpoint. Usually an integration or a monitor. Leave it alone unless it causes problems.
- Hundreds of requests across many pages from one IP. A scraper. Rate limiting is the proportionate response; rate limiting AI crawlers shows how to set limits without catching the crawlers you want.
- Requests to
/wp-login.php,/.envor admin paths. A scanner. Block it at the firewall.
How to name your own scripts so they are not mistaken for scrapers
If some of these lines turn out to be yours, change the default. Scrapy's docs recommend a user agent that identifies you with a URL or email address so site owners can contact you instead of blocking you:
# Requests
import requests
s = requests.Session()
s.headers.update({"User-Agent": "acme-linkcheck/1.0 (+https://acme.example/bot)"})
s.get("https://example.com/")# curl
curl -A "acme-uptime/1.0 (+https://acme.example/bot)" https://example.com/healthIn Go, set req.Header.Set("User-Agent", "...") on the request before sending it. With a named agent you can write robots.txt groups and firewall exceptions for your own tools, and filter them out of reports. To fetch your own pages the way AI crawlers do, rather than as a script, see checking your page as an AI crawler with curl.
Frequently asked questions
Is python-requests in my logs an AI crawler?
Not one that any AI operator documents. The documented AI crawlers send named tokens such as GPTBot or ClaudeBot, while python-requests is the default of a general Python library used by all kinds of scripts.
Will a robots.txt Disallow stop python-requests or curl?
No. Requests, Go's net/http and curl do not read robots.txt; they fetch the URL they are given. Use a rate limit or a firewall rule if you need to stop them.
Does Scrapy respect robots.txt?
In projects created with scrapy startproject, yes, because the generated settings set ROBOTSTXT_OBEY = True. A spider author can turn it off for the whole project or for single requests.
Should I block all library user agents?
Not blindly. Your own monitoring, payment webhooks or feed readers may send them, so check the IPs and paths first. Blocking them does not affect documented AI search crawlers, which use their own names.
What does Go-http-client/1.1 mean?
It is the default User-Agent that Go's net/http package sends when the program sets none; over HTTP/2 the same client may send Go-http-client/2.0. It identifies the programming language library, not the company or service behind the request.
Check what the crawlers you want actually get
Library traffic is noise around the requests that matter. Once you have sorted it out, run the free check on an important page to see whether the named AI crawlers can fetch it and what robots.txt tells each of them.
Filed under AI crawlers. Topics: server-logs, ai-crawlers, bot-protection, rate-limiting