Skip to content

robots.txt wildcards: how * and $ pattern matching works

How robots.txt wildcards work under RFC 9309 and Google's rules: what * and $ match, how the longest rule wins, and worked examples of common mistakes.

By Crawlito · 6 min read

In short

  • robots.txt supports only two special characters in paths: * matches zero or more of any character and $ marks the end of the URL; everything else, including . and ?, is literal.
  • Every rule is a prefix match, so Disallow: /fish already covers /fish.html and /fishing, and a trailing * adds nothing.
  • When several rules match, the one with the longest path pattern wins, wildcards included, and Allow should win a tie under RFC 9309.
  • Paths are case-sensitive and compared in percent-encoded form, so /Shop/ and /shop/ need separate rules.
  • Regular expression syntax such as .* or (a|b) is not supported, and a rule should start with / to be valid under the RFC grammar.
On this page

robots.txt paths support two special characters: *, which matches zero or more of any character, and $, which marks the end of the URL. Both are defined in RFC 9309, Section 2.2.3, and Google documents the same behaviour in its robots.txt specification. Everything else in a path is matched literally, from the first character, and when several rules match a URL the longest pattern wins.

That is the whole syntax. Most robots.txt mistakes with wildcards come from treating the rules like regular expressions or forgetting that every rule is already a prefix match. This guide shows exactly what each pattern matches, with examples you can test.

What * and $ mean in robots.txt

RFC 9309 defines three special characters in a robots.txt file:

CharacterMeaningExample
*Zero or more instances of any characterDisallow: /*.pdf
$End of the match patternDisallow: /*.pdf$
#Starts a comment# block internal search

Two other rules shape every match:

  • Matching starts at the first character of the path. RFC 9309 says the match must start with the first octet of the path. A rule cannot match something in the middle of a URL unless it begins with /*.
  • Every rule is a prefix match. Disallow: /blog matches any path that starts with /blog: /blog, /blog/, /blog-archive and /blogging-tips.html. You never need a trailing *. Google's specification says a trailing wildcard is ignored, so /fish* and /fish are equivalent.

The path being matched includes the query string. That is why Disallow: /*? blocks every URL that contains a question mark: ? is a literal character in robots.txt, not a regex quantifier.

What each pattern matches: worked examples

These examples follow the matching rules in Google's specification, which publishes a similar table:

Rule pathMatchesDoes not match
/fish/fish, /fish.html, /fish/salmon.html, /fishheads/Fish.asp, /catfish
/fish//fish/, /fish/?id=1, /fish/salmon.htm/fish, /fish.html, /animals/fish/
/*.php/index.php, /folder/file.php?x=1, /any.php.file.html/, /windows.PHP
/*.php$/filename.php, /folder/filename.php/filename.php?x=1, /filename.php/, /filename.php5
/fish*.php/fish.php, /fishheads/catfish.php?x=1/Fish.PHP
/*?/search?q=shoes, /shop/?sort=price/search, /shop/
/$//index.html, /?utm_source=x

Three things in that table catch people out:

  1. /fish/ does not match /fish. The trailing slash is part of the pattern. If both /fish and /fish/ exist, Disallow: /fish/ leaves the first one open.
  2. /*.php matches .php anywhere, including in the middle of /any.php.file.html. Add $ if you mean "ends in .php".
  3. $ stops matching at the end of the whole URL, query string included. /*.php$ does not match /filename.php?x=1, so it lets parameterized versions through.

How the longest match decides Allow vs Disallow

When more than one rule in a group matches a URL, RFC 9309 says the most specific match must be used, and the most specific match is the one with the most octets. If an Allow and a Disallow match equally, the Allow should win. Google's specification says it uses the rule with the longest path and, in a conflict, the least restrictive rule, including when wildcards are involved.

The length that counts is the length of the rule's pattern as written, wildcards included, not the length of the text it happens to match. Google gives this example:

User-agent: *
Allow: /page
Disallow: /*.htm

For /page.htm, both rules match. /*.htm has six characters and /page has five, so the Disallow wins and the URL is blocked. For /page.html the same logic applies, because /*.htm is a prefix match and matches .html too.

Order in the file does not matter to an RFC 9309 parser. Putting Allow lines first is a common habit from older parsers that took the first match, but it changes nothing for Google or for crawlers that follow the RFC.

A pattern that relies on longest match

This group blocks every URL with a query string except pagination:

User-agent: *
Disallow: /*?
Allow: /*?page=

/blog/?page=2 matches both. /*?page= is longer than /*?, so it is allowed. /blog/?sort=new matches only /*?, so it is blocked. Be aware that /blog/?sort=new&page=2 is still blocked, because /*?page= needs page= straight after the ?. To allow the parameter in any position you need a second rule:

Allow: /*&page=

Parameter rules get complicated quickly. If you are trying to control crawling of filtered or sorted URLs, read faceted navigation crawl traps and URL parameters and AI search before writing more patterns.

Case sensitivity and percent-encoding

Two details trip up sites with mixed-case URLs or non-ASCII paths.

Paths are case-sensitive. RFC 9309 says matching should be case-sensitive, and Google says it is. Disallow: /Private/ does not block /private/. If your server serves both forms, write a rule for each, or better, redirect one to the other so only one exists. The user-agent token is the opposite: it is matched case-insensitively, so gptbot selects the same group as GPTBot.

URLs are compared in percent-encoded form. Google says its crawlers compare rules against the percent-encoded form of the URL, and that a rule written in raw UTF-8 is converted. So these two lines are treated the same:

Disallow: /café/
Disallow: /caf%C3%A9/

If you need to match a literal * or $ in a URL, RFC 9309 says to percent-encode it. Its example is /path/file-with-a-%2A.html, which matches the URL /path/file-with-a-*.html.

Common wildcard mistakes and how to fix them

Using regular expression syntax

Disallow: /.*\.pdf
Disallow: /(cart|checkout)/

Neither works as intended. . and \ are literal characters, so the first rule only matches a path that literally starts with /.. Parentheses and | are literal too. The fix is one rule per path, using * only where you mean "anything":

Disallow: /*.pdf$
Disallow: /cart/
Disallow: /checkout/

Forgetting the leading slash

Disallow: *.pdf

The RFC grammar defines a path pattern as starting with /. Some parsers accept a leading *, but there is no reason to depend on that. Write /*.pdf.

Adding $ where you meant a prefix

Disallow: /admin$

This blocks only the exact URL /admin. /admin/, /admin/users and /admin?tab=1 stay open. Drop the $ to block the whole section.

Writing a broader rule than intended

Disallow: /*print

This blocks /print-version/, but also /blueprint/ and /printers/. When a pattern sits in the middle of a URL, make it as specific as you can, for example /*?print=1 or /*/print/.

Expecting every crawler to support wildcards

* and $ are part of RFC 9309, published in 2022, and documented by Google. Semrush, for example, lists wildcard support among the non-standard extensions SemrushBot handles, which shows that operators still describe it differently. Most modern crawlers follow the RFC, but older or smaller crawlers may treat * as a literal character. For crawlers you care about, a plain prefix rule is the most portable.

How to test a pattern before you publish it

Read the file you actually serve, not the copy in your repository:

bash
curl -s https://www.example.com/robots.txt

Then test real URLs against it with a parser that implements the RFC. Google's open-source robotstxt library takes a file, a user-agent token and a URL:

bash
bazel run robots_main -- ~/robots.txt GPTBot "https://www.example.com/blog/?page=2"

Build a short list of URLs that should be blocked and URLs that must stay open, and run every one through the parser for each crawler token you care about. Include edge cases: the URL with and without a trailing slash, with a query string, and in a different case. Our guide to testing robots.txt against each AI crawler has a repeatable routine, and robots.txt groups explained covers which group each crawler reads before any pattern is applied.

Frequently asked questions

Do I need a * at the end of a robots.txt rule?

No. Every rule is already a prefix match, and Google says a trailing wildcard is ignored. Disallow: /private and Disallow: /private* behave the same.

Does robots.txt support regular expressions?

No. Only * and $ have special meaning in paths. Characters such as ., ?, +, ( and | are matched literally.

Is robots.txt pattern matching case-sensitive?

Paths are. RFC 9309 says path matching should be case-sensitive and Google matches it that way, so /Blog/ and /blog/ need separate rules. User-agent tokens are matched case-insensitively.

Which rule wins when Allow and Disallow both match?

The rule with the longest pattern wins, counting wildcards as characters. If they are the same length, RFC 9309 says the Allow rule should be used, and Google uses the least restrictive rule.

How do I block all URLs with query strings?

Use Disallow: /*?. Then add longer Allow rules for any parameters you want crawled, such as Allow: /*?page=.

Check the rules you have published

Wildcard rules look tidy and block more, or less, than intended. Test them against real URLs before and after each change. To see how AI crawlers are treated on a specific page of yours, including what your robots.txt says to each of them, run the free check.

Filed under Access and robots.txt. Topics: robots-txt, url-parameters, google, ai-crawlers