vøiddo
← blog  /  scrapers

robots.txt readers disagree, and the built-in one is the odd one out

2026-10-20  ·  3 min read  ·  scrapers

Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.

A robots.txt file looks like a plain list of do and do not. It is not as plain as it looks. Two readers can give opposite answers for the same file and the same URL, and one of them ships inside Python.

The standard is RFC 9309, published in 2022. The built-in reader is urllib.robotparser, which predates it. We put them side by side, on purpose, because a scraper that says "yes" to a URL the site said "no" to is the one mistake that cannot be waved away.

What the standard says#

  1. Find the group for your agent by its product name, ignoring case. No group, use the * group. No * group either, everything is allowed.
  2. Inside the group a rule matches when its pattern matches the start of the URL path, query string included. * stands for any run of characters. $ pins the end.
  3. If several rules match, the longest pattern wins. A tie between Allow and Disallow goes to Allow.
  4. An empty rule, a bare Disallow:, blocks nothing.

What the built-in reader does#

Read from its source and checked against the real module (Python 3.10.12):

  • First match wins, in file order, not longest.
  • * and $ in a rule are plain characters. A rule like /*.pdf$ matches no URL at all.
  • A bare Disallow: becomes "allow everything", and because it is first-match, a bare Disallow: ahead of other rules cancels them.
  • A group name matches if it appears inside your agent name. A group called bot catches ExampleBot.
  • A blank line ends a group. Rules that follow it, before the next User-agent: line, are silently dropped.
  • An empty ? at the end of a rule is dropped, so /search? becomes /search.

Where that splits the answer#

Each row was run through both readers.

robots.txt ruleURLStandardBuilt-in
Disallow: /private/ then Allow: /private/public.html/private/public.htmlallowedblocked
Disallow: /*.pdf$/files/report.pdfblockedallowed
Disallow: /*?sessionid=/cart?sessionid=abcblockedallowed
Disallow: then Disallow: /secret//secret/plan.htmlblockedallowed
Disallow: /page then Allow: /page/pageallowedblocked
User-agent: bot then Disallow: /agent ExampleBotallowedblocked
a rule after a blank line inside a groupthe rule's pathblockedallowed

Three of those lean safe: the built-in reader blocks more than it should. Four lean the other way, and those are the ones that matter. Wildcard rules and Allow carve-outs after a broad Disallow are ordinary things to find in a robots.txt file.

how we tested

We built 3,000 random files from a small grammar of groups, Allow and Disallow rules with and without wildcards, bare rules, blank lines, comments, Sitemap and Crawl-delay lines, and asked four URL questions of each: 12,000 pairs. We compared a standalone implementation of each reading, the one running in the lab, against the real thing: Python 3.10.12's built-in reader and Protego 0.7.0. No differences for either. The two real readers disagreed with each other on 1,617 of the 12,000 pairs. That is a statement about awkward files built to provoke them, and says nothing about how often real sites trip it. Newer Python versions may behave differently.

What to do about it#

  • If you use the built-in reader, know what it will not catch. For plain files made of simple prefix rules it is fine.
  • If sites you read use wildcards or carve-outs, use a reader that follows RFC 9309. Protego is one, and the one Scrapy uses by default. Google has also published its own parser.
  • Whatever you use, keep a small file of cases, like the table above, and run it in your tests. The week a reader changes its mind, you want to hear about it from the tests.

The snippet in robots.txt first uses the built-in reader because it is short. Now you know what it leaves out. Paste your own file into the lab and see both answers.

keep reading