vøiddo
← blog  /  scrapers

robots.txt first, and what happens when a site says stop

2026-10-20  ·  2 min read  ·  scrapers

Every URL is checked against the site's robots.txt before it is fetched. When a site puts up a wall, the run stops and says so in plain words.

Two rules cover most of how our scrapers behave toward the sites they read. The site's robots.txt is obeyed for every URL, and a wall is a reason to stop, not a puzzle.

Check first, fetch second#

The check is a few lines of standard library. It runs before every request, and the answer is cached per site so it does not turn into extra traffic.

robots.txt check for every URLpython
from functools import lru_cache
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

UA = "ExampleBot/1.0 (+https://example.com/bot)"

@lru_cache(maxsize=256)
def _rules(origin: str) -> RobotFileParser:
    rp = RobotFileParser(origin + "/robots.txt")
    rp.read()
    return rp

def allowed(url: str) -> bool:
    parts = urlsplit(url)
    return _rules(f"{parts.scheme}://{parts.netloc}").can_fetch(UA, url)

A page that is closed is skipped. Not fetched "just once to see", not fetched at a low rate. Skipped.

one caveat

The built-in reader used here is short, and it is simple: it takes the first matching rule and does not understand the wildcard characters * and $. For files made of plain prefix rules that is fine. For files with wildcards or an Allow carved out of a broad Disallow it will say yes where the standard says no. robots.txt readers disagree lists where, and the lab lets you try your own file.

A wall stops the run#

Sometimes a site answers with a captcha, a "verify you are human" page or an "access denied" that has nothing to do with the page you asked for. A scraper that treats that as a retry condition ends up hammering a site that has just asked it to go away.

Ours treat it as an answer. The run raises, stops, and puts the reason in the run status.

A wall is an answer, not an error to retrypython
WALL_MARKERS = ("captcha", "verify you are human", "access denied", "unusual traffic")

class Blocked(Exception):
    """The site asked us to stop. We stop and report it."""

def looks_blocked(html: str) -> bool:
    head = html[:20_000].lower()
    return any(marker in head for marker in WALL_MARKERS)

What we do not do#

  • No captcha solving.
  • No stealth plugins and no fingerprint tricks.
  • No logins, and nothing behind a paywall.
  • Delays between requests, and a cap per run, so one run never hammers one host.
why this is not just ethics

It is also cheaper. A scraper that fights a wall burns proxy traffic, gets the IP range flagged and still returns nothing. A scraper that stops returns a clear failure the same day, and the buyer can decide what to do with the job.

keep reading