vøiddo
← blog  /  scrapers

Retry-After is a sentence, not a suggestion

2026-10-20  ·  2 min read  ·  scrapers

When a site says "slow down", it often says how long to wait. Reading that line, backing off with jitter and knowing when to give up is most of polite scraping.

A 429 Too Many Requests is a site telling you that you are too fast. Often it also tells you how long to wait, in a Retry-After header. That header is a full sentence from the site to your scraper, and most code throws it away and retries on a timer of its own invention.

What the header can hold#

The value is one of two things, and a scraper has to read both:

  • a number of seconds: Retry-After: 30
  • an HTTP date: Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

A 503 Service Unavailable can carry it too.

Illustrative: read the header, back off with jitterpython
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_after_seconds(value: str | None) -> float | None:
    """Retry-After is either a number of seconds or an HTTP date."""
    if not value:
        return None
    value = value.strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

def wait_time(attempt: int, header: str | None, cap: float = 120.0) -> float:
    told = retry_after_seconds(header)
    if told is not None:
        return min(told, cap)                                   # the site's own number beats our guess
    return min(cap, 2 ** attempt) * random.uniform(0.5, 1.0)    # exponential backoff with jitter

A date in the past reads as zero seconds, garbage reads as nothing, and a huge number is capped so that one odd header cannot park a run for a day. The jitter matters when many workers wait at once: without it they all come back in the same second and the site gets the same spike again.

A hiccup becomes a wall#

The point of waiting is that the problem is temporary. So set the rule before the run, not during it:

  1. Wait as long as the site says, or back off if it says nothing.
  2. Retry a small, fixed number of times. Three is a fine number.
  3. Come back slower than before. If you were asked to slow down, going back at the same rate is not an answer.
  4. If the same status is still there after you waited as told, it is no longer a hiccup. It is a wall with good manners. Stop and say so in the run status.

Things that are never retried#

  • A page that asks you to prove you are human.
  • An "access denied" that has nothing to do with the URL you asked for.
  • A redirect to a login page.
  • A robots.txt rule that says no. The request is not even sent.

Those are answers, not errors. Robots.txt first covers the wall side in more detail, and the proxy ladder covers why we do not answer a wall with a bigger hammer.

Eight sample responses, and your call on each, are in the lab. Some are judgement calls. That is the point.

keep reading