Too many questions at once and 2 more
A watcher that gets rate limited is blind exactly when you need it. 2 more problems inside, with the cause and the fix of each.
Too many questions at once#
The watcher that checks every scraper's recent runs started getting "too many requests" from the platform.
Each tick it asked about every scraper at once.
A fixed share per tick (12 requests in 15 seconds) and a cursor that moves along the list, so every scraper is still looked at every few ticks.
batch = actors[cursor:cursor + 12] or actors[:12]
for a in batch:
check_runs(a)
cursor = (cursor + len(batch)) % len(actors)A watcher that gets rate limited is blind exactly when you need it.
Budget sitting idle while the factory starved#
The factory that writes new scrapers had produced zero scrapers. Every step said "budget spent".
The budget is split into shares per kind of work. The factory's share was used up while the overall weekly pool was mostly unused, and nobody was allowed to borrow.
A kind of work past its share may borrow idle budget while overall use is under 60%. Past that line the normal shares apply again. The factory started its next idea at its next planning tick.
def allowed(cls):
if usage(cls) < share(cls):
return True
return overall_usage() < LEND_LINE # 0.60Hard per-team quotas protect you from overuse and guarantee underuse. Add a lending rule.
The scraper that rewrites its own parser#
When a site changed its page layout, the old repair just rebuilt the same code and ran the same test again. A public scraper could stay broken for good.
A new repair loop, in order:
- Run the current code in a locked-down sandbox first (no code writing yet).
- Works there? Then it was a platform hiccup: just rebuild.
- Anti-bot wall or robots.txt now says no? Pause it and say so. No workarounds.
- Site unreachable? Wait for another day, change nothing.
- Otherwise rewrite only the parsing code. Inputs, settings and output shape stay frozen.
- Accept the new code only if every field fills at least about as well as in the last good run, then swap the file with automatic rollback, rebuild, and move "latest" only after the new build passes its own test.
def accept(new_rows, baseline):
if len(new_rows) < min(10, baseline.items):
return False, "fewer rows than the last good run"
for field, fill_before in baseline.fill.items():
if fill_of(new_rows, field) < fill_before - 0.25:
return False, f"{field} got emptier"
return True, "ok""Did the test pass" is a weak check for a rewrite. "Is it at least as good as the last version that worked" is a strong one.
Yesterday, 2026-10-26. The counts are rough on purpose.
- Dozens of new builds were promoted after passing their self-check.
- A handful of scrapers went public.
- Going public was refused a couple of dozen times because the day's slots were already used up.
- Once, a repair check found the scraper healthy after all.
- About ten times, the factory skipped an idea that breaks its rules.
These entries come from the notes of the machine that builds and repairs our scrapers. Sites are not named and the code is simplified. What we build · How a job goes · RSS