Out of builds? Wait, do not give up · The tests that cried wolf · Thinking too long
"Not now" and "never" are different errors. Give them different handling. 2 more problems inside, with the cause and the fix of each.
Out of builds? Wait, do not give up#
There is a monthly cap on builds per scraper. Jobs that hit it burned their 3 attempts back to back and parked for good.
The error says when trying again makes sense (the 1st of next month), and the worker defers the job until then without spending an attempt. A small reserve of builds is kept for repairs, so a broken public scraper can still be fixed late in the month.
class BuildBudgetExceeded(Exception):
def __init__(self, retry_at):
self.retry_at = retry_at
try:
build(unit)
except DEFER_ERRORS as e:
job.defer(until=e.retry_at) # attempts untouched"Not now" and "never" are different errors. Give them different handling.
The tests that cried wolf#
The live failure log showed 8 "API is down" lines overnight. Nothing had been down.
The test suite has a deliberate outage drill. It wrote its fake failures into the real log, because the log path was decided when a module was imported, before any test could change it.
The log folder comes from an environment variable, and the test setup sets it to a temp folder before anything is imported.
# tests/conftest.py, loaded before any test module
os.environ.setdefault("LOGS_DIR", tempfile.mkdtemp(prefix="test-logs-"))If your tests can write to production logs, your production logs are partly fiction.
Thinking too long#
Steps that write new scraper code kept timing out at 400 seconds. Most of the time went on reasoning, not on the 10 to 20 KB of code that came out.
A hard cap on reasoning length. The call hung instead of finishing, so that went back out.
If a writing step times out, the retry asks for lower effort. A job parked for a passing reason (timeout, rate limit, overloaded service) gets requeued once by itself.
When a knob makes things hang, do not keep turning it. Find a softer knob.
Yesterday, 2026-10-24. The counts are rough on purpose.
- Dozens of new builds were promoted after passing their self-check.
- A handful of scrapers went public.
- Going public was refused dozens of times because the day's slots were already used up.
- A failed run was noticed and a repair queued once.
- A handful of times, the factory shelved an idea because the site served an anti-bot page or a stub.
These entries come from the notes of the machine that builds and repairs our scrapers. Sites are not named and the code is simplified. What we build · How a job goes · RSS