The retry that always came back an hour late · "database is locked" · Repairing a wall every morning
"Retry later" is a scheduling decision. If later is per job, order is decided by accident. 2 more problems inside, with the cause and the fix of each.
The retry that always came back an hour late#
A listing that was approved to go public missed its publish window. The platform allows only a few "make public" changes per day, and that day's slots went to other jobs.
A job refused for "no slot left" retried one hour after its own refusal. So whoever got refused first also got the next slot, no matter how important it was. A hand-set priority did not help either: every deferral recomputed the priority from scratch and threw the manual value away.
Every waiting job now retries at the moment the next slot opens, so they all meet there and the queue picks by priority. A pinned priority survives deferrals.
# before: each job comes back on its own clock
retry_at = refused_at + max(3600, retry_after)
# after: everyone meets at the slot, priority decides
retry_at = next_slot_opens_at()
job.priority = job.pinned_priority or job.priority"Retry later" is a scheduling decision. If later is per job, order is decided by accident.
"database is locked"#
Now and then a job failed with sqlite3.OperationalError: database is locked.
Two jobs opened a write transaction and then went to the network inside it: one fetched store pages, one pinged a search engine. While they waited on the internet, everyone else waited on them.
Do the slow part first, then write everything in one short transaction.
rows = [fetch_rank(q) for q in queries] # slow, no lock held
with db: # short write
db.executemany("INSERT INTO ranks VALUES (?, ?, ?)", rows)A transaction is a lock with a timer you do not control. Never put a network call inside one.
Repairing a wall every morning#
One scraper sits behind an anti-bot wall and is deliberately paused. Every day it still got a "repair", and each repair spent a test run and a build.
The repair job did not look at the "this site walls bots" note, and the watcher kept reacting to the same old failed runs.
A paused-for-a-wall scraper ends its repair with one line in the log, no run, no build. The watcher only reacts to failures newer than the last repair.
if unit.hold_reason == "bot_walled":
log("held: site walls bots, nothing to repair")
return
if failed_run.finished_at <= last_repair_at:
return # already handledA repair loop needs to know which problems are not fixable by repairing.
Yesterday, 2026-10-22. The counts are rough on purpose.
- Dozens of new builds were promoted after passing their self-check.
- A handful of scrapers went public.
- Going public was refused a couple of dozen times because the day's slots were already used up.
- A failed run was noticed and a repair queued once.
- The watcher spotted a failed run once.
These entries come from the notes of the machine that builds and repairs our scrapers. Sites are not named and the code is simplified. What we build · How a job goes · RSS