Exit 134, a page on a diet, and a phone number in the Python docs
Three things that broke in the machine room: a crash after the work was done, a marketplace that answered with a wall, and a privacy check that found a phone number in the Python docs.
Exit 134 with all 20 rows in hand#
The data was fine. The exit was not.
A browser-based scraper collected all 20 rows it was asked for, then the process died with exit code 134 (SIGABRT) and the run was marked failed. The data was there; the run still counted as broken.
Rows came from a background thread that drove a headless browser. When the limit was reached, the main function returned while that thread was still closing the browser. The interpreter began shutting down with a browser half-closed, and the browser driver aborted the process.
Main waits (with a time limit) for the producer thread to finish its cleanup, and the browser is closed in the same thread that opened it (the sync browser API is tied to its thread).
producer = threading.Thread(target=produce_rows, args=(queue,), daemon=True)
producer.start()
for row in rows_from(queue):
push(row)
if pushed_count() >= max_items:
stop.set()
break
producer.join(timeout=20) # let it close its own browser before we exitA daemon thread is not "free". Whatever it holds open at exit is still yours.
The page that went on a diet (206 KB to 1.7 KB)#
Same addresses, less than one percent of the page, and a code writer busy fixing the wrong thing.
The machine was writing a new scraper for a big marketplace. Try 1 got a 206 KB answer and parsed zero rows (fair: the parser looked at the wrong key). Tries 2 and 3 got 1.7 KB for the same addresses, and the code writer kept "fixing" a parser that was not the problem.
The site had switched to an anti-bot slider page. Our shared wall detector did not know its markers, and we could not just add them: another marketplace legitimately carries the same markers on normal pages.
Compare a page with itself. If the same URL now answers with a tenth (or less) of what it served before and no rows come out, it is a wall: stop and park the idea instead of paying for another writing round. We do not try to get around walls.
WALL_SHRINK = 10
def shrunk(url, size_now, sizes_before):
before = sizes_before.get(url)
return before is not None and size_now * WALL_SHRINK <= beforeThe best anti-bot detector for one site is that site's own earlier answer.
A phone number in the Python docs#
Documentation pages do not have phone numbers, so what did the check see?
A new crawler for documentation sites was rejected twice: "an e-mail address or phone number reached the output". Documentation pages do not have phone numbers.
The privacy check matched +000000120, an example from Python's string formatting docs. It looked like "plus and 8 to 14 digits". Worse, the error showed the first 80 characters of the page, not the match, so the code writer could not see what was wrong and could not fix it.
International numbers never have 0 right after the plus (country codes start at 1). And the error now quotes the exact match with a bit of context around it.
PHONE = re.compile(r"(?<![\w.])\+[1-9]\d{7,13}(?![\w])")
PHONE.search("format(120, '+010') -> '+000000120'") # None now
PHONE.search("call +14155550123") # still caughtAn error message should carry its evidence. "Something matched somewhere" fixes nothing.
Yesterday, 2026-10-20. The counts are rough on purpose.
- More than a dozen new builds were promoted after passing their self-check.
- A handful of scrapers went public.
- Going public was refused a couple of dozen times because the day's slots were already used up.
- A failed run was noticed and a repair queued twice.
- The factory gave up on an idea after its check rounds a few times.
These entries come from the notes of the machine that builds and repairs our scrapers. Sites are not named and the code is simplified. What we build · how a job goes · rss