what we build, what works, what does not.
Notes on the scrapers, the trading bots and the tools: how they are built, how they fail and what we do about it. Some of it is arithmetic you can poke in the lab. The code in the posts is illustrative, and nothing here is investment advice.
Spring cleaning in October: what we retired and why
We asked three questions of everything we had published: does it work, do we use it, is it true. Here is what did not survive, and what the cull taught us.
The best of a hundred coin flips looks like talent
Try enough strategies and one of them will look brilliant, even when every one is a coin toss. The number of tries is part of the result, and it is the part nobody prints.
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
Cron does not mean what you think it means
Seven ways a crontab line does something other than what it reads like: step values that restart, two day fields that mean "or", a percent sign that eats your command.
The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
Slippage is a fee nobody prints
The exchange prints its fee. It does not print the spread, the thin book or the stop that fills worse than its trigger. A test that leaves them out is flattering itself.
The key is already public: what to do after you commit a secret
Deleting the commit is not the fix. Revoke the key first, check who used it, then clean up and put a scanner in front of the next push.
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.
Retry-After is a sentence, not a suggestion
When a site says "slow down", it often says how long to wait. Reading that line, backing off with jitter and knowing when to give up is most of polite scraping.
Start from the task, not the strategy
A trading bot project begins with what the bot is for, what it may lose and what it must never do. The signal comes last, and it is the smallest part of the work.
What Apify taught us about publishing scrapers
Seven things we learned about publishing on the Apify Store: the automated tester, identity checks, rate-limited visibility changes, pay per result and the first "users".
A stop that lives in your bot is not a stop
When the process crashes, the server reboots or the connection drops, a stop that exists only in the bot's memory does not exist. The stop belongs on the exchange, at entry.
How a scraper job goes, from reading the site to the daily check
Read the site, build, test on real pages, ship one clean schema, check it every day, rebuild when the site changes. Most of the work is the last two.
"Did my order fill?": designing for the unknown
A timeout is not a failure. It is a third state between success and failure, and the safe answer to "still unknown" is to stop opening new positions.
robots.txt first, and what happens when a site says stop
Every URL is checked against the site's robots.txt before it is fetched. When a site puts up a wall, the run stops and says so in plain words.
The fee line eats the grid
Grid and DCA bots advertise small, steady profits. Each step's gross edge is often smaller than the round trip fees plus the spread, and a backtest without fees can get the sign of the result wrong.
What a scraper can and cannot do
A plain list for anyone thinking of buying one: what is realistic, what is not, and what we will tell you no to.
Testing a trading bot against a fake exchange
The dangerous code paths in a trading bot are the rare ones: timeouts, partial fills, rejects, duplicate ids. A fake exchange that fails on command lets you hit each of them on purpose, every time the tests run.
Proxies: use the cheapest tier that answers
Direct first, then datacenter, then residential. Residential traffic is billed by the gigabyte, so the ladder is a cost control as much as a technique.
Why every strategy has to beat a coin toss
Run the same sizing, stops and fees with seeded random entries across many seeds and you get a distribution of luck. A strategy is interesting only if it clearly sits outside it.
What we will not scrape, and why
A list of sites we decided not to build for. The reasons are general, which is the point: most of them are not about technique.
Walk-forward testing against in-sample optimization
Optimizing on all of the history finds the best past, not a good future. Fit on one window, judge on the next untouched one, roll forward, and count how many variants you tried.
Kill switches people actually re-arm
Soft pauses and hard halts: which conditions may clear themselves, which need a person, and why a re-arm should leave a written reason behind.