What we will not scrape, and why
A list of sites we decided not to build for. The reasons are general, which is the point: most of them are not about technique.
The list of things we could scrape is long. The list of things we will not is shorter, and more useful to a buyer, because it tells them what we are going to say no to before they ask.
The short list#
| Kind of site | Why we skip it |
|---|---|
| Networks of people profiles | The output is personal data about people who did not agree to it. |
| Contact and e-mail finders | Personal data again, and the typical use is spam. |
| Classifieds of private sellers | A private seller is a person. Their name, phone and photos are personal data. |
| Sites whose robots.txt closes the data | They said no. We read it and obey it. |
| Sites behind behavioural bot walls | The wall is the site's answer. We stop instead of fighting it. |
| Google-owned properties | Legal risk for page scrapers. |
What the rules look like inside an actor#
- No personal data. There are no contact extractors, no lead lists, no people profiles. Review and job outputs carry no person fields, and e-mail addresses and phone numbers that appear in free text are removed.
- No logins and nothing behind a paywall.
- Polite pacing. Delays between requests, a cap per run, and one run never hammers one host.
- Honest output. A field the page does not show stays empty. Nothing is guessed or filled in.
What is left#
Plenty. Online stores and marketplaces (products, prices, stock, ratings), property portals, news and articles, public job boards on the common hiring platforms, public registries and open government data, app and extension stores, package registries. Document and image utilities, too: HTML to PDF or image, QR codes, image conversion, PDF text.
Everything on that list is public, structured by the site itself, and not about a private person. That is the shape of a job we take.
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.