we build scrapers and keep them working.
We build data extractors, run them on Apify, and rebuild them when the sites change. We do not sell lists of people, and we do not fight sites that do not want to be read.
the kinds of job
- Online stores and marketplaces: products, prices, stock, ratings, category and search listings.
- Property portals: listings with price, size, rooms and location.
- News and articles: headline, date, body text, canonical link.
- Public job boards on the common hiring platforms: title, team, location, posting date, description.
- Public registries and open government data: filings, recalls, grants, advisories, notices.
- App and extension stores, package registries: metadata, versions, public ratings.
- Document and image utilities: HTML to PDF or image, QR codes, image conversion, PDF text.
from reading the site to the daily check
- Read the site and its rules. What it shows, what its robots.txt allows, whether the data is public at all.
- Build. Plain HTTP when the page is plain, a real browser when the page needs one.
- Test on real pages, including the ugly ones: missing prices, sold out items, odd currencies.
- Ship one documented output schema.
- Check it every day with a small default input.
- Rebuild when the site changes, and adapt it to your fields, regions or schedule.
The long version is on the blog: how a scraper job goes.
what the actors follow
| Rule | What it means |
|---|---|
| robots.txt first | It is read and obeyed for every URL. A closed page is skipped, never fetched "just once". |
| A wall is an answer | No captcha solving, no stealth plugins, no fingerprint tricks. When a site shows a wall the run stops and says so in plain words. |
| No personal data | No contact extractors, no lead lists, no people profiles. Review and job outputs carry no person fields, and e-mails and phone numbers in free text are removed. |
| No logins | Nothing behind a login and nothing behind a paywall. |
| Polite pacing | Delays between requests and a cap per run. One run never hammers one host. |
| Cheapest proxy that works | Direct first, then datacenter, then residential. Browser runs block images, media and fonts. |
| One documented schema | A field the page does not show stays empty. Nothing is guessed or filled in. |
| Checked every day | Every actor has a small default input that must work. A site change means a rebuild, not a shrug. |
| Pay per result | A buyer pays for rows delivered, not for time spent. |
a few pieces of how they are built
These are written for this page. They show the shape of the real thing without exposing it.
the input is the first contract
A listing scraper asks for a few start pages, a cap, a country and one switch. The proxy block defaults to off, and the prefilled values are what runs when anybody clicks start without typing.
{
"title": "Store listing input",
"type": "object",
"schemaVersion": 1,
"properties": {
"startUrls": {"title": "Category or search pages", "type": "array", "editor": "requestListSources",
"prefill": [{"url": "https://shop.example.com/c/headphones"}]},
"maxItems": {"title": "Max results", "type": "integer", "minimum": 1, "default": 100, "prefill": 20},
"country": {"title": "Country", "type": "string", "enum": ["US", "GB", "DE", "FR"], "default": "US"},
"includeOutOfStock": {"title": "Include out of stock", "type": "boolean", "default": false},
"proxy": {"title": "Proxy", "type": "object", "editor": "proxy", "default": {"useApifyProxy": false}}
},
"required": ["startUrls"]
}robots.txt is checked before every request
from functools import lru_cache
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
UA = "ExampleBot/1.0 (+https://example.com/bot)"
@lru_cache(maxsize=256)
def _rules(origin: str) -> RobotFileParser:
rp = RobotFileParser(origin + "/robots.txt")
rp.read()
return rp
def allowed(url: str) -> bool:
parts = urlsplit(url)
return _rules(f"{parts.scheme}://{parts.netloc}").can_fetch(UA, url)a wall stops the run
WALL_MARKERS = ("captcha", "verify you are human", "access denied", "unusual traffic")
class Blocked(Exception):
"""The site asked us to stop. We stop and report it."""
def looks_blocked(html: str) -> bool:
head = html[:20_000].lower()
return any(marker in head for marker in WALL_MARKERS)the cheapest proxy tier that answers
import httpx
LADDER = ("direct", "datacenter", "residential") # proxies["direct"] is None
async def fetch_with_ladder(url, proxies):
for tier in LADDER:
async with httpx.AsyncClient(proxy=proxies.get(tier), timeout=30, headers={"User-Agent": UA}) as client:
resp = await client.get(url)
if resp.status_code == 200 and not looks_blocked(resp.text):
return tier, resp
if resp.status_code == 404:
return tier, None # the page is gone, a better proxy will not bring it back
raise Blocked(f"{url}: no tier got a real page")one record, whatever the shop prints
import re
from dataclasses import dataclass
from decimal import Decimal
from typing import Optional
@dataclass
class Product:
url: str
title: str
price: Optional[Decimal]
currency: Optional[str]
in_stock: Optional[bool]
rating: Optional[float]
reviews_count: Optional[int]
SYMBOLS = {"$": "USD", "£": "GBP", "€": "EUR"}
def parse_price(text: str) -> tuple[Optional[Decimal], Optional[str]]:
m = re.search(r"([$£€])\s*([\d.,]+)", text or "")
if not m:
return None, None # not shown on the page: stays empty, never guessed
raw = m.group(2).rstrip(".,")
if re.fullmatch(r"\d{1,3}(,\d{3})+(\.\d+)?", raw): # 1,299 or 1,299.00
raw = raw.replace(",", "")
elif "," in raw: # 1.299,00 or 12,50
raw = raw.replace(".", "").replace(",", ".")
return Decimal(raw), SYMBOLS[m.group(1)]pay per result, stop cleanly at the limit
import time
from dataclasses import asdict
from apify import Actor
async def main():
async with Actor:
inp = await Actor.get_input() or {}
max_items, pushed = int(inp.get("maxItems", 100)), 0
deadline = time.time() + 55 * 60 # finish with what we have instead of being cut off
async for product in crawl(inp["startUrls"]): # yields Product records, robots-checked
if pushed >= max_items or time.time() > deadline:
break
res = await Actor.push_data(asdict(product)) # one row = one charged result
pushed += 1
if getattr(res, "event_charge_limit_reached", False):
break # the buyer's spending limit: stop, do not fail
await Actor.set_status_message(f"Done: {pushed} results")how a job goes robots.txt first the proxy ladder what apify taught us
the apify store, and the utilities next door
Our scrapers are on the Apify Store, billed per result. Five of the current listings:
More follow as they pass their tests.
Separately from the scrapers, we run small file and data utilities: convert, validate, look things up. They work in the browser at utils.voiddo.com, over HTTP and as an MCP endpoint for agents at data.voiddo.com, and from code through the npm client @v0idd0/mint-utils. These are not the scrapers.
what we say no to
We do not build scrapers for networks of people profiles, contact or e-mail finders, classifieds of private sellers, sites whose robots.txt closes the data, sites behind behavioural bot walls, or Google-owned properties. The reasons are mostly about people, not technique: what we will not scrape, and why.
If you are not sure whether your job fits, read what a scraper can and cannot do, or write to support@voiddo.com with the site and the fields you need.
about scrapers
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.
Retry-After is a sentence, not a suggestion
When a site says "slow down", it often says how long to wait. Reading that line, backing off with jitter and knowing when to give up is most of polite scraping.
What Apify taught us about publishing scrapers
Seven things we learned about publishing on the Apify Store: the automated tester, identity checks, rate-limited visibility changes, pay per result and the first "users".
How a scraper job goes, from reading the site to the daily check
Read the site, build, test on real pages, ship one clean schema, check it every day, rebuild when the site changes. Most of the work is the last two.
robots.txt first, and what happens when a site says stop
Every URL is checked against the site's robots.txt before it is fetched. When a site puts up a wall, the run stops and says so in plain words.
What a scraper can and cannot do
A plain list for anyone thinking of buying one: what is realistic, what is not, and what we will tell you no to.
Proxies: use the cheapest tier that answers
Direct first, then datacenter, then residential. Residential traffic is billed by the gigabyte, so the ladder is a cost control as much as a technique.
What we will not scrape, and why
A list of sites we decided not to build for. The reasons are general, which is the point: most of them are not about technique.