What a scraper can and cannot do
A plain list for anyone thinking of buying one: what is realistic, what is not, and what we will tell you no to.
Most disappointment with scrapers comes from a mismatch between what was imagined and what is possible. This is the list we would want to read before buying one.
What a scraper can do#
- Read public pages on a schedule and return them as clean, typed rows: products, prices, stock, ratings, listings, headlines, job posts, filings, notices, package versions.
- Return one documented schema, so the next run is comparable with the last one.
- Normalise what sites print differently: currencies, number formats, dates, units.
- Be adapted to your fields, regions or schedule.
- Run on a platform (Apify Store, pay per result) or behind an HTTP API, and be called from a script or an agent.
- Be rebuilt when a site changes its layout.
What it cannot do#
- Read what the page does not show. If a price is hidden until login, the field is empty. We do not guess it, and we do not infer it from the neighbours.
- Promise that a site never changes. Sites do. What we can promise is a daily check and a rebuild.
- Get past a wall. If a site shows a captcha or a block page, the run stops. We do not solve captchas or hide the scraper.
- Collect people. No contact lists, no profiles, no e-mail or phone harvesting.
- Replace permission. A site that closes its data in robots.txt stays closed, however technically easy the page is.
- Be free to run at any scale. Browser runs and residential proxies cost real money per gigabyte. That is what the pay per result price covers.
What we say no to#
See what we will not scrape. The short version: anything about private people, anything a site has closed, anything behind a bot wall.
If you are not sure#
Write to us with the site and the fields you need. The address is on the contact page.
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.