How a scraper job goes, from reading the site to the daily check

A scraper is a promise that a page you do not control will keep looking the way it looked yesterday. It will not. So the job is not "write a scraper". The job is write one, then keep it true.

The loop#

1. Read the site and its rules. What it shows, what it hides, what its robots.txt allows, whether the data is public at all. If the answer to the last one is no, the job ends here.
2. Build. Plain HTTP when the page is plain, a real browser when the page needs one.
3. Test on real pages, not on the three that were open while writing the code. Pages with missing prices, sold-out items, odd currencies, empty ratings.
4. Ship one documented output schema. A buyer should be able to read the field list once and trust it.
5. Check it every day with a small default input. A site change shows up as an empty field or a failed run, not as an angry email.
6. Rebuild when the site changes, and adapt the output to a buyer's fields, regions or schedule when they ask.

What a listing scraper asks for#

The input is the first contract. This is the shape we use for a store listing scraper: a few start pages, a cap, a country, one switch, and a proxy block that defaults to off.

{
  "title": "Store listing input",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {"title": "Category or search pages", "type": "array", "editor": "requestListSources",
                  "prefill": [{"url": "https://shop.example.com/c/headphones"}]},
    "maxItems": {"title": "Max results", "type": "integer", "minimum": 1, "default": 100, "prefill": 20},
    "country": {"title": "Country", "type": "string", "enum": ["US", "GB", "DE", "FR"], "default": "US"},
    "includeOutOfStock": {"title": "Include out of stock", "type": "boolean", "default": false},
    "proxy": {"title": "Proxy", "type": "object", "editor": "proxy", "default": {"useApifyProxy": false}}
  },
  "required": ["startUrls"]
}

The prefill values matter more than they look. They are what runs when anybody, including the platform's own tester, clicks start without typing anything. They have to be small, safe and boring.

One record, whatever the shop prints#

Shops print prices as $1,299.00, 1.299,00 € and 12,50. The output should not care. Every field is either parsed or empty, and an empty field stays empty.

import re
from dataclasses import dataclass
from decimal import Decimal
from typing import Optional

@dataclass
class Product:
    url: str
    title: str
    price: Optional[Decimal]
    currency: Optional[str]
    in_stock: Optional[bool]
    rating: Optional[float]
    reviews_count: Optional[int]

SYMBOLS = {"$": "USD", "£": "GBP", "€": "EUR"}

def parse_price(text: str) -> tuple[Optional[Decimal], Optional[str]]:
    m = re.search(r"([$£€])\s*([\d.,]+)", text or "")
    if not m:
        return None, None                         # not shown on the page: stays empty, never guessed
    raw = m.group(2).rstrip(".,")
    if re.fullmatch(r"\d{1,3}(,\d{3})+(\.\d+)?", raw):   # 1,299 or 1,299.00
        raw = raw.replace(",", "")
    elif "," in raw:                                     # 1.299,00 or 12,50
        raw = raw.replace(".", "").replace(",", ".")
    return Decimal(raw), SYMBOLS[m.group(1)]

The comment on the first return is the whole policy. A scraper that fills a gap with a plausible number produces data that is wrong in a way nobody can see.

The boring part is the product#

Anybody can write the first run. What a buyer pays for is that the same input still returns the same fields next month. That means a daily check, a short list of fields that must be non-empty, and somebody who looks when the check goes red.