The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
On a product page the price is text. It is the last thing you would call a number: it has a currency symbol in front or behind, a space or a dot or a comma or an apostrophe that means "thousands" in one country and "decimals" in the next, and sometimes two prices at once.
The same number, seven ways#
| As printed | What it means |
|---|---|
$1,299.00 | 1299.00, probably US dollars |
1.299,00 € | 1299.00 euros |
1 299,00 ₽ | 1299.00 roubles, the space is thousands |
CHF 1'299.50 | 1299.50 Swiss francs, the apostrophe is thousands |
¥1,200 | 1200, but yen or yuan: the symbol is shared |
CA$19.99 | 19.99, and not US dollars |
1.299 | 1.299 or 1299, and the text does not say |
Rules that hold up#
- If both a dot and a comma appear, the last one is the decimal point.
1.299,00and1,299.00both come out as 1299.00. - A lone separator followed by exactly three digits is ambiguous. A comma there is usually thousands in English-language shops. A dot there could be either. When the text cannot tell you, leave the field empty.
- Read the currency from a code or from a symbol that means one thing. The pound and euro signs do.
¥,krand a bare$are shared by several currencies, so a careful parser either takes the currency from the page's own data or leaves it empty. - Prefer the page's own structured data. Many shops carry a machine-readable price next to the printed one, in a
priceandpriceCurrencypair. When it is there, use it, and keep the printed string as the fallback. - Two prices on a line is a question, not a parse. "From £9.99 (was £14.99)" has a current price and a former one. Decide which your schema wants and store the other under its own name, or drop it.
- Keep money out of floats. In Python,
0.1 + 0.2is0.30000000000000004. Store prices as decimal strings, or as integer minor units, and convert at the edge.
The one rule behind the rest#
A price that cannot be read is empty. It is not 0, and it is not the neighbouring product's price, and it is not "probably 1299". Downstream, an empty field is visible and can be counted. A guess looks like data and gets traded on.
That is also why a daily check on a scraper should watch the share of empty prices as well as the share of failures. If a shop changes its format, the first sign is empty prices going from a few percent of the catalogue to most of it, long before anything crashes.
The lab has a parser that follows these rules. Paste in the strangest price you have seen on a shop page and see whether it reads it, or has the grace to say it cannot.
robots.txt readers disagree, and the built-in one is the odd one out
Python's built-in reader takes the first matching rule and ignores wildcards. The standard takes the longest match. We ran both on 12,000 random files and listed where they split.
A timestamp without a zone is a guess
Scraped dates arrive as "2026-03-08 02:30" with nothing else. Some of those times never happened, some happened twice, and the gap between two cities moves during the year.
Retry-After is a sentence, not a suggestion
When a site says "slow down", it often says how long to wait. Reading that line, backing off with jitter and knowing when to give up is most of polite scraping.