The price is a string: reading money off a web page
Shops print the same number a dozen ways. A careful parser reads what is clear, leaves the rest empty and never fills a gap with a plausible guess.
On a product page the price is text. It is the last thing you would call a number: it has a currency symbol in front or behind, a space or a dot or a comma or an apostrophe that means "thousands" in one country and "decimals" in the next, and sometimes two prices at once.
One price, seven ways to print it#
| As printed | What it means |
|---|---|
$1,299.00 | 1299.00, probably US dollars |
1.299,00 € | 1299.00 euros |
1 299,00 ₽ | 1299.00 roubles, the space is thousands |
CHF 1'299.50 | 1299.50 Swiss francs, the apostrophe is thousands |
¥1,200 | 1200, but yen or yuan: the symbol is shared |
CA$19.99 | 19.99, and not US dollars |
1.299 | 1.299 or 1299, and the text does not say |
Rules that hold up#
- If both a dot and a comma appear, the last one is the decimal point.
1.299,00and1,299.00both come out as 1299.00. - A lone separator followed by exactly three digits is ambiguous. A comma there is usually thousands in English-language shops. A dot there could be either. When the text cannot tell you, leave the field empty.
- Read the currency from a code or from a symbol that means one thing. The pound and euro signs do.
¥,krand a bare$are shared by several currencies, so a careful parser either takes the currency from the page's own data or leaves it empty. - Prefer the page's own structured data. Many shops carry a machine-readable price next to the printed one, in a
priceandpriceCurrencypair. When it is there, use it, and keep the printed string as the fallback. - Two prices on a line is a question, not a parse. "From £9.99 (was £14.99)" has a current price and a former one. Decide which your schema wants and store the other under its own name, or drop it.
- Keep money out of floats. In Python,
0.1 + 0.2is0.30000000000000004. Store prices as decimal strings, or as integer minor units, and convert at the edge.
The one rule behind the rest#
A price that cannot be read is empty. It is not 0, and it is not the neighbouring product's price, and it is not "probably 1299". Downstream, an empty field is visible and can be counted. A guess looks like data and gets traded on.
That is also why a daily check on a scraper should watch the share of empty prices as well as the share of failures. If a shop changes its format, the first sign is empty prices going from a few percent of the catalogue to most of it, long before anything crashes.
The lab has a parser that follows these rules. Paste in the strangest price you have seen on a shop page and see whether it reads it, or has the grace to say it cannot.
written
7 min readRemote job APIs: five free feeds, and the catch in each
Remote OK, We Work Remotely, Jobicy, Working Nomads and Arbeitnow publish their jobs without a key. What each feed holds, how far back it goes, the terms, and sixty lines of Python that put them in one list.
5 min readFacebook Ad Library API: which ads you get, which you don't
Meta's API returns political ads from everywhere and every kind of ad shown in the UK or the EU. An ad shown only in the US for running shoes is not in it. What each way into the Ad Library gives you, and how to keep the ads the API leaves out.
6 min readFree exchange rate API with no key: ECB rates in Python
The European Central Bank publishes its daily euro rates as plain files: no key, no sign-up. What is in them, the three things that trip people up (weekends, holidays, retired currencies) and about forty lines of Python that handle all three.