vøiddo
← blog  /  scrapers

Proxies: use the cheapest tier that answers

2026-10-20  ·  2 min read  ·  scrapers

Direct first, then datacenter, then residential. Residential traffic is billed by the gigabyte, so the ladder is a cost control as much as a technique.

A proxy is not a switch you turn on. It is a bill. Residential proxies in particular are charged by traffic, and a browser run that loads every image and font on every page can spend more on bandwidth than the data is worth.

The ladder#

We try the cheapest tier first and climb only when the page does not come back.

Cheapest tier that returns a real pagepython
import httpx

LADDER = ("direct", "datacenter", "residential")      # proxies["direct"] is None

async def fetch_with_ladder(url, proxies):
    for tier in LADDER:
        async with httpx.AsyncClient(proxy=proxies.get(tier), timeout=30, headers={"User-Agent": UA}) as client:
            resp = await client.get(url)
        if resp.status_code == 200 and not looks_blocked(resp.text):
            return tier, resp
        if resp.status_code == 404:
            return tier, None                     # the page is gone, a better proxy will not bring it back
    raise Blocked(f"{url}: no tier got a real page")

Two details carry the weight:

  • A 404 ends the climb. A better proxy does not bring back a deleted page. Without that line, every dead link in a catalogue would be tried on the most expensive tier.
  • "No tier got a real page" raises Blocked. That is the same exception the wall check uses (see robots.txt first). The run stops and reports; it does not keep trying.

Cheaper than a better proxy: not downloading things#

For browser runs, the biggest saving is blocking images, media and fonts. The data is in the document. The pictures are the bandwidth. Blocking them is a few lines of configuration, and it cuts traffic on residential tiers by more than any proxy negotiation will.

What the platform adds#

Residential proxy is a paid-plan feature on Apify and is billed by traffic. That is one more reason the default input of every actor runs direct: a first run with no proxy should work, and a buyer should turn proxies on on purpose.

keep reading