What Apify taught us about publishing scrapers

Writing the scraper is the easy half. This is what we learned about putting one in front of people on the Apify Store.

1. The first user is a robot#

Apify runs an automated test of every public actor with its default input. It has to finish fast with at least one row, or the actor gets marked for maintenance and, later, deprecated. So the default input is small, safe and boring on purpose.

A related warning: the first "users" of a new public actor are mostly that tester. Early run counts are not demand, and we do not quote them.

2. Identity and payout come before price#

Publishing to the Store needs a verified developer identity. Charging needs payout details first. Decide the price before the actor goes public, not after.

3. Visibility changes are rate limited#

Going from private to public is limited per day. It is not in the docs; the API tells you when you hit it. Release in batches, and keep a way to stop an actor at once if a site asks or it misbehaves: a tiny build that collects nothing can replace the real one immediately.

4. Residential proxy is a paid, metered feature#

Billed by traffic. Block images, media and fonts in browser runs and use the ladder from the proxy post.

5. Pay per result means one row is one charge#

Every pushed row is a result. The platform caps it at the buyer's spending limit and tells the run when the limit is reached, so the run can stop cleanly instead of failing half way. This is the loop we use.

import time
from dataclasses import asdict
from apify import Actor

async def main():
    async with Actor:
        inp = await Actor.get_input() or {}
        max_items, pushed = int(inp.get("maxItems", 100)), 0
        deadline = time.time() + 55 * 60               # finish with what we have instead of being cut off
        async for product in crawl(inp["startUrls"]):    # yields Product records, robots-checked
            if pushed >= max_items or time.time() > deadline:
                break
            res = await Actor.push_data(asdict(product))   # one row = one charged result
            pushed += 1
            if getattr(res, "event_charge_limit_reached", False):
                break                                  # the buyer's spending limit: stop, do not fail
        await Actor.set_status_message(f"Done: {pushed} results")

Pay per result is also the honest model for scraping: a buyer pays for rows delivered, not for the time a run took.

6. Store text has hard limits#

Title, SEO title and SEO description all have length caps. Short, plain and honest beats keyword stuffing, and it reads better in a search result.

7. Plan for the day the site changes#

Every actor has a default input that has to work, and a daily check. A site change means a rebuild, not a shrug.