← blog  /  scrapers

Substack API in Python: every post of a newsletter with the archive cursor

2026-11-18  ·  5 min read  ·  scrapers

Substack's RSS feed stops at the newest 20 posts. Its archive endpoint has them all, 25 a page, and it now pages by a cursor: an offset, even offset=0, has answered HTTP 400 since early November. The code that works, measured on three newsletters with 907 to 1,510 posts.

Every Substack newsletter has an RSS feed, and it is the first thing people try. On 17 November 2026 we read three public newsletters through the feed and through the JSON endpoints the Substack website itself uses: Substack's own blog and two large newsletters, one with 1,385 posts and one with 1,510. robots.txt leaves all of these endpoints open.

The short version#

  • The RSS feed (/feed) held the newest 20 posts on all three. The archives held 907, 1,385 and 1,510.
  • The archive endpoint (/api/v1/archive) has them all, 25 at a time. Each answer is an object, {"posts": [...], "nextCursor": "..."}, and the next page is the same request with cursor= set to that value. A limit of 26 or more is an error (HTTP 400).
  • The offset is gone. On 31 October an offset parameter still paged the archive, up to 50 posts a page. On 17 November every request with an offset, offset=0 included, answered HTTP 400: "param":"offset","msg":"Invalid value". Code that pages by offset now stops at its first request.
  • Rate limits: with 3 seconds between pages, no request got 429 Too Many Requests, and the full walks took 116, 180 and 195 seconds. On 31 October, with 1 second between pages, the two big newsletters answered 429 three times; waiting 30 seconds was enough each time.
  • Paid posts were 17 % and 19 % of the two big archives. The archive lists them with title and date like any other post. The post endpoint gives a reader without a subscription the free part only: 1,371 of 2,814 words in one, nothing at all in another.

The code#

Every post of a Substack newsletter, by cursor, with the 429s handledpython
import json, time, urllib.error, urllib.parse, urllib.request, urllib.robotparser

UA = {"User-Agent": "substack-reader/1.0 (you@example.com)"}
waits = []                                  # seconds waited after each 429, to see how hard the site pushed back


def get_json(url, tries=5):
    """GET and parse JSON. On 429 Too Many Requests wait (Retry-After if it gives seconds, else 30 s more each time)."""
    for k in range(tries):
        try:
            with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            if e.code != 429 or k == tries - 1:
                raise
            after = e.headers.get("Retry-After") or ""
            waits.append(int(after) if after.isdigit() else 30 * (k + 1))
            time.sleep(waits[-1])


def robots(host):
    """robots.txt rules of a host. RobotFileParser.read() asks with Python's own User-Agent, Substack answers 403, and
    the parser then calls every path closed; so fetch the file with our User-Agent and parse() it."""
    with urllib.request.urlopen(urllib.request.Request(f"https://{host}/robots.txt", headers=UA), timeout=30) as r:
        rules = urllib.robotparser.RobotFileParser()
        rules.parse(r.read().decode("utf-8", "replace").splitlines())
        return rules


def archive(pub, pause=3.0):
    """Every post of a newsletter, newest first. Each page is {"posts": [...], "nextCursor": "..."}, 25 posts at most;
    the next page is the same request with cursor=<nextCursor>. An offset is refused (HTTP 400), even offset=0."""
    posts, cursor, seen = [], None, set()
    while True:
        url = f"https://{pub}/api/v1/archive?sort=new&limit=25"
        if cursor:
            url += "&cursor=" + urllib.parse.quote(cursor)
        page = get_json(url)
        posts += page["posts"]
        cursor = page.get("nextCursor")
        if not page["posts"] or not cursor or cursor in seen:
            return posts
        seen.add(cursor)
        time.sleep(pause)


def post(pub, slug):
    """One post with its body. A paid post comes back with only its free part unless you are a paying reader."""
    return get_json(f"https://{pub}/api/v1/posts/{slug}")


def row(p):
    return {"title": p["title"], "date": p["post_date"][:10], "url": p["canonical_url"],
            "paid": p["audience"] != "everyone", "words": p["wordcount"],
            "likes": p["reaction_count"], "comments": p["comment_count"]}
Run itpython
posts = archive("on.substack.com")
print(len(posts), "posts")
print(row(posts[0]))

On Substack's blog that printed 907 posts, then the newest one as a row: its title, date 2026-11-12, link, paid: False, 1,312 words, 1,671 likes and 0 comments.

The cursor is a date#

nextCursor is base64 for a small JSON object holding the date of the last post on the page: {"post_date":"2026-06-03T17:02:00.083941Z"} after the first page of Substack's blog. The second page began with a post of 2 June, and no post was on both pages. Pass the cursor back as it came, URL encoded (it can end in ==), and do not build one yourself. archive() stops on an empty page, a missing cursor or a cursor it has seen before, so a change on Substack's side ends the loop instead of running it forever.

Where the newsletters live#

Two of the three newsletters moved from name.substack.com to their own domain, and the old address redirects there. Python's urllib follows the redirect, so archive() works with either address. Check robots.txt on the domain you end up on, though. On all five hosts it leaves /feed, /api/v1/archive and /api/v1/posts/ open and closes the comment pages (/p/*/comment/*), so archive() reads the comment count from each row and never opens a comment page.

Two traps in Python's robots.txt reader#

  • RobotFileParser.read() is refused. It asks for robots.txt with Python's own User-Agent, Substack answers HTTP 403, and the parser then takes every path as closed: can_fetch() said no even to /feed. robots() above fetches the file with your User-Agent and hands it to parse().
  • A * inside a path is not read as a wildcard. On Substack's own file the standard reader said yes to /p/x/comment/1, which /p/*/comment/* closes. Check the rules with a * in them by hand for the paths you read.

Many newsletters, every day#

For a list of newsletters on a schedule, our Substack scraper on Apify does the reading. Paste publication addresses (an old name.substack.com address of a newsletter that moved to its own domain does not work) and it returns their posts, 50 from each newsletter and 100 in a run unless you raise the limits (up to 5,000 and 50,000), with title, subtitle, date, link, paywall flag, reactions, comments, word count and section, and the post text if you ask for it. It is billed per result row.

written