Substack API in Python: every post of a newsletter with the archive cursor
Substack's RSS feed stops at the newest 20 posts. Its archive endpoint has them all, 25 a page, and it now pages by a cursor: an offset, even offset=0, has answered HTTP 400 since early November. The code that works, measured on three newsletters with 907 to 1,510 posts.
Every Substack newsletter has an RSS feed, and it is the first thing people try. On 17 November 2026 we read three public newsletters through the feed and through the JSON endpoints the Substack website itself uses: Substack's own blog and two large newsletters, one with 1,385 posts and one with 1,510. robots.txt leaves all of these endpoints open.
The short version#
- The RSS feed (
/feed) held the newest 20 posts on all three. The archives held 907, 1,385 and 1,510. - The archive endpoint (
/api/v1/archive) has them all, 25 at a time. Each answer is an object,{"posts": [...], "nextCursor": "..."}, and the next page is the same request withcursor=set to that value. A limit of 26 or more is an error (HTTP 400). - The offset is gone. On 31 October an
offsetparameter still paged the archive, up to 50 posts a page. On 17 November every request with an offset,offset=0included, answered HTTP 400:"param":"offset","msg":"Invalid value". Code that pages by offset now stops at its first request. - Rate limits: with 3 seconds between pages, no request got 429 Too Many Requests, and the full walks took 116, 180 and 195 seconds. On 31 October, with 1 second between pages, the two big newsletters answered 429 three times; waiting 30 seconds was enough each time.
- Paid posts were 17 % and 19 % of the two big archives. The archive lists them with title and date like any other post. The post endpoint gives a reader without a subscription the free part only: 1,371 of 2,814 words in one, nothing at all in another.
The code#
import json, time, urllib.error, urllib.parse, urllib.request, urllib.robotparser
UA = {"User-Agent": "substack-reader/1.0 (you@example.com)"}
waits = [] # seconds waited after each 429, to see how hard the site pushed back
def get_json(url, tries=5):
"""GET and parse JSON. On 429 Too Many Requests wait (Retry-After if it gives seconds, else 30 s more each time)."""
for k in range(tries):
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
return json.load(r)
except urllib.error.HTTPError as e:
if e.code != 429 or k == tries - 1:
raise
after = e.headers.get("Retry-After") or ""
waits.append(int(after) if after.isdigit() else 30 * (k + 1))
time.sleep(waits[-1])
def robots(host):
"""robots.txt rules of a host. RobotFileParser.read() asks with Python's own User-Agent, Substack answers 403, and
the parser then calls every path closed; so fetch the file with our User-Agent and parse() it."""
with urllib.request.urlopen(urllib.request.Request(f"https://{host}/robots.txt", headers=UA), timeout=30) as r:
rules = urllib.robotparser.RobotFileParser()
rules.parse(r.read().decode("utf-8", "replace").splitlines())
return rules
def archive(pub, pause=3.0):
"""Every post of a newsletter, newest first. Each page is {"posts": [...], "nextCursor": "..."}, 25 posts at most;
the next page is the same request with cursor=<nextCursor>. An offset is refused (HTTP 400), even offset=0."""
posts, cursor, seen = [], None, set()
while True:
url = f"https://{pub}/api/v1/archive?sort=new&limit=25"
if cursor:
url += "&cursor=" + urllib.parse.quote(cursor)
page = get_json(url)
posts += page["posts"]
cursor = page.get("nextCursor")
if not page["posts"] or not cursor or cursor in seen:
return posts
seen.add(cursor)
time.sleep(pause)
def post(pub, slug):
"""One post with its body. A paid post comes back with only its free part unless you are a paying reader."""
return get_json(f"https://{pub}/api/v1/posts/{slug}")
def row(p):
return {"title": p["title"], "date": p["post_date"][:10], "url": p["canonical_url"],
"paid": p["audience"] != "everyone", "words": p["wordcount"],
"likes": p["reaction_count"], "comments": p["comment_count"]}posts = archive("on.substack.com")
print(len(posts), "posts")
print(row(posts[0]))On Substack's blog that printed 907 posts, then the newest one as a row: its title, date 2026-11-12, link, paid: False, 1,312 words, 1,671 likes and 0 comments.
The cursor is a date#
nextCursor is base64 for a small JSON object holding the date of the last post on the page: {"post_date":"2026-06-03T17:02:00.083941Z"} after the first page of Substack's blog. The second page began with a post of 2 June, and no post was on both pages. Pass the cursor back as it came, URL encoded (it can end in ==), and do not build one yourself. archive() stops on an empty page, a missing cursor or a cursor it has seen before, so a change on Substack's side ends the loop instead of running it forever.
Where the newsletters live#
Two of the three newsletters moved from name.substack.com to their own domain, and the old address redirects there. Python's urllib follows the redirect, so archive() works with either address. Check robots.txt on the domain you end up on, though. On all five hosts it leaves /feed, /api/v1/archive and /api/v1/posts/ open and closes the comment pages (/p/*/comment/*), so archive() reads the comment count from each row and never opens a comment page.
Two traps in Python's robots.txt reader#
RobotFileParser.read()is refused. It asks for robots.txt with Python's own User-Agent, Substack answers HTTP 403, and the parser then takes every path as closed:can_fetch()said no even to/feed.robots()above fetches the file with your User-Agent and hands it toparse().- A
*inside a path is not read as a wildcard. On Substack's own file the standard reader said yes to/p/x/comment/1, which/p/*/comment/*closes. Check the rules with a*in them by hand for the paths you read.
Many newsletters, every day#
For a list of newsletters on a schedule, our Substack scraper on Apify does the reading. Paste publication addresses (an old name.substack.com address of a newsletter that moved to its own domain does not work) and it returns their posts, 50 from each newsletter and 100 in a run unless you raise the limits (up to 5,000 and 50,000), with title, subtitle, date, link, paywall flag, reactions, comments, word count and section, and the post text if you ask for it. It is billed per result row.
written
12 min readWorkable, Recruitee, Personio and BambooHR job board APIs: no key, and a date that costs a request
Workable, Recruitee, Personio and BambooHR give a board's open jobs with no key, whole in one request; only BambooHR's list has no date. The four addresses, the catch in each, and about sixty lines of Python that put all four into one shape.
11 min readWorkday jobs API in Python: a list that stops at 2,000 jobs, and dates that stop at 30+ days
Read a Workday careers site as JSON in Python: 20 jobs a request, and a list that stops at 2,000 of NVIDIA's 2,807 jobs. The two endpoints, facets that cut a big site into slices, and about forty lines of Python, measured on four career sites.
7 min readRemote job APIs: five free feeds, and the catch in each
Remote OK, We Work Remotely, Jobicy, Working Nomads and Arbeitnow publish their jobs without a key. What each feed holds, how far back it goes, the terms, and sixty lines of Python that put them in one list.