重开仓库内容：改推五个技能（browser-harness / humanizer / humanizer-zh / product-planning / session-mechanism）
按授权清空原有内容后重新提交（原 oil-ui-pro 一并移出，可从历史恢复）。 browser-harness 剔除 .venv 等运行环境；根 .gitignore 补记 .venv/ 与 node_modules/。
This commit is contained in:
admin committed
1 parent
9a152a1952
commit
777f7fe5d0
240 files changed
+52518
-4409
No files matched your search
@@ -0,0 +1,338 @@
|
||||
# Substack — Data Extraction
|
||||
|
||||
Field-tested against multiple Substack publications on 2026-04-27.
|
||||
No authentication required for any approach documented here.
|
||||
All endpoints work via `http_get` without a browser.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Substack exposes a clean public REST API at `{publication}.substack.com/api/v1/`.
|
||||
Every publication hosted on Substack (custom domain or `{name}.substack.com`)
|
||||
responds to the same API paths. No API key, no login, no browser required.
|
||||
|
||||
**What you can do:**
|
||||
- List all posts from any publication (`/api/v1/posts`)
|
||||
- Fetch full post content by slug (`/api/v1/posts/{slug}`)
|
||||
- Fetch post comments (`/api/v1/post/{post_id}/comments`)
|
||||
- Read the RSS feed (`/feed`) for title/date/link/description metadata
|
||||
|
||||
**Limitations:**
|
||||
- Paid-only post bodies return a truncated HTML preview for `body_html` (not the full article)
|
||||
- No cross-publication search API accessible without a logged-in session
|
||||
- Comment endpoint uses `post_id` (integer), not slug
|
||||
|
||||
---
|
||||
|
||||
## Approach 1 (Recommended): Publication Post List
|
||||
|
||||
`GET https://{subdomain}.substack.com/api/v1/posts?limit=N&offset=N`
|
||||
|
||||
Works for any Substack publication. Returns posts sorted newest-first.
|
||||
|
||||
```python
|
||||
from helpers import http_get
|
||||
import json
|
||||
|
||||
def substack_list_posts(publication_url, limit=20, offset=0):
|
||||
"""List posts from a Substack publication.
|
||||
|
||||
Args:
|
||||
publication_url: Base URL of the publication, e.g.
|
||||
'https://www.slowboring.com' or
|
||||
'https://simonwillison.substack.com'
|
||||
limit: Number of posts to return (max observed: 100)
|
||||
offset: Pagination offset
|
||||
|
||||
Returns list of post dicts with keys: title, subtitle, slug,
|
||||
canonical_url, post_date, audience, wordcount, reactions, restacks.
|
||||
audience is 'everyone' (free) or 'only_paid' (paywalled).
|
||||
"""
|
||||
url = f"{publication_url.rstrip('/')}/api/v1/posts?limit={limit}&offset={offset}"
|
||||
posts = json.loads(http_get(url))
|
||||
return [
|
||||
{
|
||||
"title": p.get("title"),
|
||||
"subtitle": p.get("subtitle"),
|
||||
"slug": p.get("slug"),
|
||||
"url": p.get("canonical_url"),
|
||||
"post_date": p.get("post_date"),
|
||||
"audience": p.get("audience"), # 'everyone' or 'only_paid'
|
||||
"wordcount": p.get("wordcount"),
|
||||
"reactions": p.get("reactions"), # e.g. {"❤": 221}
|
||||
"restacks": p.get("restacks"),
|
||||
"cover_image": p.get("cover_image"),
|
||||
"post_id": p.get("id"),
|
||||
}
|
||||
for p in posts
|
||||
]
|
||||
|
||||
posts = substack_list_posts("https://www.slowboring.com", limit=10)
|
||||
# [
|
||||
# {
|
||||
# "title": "What to make of the generic ballot",
|
||||
# "subtitle": "Plus ties, Mamdani, the Obama legacy, and fundraising's diminishing returns",
|
||||
# "slug": "what-to-make-of-the-generic-ballot",
|
||||
# "url": "https://www.slowboring.com/p/what-to-make-of-the-generic-ballot",
|
||||
# "post_date": "2026-04-24T10:03:26.581Z",
|
||||
# "audience": "everyone",
|
||||
# "wordcount": 4369,
|
||||
# "reactions": {"❤": 221},
|
||||
# "restacks": 10,
|
||||
# "post_id": 194950421,
|
||||
# },
|
||||
# ...
|
||||
# ]
|
||||
|
||||
# Filter for free (non-paywalled) posts only
|
||||
free_posts = [p for p in posts if p["audience"] == "everyone"]
|
||||
```
|
||||
|
||||
### Pagination
|
||||
|
||||
```python
|
||||
def substack_all_posts(publication_url, max_posts=200):
|
||||
"""Fetch all posts from a publication via paginated API."""
|
||||
all_posts = []
|
||||
offset = 0
|
||||
batch_size = 50
|
||||
while len(all_posts) < max_posts:
|
||||
batch = substack_list_posts(publication_url, limit=batch_size, offset=offset)
|
||||
if not batch:
|
||||
break
|
||||
all_posts.extend(batch)
|
||||
if len(batch) < batch_size:
|
||||
break # last page
|
||||
offset += batch_size
|
||||
return all_posts[:max_posts]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Approach 2: Full Post Content by Slug
|
||||
|
||||
`GET https://{subdomain}.substack.com/api/v1/posts/{slug}`
|
||||
|
||||
Returns the full post including `body_html` for free posts. Paywalled posts
|
||||
return a truncated HTML preview for `body_html` (not the full article).
|
||||
|
||||
```python
|
||||
from helpers import http_get
|
||||
import json, re
|
||||
|
||||
def substack_get_post(publication_url, slug):
|
||||
"""Fetch full content of a single Substack post by slug.
|
||||
|
||||
Returns title, body as plain text, body_html, author, date,
|
||||
and metadata. body_html is a truncated preview for paywalled posts.
|
||||
"""
|
||||
url = f"{publication_url.rstrip('/')}/api/v1/posts/{slug}"
|
||||
post = json.loads(http_get(url))
|
||||
|
||||
body_html = post.get("body_html")
|
||||
body_text = None
|
||||
if body_html:
|
||||
# Strip HTML tags for plain text
|
||||
body_text = re.sub(r'<[^>]+>', ' ', body_html)
|
||||
body_text = re.sub(r'\s+', ' ', body_text).strip()
|
||||
|
||||
return {
|
||||
"title": post.get("title"),
|
||||
"subtitle": post.get("subtitle"),
|
||||
"slug": post.get("slug"),
|
||||
"url": post.get("canonical_url"),
|
||||
"post_date": post.get("post_date"),
|
||||
"audience": post.get("audience"),
|
||||
"wordcount": post.get("wordcount"),
|
||||
"reactions": post.get("reactions"),
|
||||
"restacks": post.get("restacks"),
|
||||
"body_html": body_html, # full article if free; truncated preview if paywalled
|
||||
"body_text": body_text, # full plain text if free; truncated if paywalled
|
||||
"truncated_preview": post.get("truncated_body_text"), # always present
|
||||
"post_id": post.get("id"),
|
||||
"publication_id": post.get("publication_id"),
|
||||
}
|
||||
|
||||
post = substack_get_post(
|
||||
"https://www.slowboring.com",
|
||||
"what-to-make-of-the-generic-ballot"
|
||||
)
|
||||
# Free post (audience == "everyone"):
|
||||
# {
|
||||
# "title": "What to make of the generic ballot",
|
||||
# "audience": "everyone",
|
||||
# "wordcount": 4369,
|
||||
# "body_html": "<p>I suppose this isn't a huge surprise...</p>...", # ~40KB full article
|
||||
# "body_text": "I suppose this isn't a huge surprise ...", # ~25KB plain text
|
||||
# "post_id": 194950421,
|
||||
# }
|
||||
|
||||
# Paywalled post (audience == "only_paid"):
|
||||
# post["body_html"] -> truncated HTML preview (a few hundred bytes, not the full article)
|
||||
# post["body_text"] -> truncated plain text (stripped from truncated HTML)
|
||||
# post["truncated_preview"] -> short plaintext excerpt (separate, always present)
|
||||
# Use audience == "everyone" as the reliable signal for full content availability.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Approach 3: Post Comments
|
||||
|
||||
`GET https://{subdomain}.substack.com/api/v1/post/{post_id}/comments?limit=N`
|
||||
|
||||
Note: uses **integer `post_id`**, not slug. Get `post_id` from the post list
|
||||
or post detail responses.
|
||||
|
||||
```python
|
||||
from helpers import http_get
|
||||
import json
|
||||
|
||||
def substack_get_comments(publication_url, post_id, limit=50):
|
||||
"""Fetch top-level comments for a Substack post.
|
||||
|
||||
Args:
|
||||
publication_url: Base URL of the publication
|
||||
post_id: Integer post ID (from post list or post detail)
|
||||
limit: Max comments to return
|
||||
|
||||
Returns list of comment dicts.
|
||||
"""
|
||||
url = f"{publication_url.rstrip('/')}/api/v1/post/{post_id}/comments?limit={limit}"
|
||||
data = json.loads(http_get(url))
|
||||
comments = data.get("comments", [])
|
||||
return [
|
||||
{
|
||||
"comment_id": c.get("id"),
|
||||
"author": c.get("name"),
|
||||
"author_handle": c.get("handle"),
|
||||
"body": c.get("body"),
|
||||
"date": c.get("date"),
|
||||
"reaction_count": c.get("reaction_count"), # e.g. {"❤": 99}
|
||||
"children_count": c.get("children_count"), # reply count
|
||||
"restacks": c.get("restacks"),
|
||||
}
|
||||
for c in comments
|
||||
if not c.get("deleted")
|
||||
]
|
||||
|
||||
comments = substack_get_comments("https://www.slowboring.com", 194950421, limit=10)
|
||||
# [
|
||||
# {
|
||||
# "comment_id": 248392394,
|
||||
# "author": "John from FL",
|
||||
# "body": "Sam asks: \"don't they kind of have a point...\"",
|
||||
# "date": "2026-04-24T10:20:21.997Z",
|
||||
# "reaction_count": {"❤": 99},
|
||||
# "children_count": 3,
|
||||
# },
|
||||
# ...
|
||||
# ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Approach 4: RSS Feed (Lightweight Metadata)
|
||||
|
||||
`GET https://{subdomain}.substack.com/feed`
|
||||
|
||||
Returns an RSS 2.0 feed. Useful when you only need title/date/link/description
|
||||
without hitting the JSON API. Works as a quick check without parsing JSON.
|
||||
|
||||
```python
|
||||
from helpers import http_get
|
||||
import re
|
||||
|
||||
def substack_rss(publication_url, max_items=20):
|
||||
"""Fetch recent post metadata via RSS feed.
|
||||
|
||||
Lighter than the JSON API — only returns title, link, pubDate,
|
||||
and description (short excerpt). Does not include body_html or wordcount.
|
||||
"""
|
||||
rss = http_get(f"{publication_url.rstrip('/')}/feed")
|
||||
items = re.findall(
|
||||
r'<item>(.*?)</item>',
|
||||
rss,
|
||||
re.DOTALL
|
||||
)[:max_items]
|
||||
|
||||
results = []
|
||||
for item in items:
|
||||
title = re.search(r'<title><!\[CDATA\[(.*?)\]\]></title>', item)
|
||||
link = re.search(r'<link>(https?://[^<]+)</link>', item)
|
||||
date = re.search(r'<pubDate>(.*?)</pubDate>', item)
|
||||
desc = re.search(r'<description><!\[CDATA\[(.*?)\]\]></description>', item, re.DOTALL)
|
||||
results.append({
|
||||
"title": title.group(1) if title else None,
|
||||
"link": link.group(1) if link else None,
|
||||
"pub_date": date.group(1) if date else None,
|
||||
"description": desc.group(1).strip() if desc else None,
|
||||
})
|
||||
return results
|
||||
|
||||
feed = substack_rss("https://www.slowboring.com", max_items=5)
|
||||
# [
|
||||
# {
|
||||
# "title": "Sunday Mailbag + Thread",
|
||||
# "link": "https://www.slowboring.com/p/sunday-mailbag-thread-48b",
|
||||
# "pub_date": "Sun, 26 Apr 2026 17:02:04 GMT",
|
||||
# "description": "Ask your questions below.",
|
||||
# },
|
||||
# ...
|
||||
# ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Publication URL Formats
|
||||
|
||||
Substack publications use one of two URL formats:
|
||||
|
||||
```python
|
||||
# Format 1: native subdomain (older or simpler publications)
|
||||
"https://simonwillison.substack.com"
|
||||
|
||||
# Format 2: custom domain (larger publications, purchased domain)
|
||||
"https://www.slowboring.com" # Matthew Yglesias — Slow Boring
|
||||
"https://unchartedterritories.tomaspueyo.com" # Tomas Pueyo
|
||||
|
||||
# Both formats use identical API paths:
|
||||
# {base_url}/api/v1/posts
|
||||
# {base_url}/api/v1/posts/{slug}
|
||||
# {base_url}/api/v1/post/{post_id}/comments
|
||||
# {base_url}/feed
|
||||
```
|
||||
|
||||
If you only know a publication's Substack handle (e.g., `matthewyglesias`),
|
||||
the canonical subdomain URL is `https://matthewyglesias.substack.com`. Custom
|
||||
domain URLs are listed on the publication's about page or in the RSS feed's
|
||||
`<link>` element.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Paywalled post `body_html` is a truncated preview, not `null`** — the API
|
||||
returns a short HTML excerpt (typically a few hundred to a few KB). It is
|
||||
never `null`. The reliable way to detect full content availability is
|
||||
`audience == "everyone"`. For paywalled posts, compare `len(body_html)` to
|
||||
`wordcount * ~7` (average bytes per word) — a large gap means truncation.
|
||||
`truncated_body_text` (plaintext) is always present regardless of audience.
|
||||
- **Comments endpoint uses integer `post_id`, not slug** — `/api/v1/post/{id}/comments`
|
||||
is correct. `/api/v1/posts/{slug}/comments` returns 404.
|
||||
- **`reactions` field is a dict with emoji keys**, e.g. `{"❤": 221}` — not a
|
||||
plain integer. Sum the values for total reaction count:
|
||||
`total = sum(post["reactions"].values())`.
|
||||
- **`limit` on post list is not strictly capped** — values up to at least 100
|
||||
work; beyond that behavior is untested.
|
||||
- **Custom domains and `{name}.substack.com` are interchangeable** — use
|
||||
whichever you have. The `x-sub` response header always reflects the internal
|
||||
publication handle.
|
||||
- **`audience` values**: only `"everyone"` and `"only_paid"` observed. A third
|
||||
value `"founding"` exists in Substack's data model but is rare.
|
||||
- **No unauthenticated cross-publication search** — `substack.com/api/v1/search`
|
||||
returns HTML (a React page), not JSON. To find publications, use external
|
||||
search engines (`site:substack.com {query}`) or the RSS discovery approach.
|
||||
- **Podcast posts** have `type == "podcast"` and `podcast_url` set; their
|
||||
`body_html` may be a show-notes HTML block. Check `type` to distinguish
|
||||
newsletter posts from podcast episodes.
|
||||
Reference in new issue
Block a user