重开仓库内容：改推五个技能（browser-harness / humanizer / humanizer-zh / product-planning / session-mechanism）
按授权清空原有内容后重新提交（原 oil-ui-pro 一并移出，可从历史恢复）。 browser-harness 剔除 .venv 等运行环境；根 .gitignore 补记 .venv/ 与 node_modules/。
This commit is contained in:
admin committed
1 parent
9a152a1952
commit
777f7fe5d0
240 files changed
+52518
-4409
No files matched your search
@@ -0,0 +1,306 @@
|
||||
# Wayback Machine — CDX API & Snapshot Retrieval
|
||||
|
||||
`https://web.archive.org` — all public data, no auth or API key required. Everything here is pure `http_get` — no browser needed.
|
||||
|
||||
> **NOTE:** A comprehensive Internet Archive skill (covering CDX, item metadata, and search) already exists at `domain-skills/archive-org/scraping.md`. This file is a focused, CDX-first quick-reference for Wayback Machine snapshot work specifically.
|
||||
|
||||
## Start here: CDX API
|
||||
|
||||
The CDX (Capture/Crawl Index) API is the single fastest way to query the Wayback Machine. It returns structured JSON and supports filtering, collapsing, pagination, and nearest-date lookups.
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# Find all snapshots of a URL — the minimal starting query
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json&limit=10"
|
||||
"&fl=timestamp,original,statuscode,mimetype,length",
|
||||
timeout=40.0 # CDX is slow — never use less than 40s
|
||||
)
|
||||
rows = json.loads(r)
|
||||
# rows[0] is ALWAYS the header row — slice rows[1:] for data
|
||||
for ts, orig, status, mime, length in rows[1:]:
|
||||
print(f"https://web.archive.org/web/{ts}/{orig} [{status}]")
|
||||
```
|
||||
|
||||
**All CDX values are strings**, even numeric ones (`status='200'`, `length='4821'`). Cast explicitly with `int()` / `float()`.
|
||||
|
||||
---
|
||||
|
||||
## Core CDX patterns
|
||||
|
||||
### Nearest snapshot to a target date
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json&limit=1"
|
||||
"&fl=timestamp,original,statuscode"
|
||||
"&closest=20230601120000&sort=closest",
|
||||
timeout=60.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
ts, orig, status = rows[1] # rows[0] is header
|
||||
snap_url = f"https://web.archive.org/web/{ts}/{orig}"
|
||||
# Timestamp format: 14-digit YYYYMMDDHHMMSS
|
||||
# Prefix shorthand: '20230601' (day), '202306' (month), '2023' (year)
|
||||
```
|
||||
|
||||
### One snapshot per month (collapsed)
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json"
|
||||
"&collapse=timestamp:6" # :6 = one per YYYYMM
|
||||
"&from=20220101&to=20230101"
|
||||
"&fl=timestamp,original,statuscode",
|
||||
timeout=60.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
for ts, orig, status in rows[1:]:
|
||||
print(f"{ts[:4]}-{ts[4:6]} https://web.archive.org/web/{ts}/{orig}")
|
||||
|
||||
# collapse=timestamp:N — collapse by first N timestamp digits:
|
||||
# :4 = one per year
|
||||
# :6 = one per month (most common)
|
||||
# :8 = one per day
|
||||
# :10 = one per hour
|
||||
# Keeps the FIRST capture of each period — not the last.
|
||||
```
|
||||
|
||||
### All pages under a domain or path
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# matchType=prefix — all URLs starting with the given path
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com/blog/&matchType=prefix&output=json"
|
||||
"&limit=20&fl=timestamp,original,statuscode"
|
||||
"&filter=statuscode:200", # only successful captures
|
||||
timeout=60.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
for row in rows[1:]:
|
||||
print(row)
|
||||
|
||||
# matchType options:
|
||||
# exact (default) — this URL only
|
||||
# prefix — URL + all subpaths
|
||||
# host — all subdomains of the host
|
||||
# domain — host + all subdomains (broadest)
|
||||
```
|
||||
|
||||
### Filter by status code or MIME type
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# Only successful HTML captures — combine multiple filters
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json"
|
||||
"&filter=statuscode:200"
|
||||
"&filter=mimetype:text/html"
|
||||
"&fl=timestamp,original,length"
|
||||
"&limit=10",
|
||||
timeout=40.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
|
||||
# filter= uses regex. Examples:
|
||||
# &filter=statuscode:200 exact match
|
||||
# &filter=!statuscode:200 negation (all non-200)
|
||||
# &filter=statuscode:[23].. 2xx and 3xx only
|
||||
# &filter=mimetype:text/html HTML only
|
||||
# &filter=original:.*\\.pdf URLs ending in .pdf
|
||||
# Multiple &filter= params are ANDed together.
|
||||
```
|
||||
|
||||
### CDX field reference
|
||||
|
||||
| Field | Description | Example value |
|
||||
|---|---|---|
|
||||
| `urlkey` | SURT-format URL (reversed domain) | `com,example)/` |
|
||||
| `timestamp` | Capture time, 14-digit `YYYYMMDDHHMMSS` | `20230601114925` |
|
||||
| `original` | Original crawled URL (includes port if non-standard) | `https://example.com/` |
|
||||
| `mimetype` | Content-Type at crawl time | `text/html` |
|
||||
| `statuscode` | HTTP status at crawl time (string) | `200` |
|
||||
| `digest` | SHA-1 of body, base32-encoded | `I4YBMQ6PHPWE2TD6TIXNWHZB6MXRNTSR` |
|
||||
| `length` | Content-length in bytes (string) | `4821` |
|
||||
|
||||
Default `fl=` when omitted: all 7 fields above in that order.
|
||||
|
||||
---
|
||||
|
||||
## Availability API (DO NOT USE as primary)
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# WARNING: This API is BROKEN — returns empty archived_snapshots
|
||||
# for URLs that ARE in the archive. Confirmed broken 2026-04-18.
|
||||
# Use CDX with ?sort=closest&limit=1 instead (see above).
|
||||
|
||||
# Left here for reference only — do not rely on it:
|
||||
r = http_get(
|
||||
"https://archive.org/wayback/available"
|
||||
"?url=example.com×tamp=20240101",
|
||||
timeout=20.0
|
||||
)
|
||||
data = json.loads(r)
|
||||
# Returns: {"url": "example.com", "archived_snapshots": {}}
|
||||
# Even for well-archived URLs. Do not trust empty results.
|
||||
|
||||
# CORRECT replacement:
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json&limit=1"
|
||||
"&fl=timestamp,original,statuscode"
|
||||
"&closest=20240101000000&sort=closest",
|
||||
timeout=60.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
if len(rows) > 1:
|
||||
ts, orig, status = rows[1]
|
||||
snap_url = f"https://web.archive.org/web/{ts}/{orig}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Paginate large result sets
|
||||
|
||||
```python
|
||||
import json
|
||||
from urllib.parse import quote
|
||||
|
||||
def cdx_all_snapshots(url, fl="timestamp,original,statuscode", page_size=500):
|
||||
"""Yield all CDX rows for a URL, page by page."""
|
||||
base = (
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
f"?url={quote(url, safe='')}&output=json"
|
||||
f"&fl={fl}&limit={page_size}&showResumeKey=true"
|
||||
)
|
||||
resume_key = None
|
||||
while True:
|
||||
endpoint = base if resume_key is None else f"{base}&resumeKey={quote(resume_key)}"
|
||||
rows = json.loads(http_get(endpoint, timeout=60.0))
|
||||
# With showResumeKey=true, last two rows are [] and ['<key>']
|
||||
has_resume = len(rows) >= 2 and rows[-2] == [] and rows[-1] != []
|
||||
data_rows = rows[1:-2] if has_resume else rows[1:]
|
||||
for row in data_rows:
|
||||
yield row
|
||||
if not has_resume:
|
||||
break
|
||||
resume_key = rows[-1][0]
|
||||
|
||||
for ts, orig, status in cdx_all_snapshots("example.com"):
|
||||
snap_url = f"https://web.archive.org/web/{ts}/{orig}"
|
||||
# process...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retrieve the archived page
|
||||
|
||||
```python
|
||||
# Direct snapshot URL: /web/{14-digit-timestamp}/{original-url}
|
||||
snap_url = "https://web.archive.org/web/20230601114925/https://example.com/"
|
||||
content = http_get(snap_url, timeout=30.0)
|
||||
# Returns archived HTML with Wayback toolbar injected inside:
|
||||
# <!-- BEGIN WAYBACK TOOLBAR INSERT --> ... <!-- END WAYBACK TOOLBAR INSERT -->
|
||||
# Strip those comments + their contents if you want the original HTML.
|
||||
|
||||
# Canonical form for "get latest available" — use 14 zeros:
|
||||
latest = "https://web.archive.org/web/20240101000000*/example.com"
|
||||
# The * suffix returns a calendar page (HTML), not the archived page itself.
|
||||
# Use CDX to find the real timestamp, then fetch the direct URL.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced CDX: deduplicate by content digest
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# Find only snapshots where the content CHANGED — dedup by SHA-1 digest
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&output=json"
|
||||
"&collapse=digest" # one capture per unique body hash
|
||||
"&fl=timestamp,original,digest,length"
|
||||
"&filter=statuscode:200",
|
||||
timeout=60.0
|
||||
)
|
||||
rows = json.loads(r)
|
||||
# rows[1:] are unique content versions across all time
|
||||
# Useful for detecting when a page actually changed, vs. being re-crawled identically
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CDX summary/count query
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# showNumPages=true returns total page count, not records
|
||||
# Use for estimating result size before a full fetch
|
||||
r = http_get(
|
||||
"https://web.archive.org/cdx/search/cdx"
|
||||
"?url=example.com&matchType=prefix"
|
||||
"&showNumPages=true",
|
||||
timeout=30.0
|
||||
)
|
||||
page_count = int(r.strip()) # returns plain integer, not JSON
|
||||
# 1 page ~ 150,000 records by default
|
||||
# Combine with &page=N for manual page-based pagination:
|
||||
# ?url=...&output=json&page=0, ?url=...&output=json&page=1, etc.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate limits & timeouts
|
||||
|
||||
| API | Typical latency | Safe timeout | Notes |
|
||||
|---|---|---|---|
|
||||
| CDX search | 5–40s | 60s | Intermittently slow; retry on timeout |
|
||||
| Snapshot fetch (`/web/`) | 2–10s | 30s | Reliable |
|
||||
| Metadata (`/metadata/`) | <1s | 20s | Fast, stable |
|
||||
| Advanced search | <1s | 20s | Fast, stable |
|
||||
|
||||
No API key required. No documented rate limit. Be respectful: add `time.sleep(1)` between CDX calls in loops. 3 rapid sequential CDX calls (~10s) complete fine; 10+ rapid calls produce timeouts.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **CDX is slow — always set `timeout=60.0` for CDX calls.** 40s minimum, 60s recommended. Metadata and search APIs are fine at 20s. CDX slowness is server-side and unpredictable.
|
||||
|
||||
- **Availability API (`/wayback/available`) is broken.** Returns `{"archived_snapshots": {}}` even for URLs with thousands of captures. Tested 2026-04-18 — do not use. Replacement: CDX with `?sort=closest&limit=1`.
|
||||
|
||||
- **`rows[0]` is always the header when `output=json`.** Always slice `rows[1:]` for data. Forgetting this causes silent type errors because you're destructuring column names, not values.
|
||||
|
||||
- **`output=json` must be explicit.** Omitting it returns space-separated text. There is no default JSON mode.
|
||||
|
||||
- **All CDX values are strings.** `statuscode='200'` not `200`, `length='4821'` not `4821`. Cast: `int(row[4])`, `int(row[6])`.
|
||||
|
||||
- **`original` preserves non-standard ports.** Old crawls captured `http://www.example.com:80/` — the `:80` is part of the `original` field. Build playback URLs verbatim: `f"https://web.archive.org/web/{ts}/{orig}"` works correctly with the port.
|
||||
|
||||
- **`from=` / `to=` timestamps are exclusive at the boundary.** `to=20231231` means `to=20231231000000` — it excludes captures from Dec 31 itself. Use `to=20240101` to include all of 2023.
|
||||
|
||||
- **`collapse=timestamp:6` keeps the FIRST capture of each period.** Not the most recent. Reverse the result set or filter client-side if you need the last.
|
||||
|
||||
- **CDX `matchType=domain` can return millions of rows for popular sites.** Always add `&limit=` or `&showNumPages=true` first to estimate size.
|
||||
|
||||
- **`showResumeKey=true` appends two sentinel rows.** The second-to-last row is `[]` (empty separator), the last row is `['<resume_key_string>']`. Slice `rows[1:-2]` for data rows when a resume key is present.
|
||||
|
||||
- **Wayback toolbar is injected into every archived HTML page.** The injection is wrapped in `<!-- BEGIN WAYBACK TOOLBAR INSERT -->` / `<!-- END WAYBACK TOOLBAR INSERT -->` comments. Strip them if you need original HTML fidelity.
|
||||
Reference in new issue
Block a user