workbuddy_skills/browser-harness/agent-workspace/domain-skills/goodreads/scraping.md

# Goodreads — Book Data Extraction

Field-tested against goodreads.com on 2026-04-18 via `http_get` (no browser required).
All five URL types return full HTML with no bot-wall, CAPTCHA, or login gate.

## Access Summary

| Page type          | `http_get` works? | Data format              |
|--------------------|-------------------|--------------------------|
| Book show page     | Yes               | `__NEXT_DATA__` + JSON-LD |
| Search results     | Yes               | Server-rendered HTML (schema.org microdata) |
| Author show page   | Yes               | Server-rendered HTML + OG meta |
| Listopia list page | Yes               | Server-rendered HTML (schema.org microdata) |

Goodreads shut down its public API in 2020. All extraction is HTML-based.
Open Library is a reliable supplement with a free JSON API (see [Open Library fallback](#open-library-api-fallback)).

---

## Book Page — Full Data (`__NEXT_DATA__`)

URL pattern: `https://www.goodreads.com/book/show/{book_id}` or `/{book_id}.{Slug}`

The slug is optional — numeric ID alone works and redirects cleanly.

```python
import re, json
from helpers import http_get

def parse_book(book_id):
    html = http_get(f"https://www.goodreads.com/book/show/{book_id}")

    # Parse Apollo state from Next.js page
    nd = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.DOTALL)
    ap = json.loads(nd.group(1))['props']['pageProps']['apolloState']

    # The primary Book entity matches the URL's legacy ID
    book = next(v for v in ap.values()
                if v.get('__typename') == 'Book' and v.get('legacyId') == int(book_id))
    work = next((v for v in ap.values() if v.get('__typename') == 'Work'), {})
    author_ref = book['primaryContributorEdge']['node']['__ref']
    author = ap.get(author_ref, {})

    stats = work.get('stats', {})
    work_details = work.get('details', {})
    book_details = book.get('details', {})

    return {
        'title':            book['title'],
        'title_complete':   book['titleComplete'],
        'book_id':          book['legacyId'],
        'url':              book['webUrl'],
        'cover_url':        book['imageUrl'],
        # Strip HTML tags from description
        'description':      re.sub(r'<[^>]+>', '', book.get('description({"stripped":true})',
                                   book.get('description', ''))).strip(),
        'genres':           [g['genre']['name'] for g in book.get('bookGenres', [])],
        'series':           [{'name': s['series']['title'], 'position': s.get('userPosition')}
                             for s in book.get('bookSeries', [])],
        # Author
        'author_name':      author.get('name'),
        'author_url':       author.get('webUrl'),
        # Edition details
        'format':           book_details.get('format'),
        'num_pages':        book_details.get('numPages'),
        'publisher':        book_details.get('publisher'),
        'language':         (book_details.get('language') or {}).get('name'),
        'isbn':             book_details.get('isbn'),
        'isbn13':           book_details.get('isbn13'),
        'pub_timestamp_ms': book_details.get('publicationTime'),
        # Ratings (from Work, not Book)
        'avg_rating':       stats.get('averageRating'),
        'ratings_count':    stats.get('ratingsCount'),
        'text_reviews':     stats.get('textReviewsCount'),
        # ratings_dist is list of counts for [1-star, 2-star, 3-star, 4-star, 5-star]
        'ratings_dist':     stats.get('ratingsCountDist'),
        # Awards
        'awards':           [a['name'] + (' — ' + a['category'] if a.get('category') else '')
                             for a in work_details.get('awardsWon', [])],
    }

# Example
book = parse_book(149267)  # The Stand by Stephen King
# book['title']        => "The Stand"
# book['avg_rating']   => 4.35
# book['ratings_count']=> 845591
# book['genres']       => ["Horror", "Fiction", "Fantasy", ...]
# book['awards']       => ["Locus Award — Best SF Novel", ...]
```

**Field notes:**
- `book['legacyId']` is the integer in the URL (e.g. `149267`). Use it to match the correct entity — the `apolloState` often contains 2-3 Book entries for different editions.
- Ratings and awards live in the `Work` entity, not `Book`. The `Work` is always `__typename == 'Work'`.
- `description` comes in two forms: `description` (HTML) and `description({"stripped":true})` (plain text). Prefer the stripped version.
- `pub_timestamp_ms` is a Unix timestamp in **milliseconds**. Convert: `datetime.fromtimestamp(ts/1000)`.
- `isbn` / `isbn13` are often `null` on older editions — the JSON-LD path (below) is no more reliable.

---

## Book Page — Fast Path (JSON-LD)

Use when you only need title, author, rating, page count, and awards. ~3× less parsing code.

```python
import re, json
from helpers import http_get

def parse_book_fast(book_id):
    html = http_get(f"https://www.goodreads.com/book/show/{book_id}")
    blocks = re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.DOTALL)
    if not blocks:
        return None
    ld = json.loads(blocks[0])
    return {
        'title':        ld.get('name'),
        'author':       ld['author'][0]['name'] if ld.get('author') else None,
        'avg_rating':   ld.get('aggregateRating', {}).get('ratingValue'),
        'ratings_count':ld.get('aggregateRating', {}).get('ratingCount'),
        'review_count': ld.get('aggregateRating', {}).get('reviewCount'),
        'num_pages':    ld.get('numberOfPages'),
        'isbn':         ld.get('isbn'),
        'cover_url':    ld.get('image'),
        'awards':       ld.get('awards'),   # single string, comma-separated
        'format':       ld.get('bookFormat'),
    }

book = parse_book_fast(149267)
# book['avg_rating']   => 4.35
# book['ratings_count']=> 845591
```

**JSON-LD does NOT include:** description, genres, series membership, per-star rating distribution, publisher, language.
Use `parse_book()` (the `__NEXT_DATA__` path) when you need any of those.

---

## Search Results

URL: `https://www.goodreads.com/search?q={query}&search_type=books&page={n}`

Search uses server-rendered HTML with schema.org microdata `<tr>` rows. No `__NEXT_DATA__`.

```python
import re, json
from helpers import http_get

def search_books(query, page=1):
    from urllib.parse import quote_plus
    url = f"https://www.goodreads.com/search?q={quote_plus(query)}&search_type=books&page={page}"
    html = http_get(url)

    rows = re.findall(
        r'<tr itemscope itemtype="http://schema.org/Book">(.*?)</tr>',
        html, re.DOTALL
    )

    results = []
    for row in rows:
        bid    = re.search(r'<div id="(\d+)" class="u-anchorTarget">', row)
        title  = re.search(r"itemprop='name'[^>]*>([^<]+)</span>", row)
        author = re.search(r'class="authorName"[^>]*><span[^>]*>([^<]+)</span>', row)
        avg    = re.search(r'(\d+\.\d+)\s*avg rating', row)
        cnt    = re.search(r'(\d[\d,]*)\s*rating', row)
        cover  = re.search(r'img alt="[^"]*" class="bookCover"[^>]*src="([^"]+)"', row)
        if not (bid and title):
            continue
        results.append({
            'book_id':      bid.group(1),
            'title':        title.group(1).strip(),
            'author':       author.group(1).strip() if author else None,
            'avg_rating':   float(avg.group(1)) if avg else None,
            'ratings_count':cnt.group(1).replace(',', '') if cnt else None,
            'cover_url':    cover.group(1) if cover else None,
            'url':          f"https://www.goodreads.com/book/show/{bid.group(1)}",
        })

    total_m = re.search(r'([\d,]+)\s+results', html)
    total   = int(total_m.group(1).replace(',', '')) if total_m else None

    return {'total': total, 'page': page, 'results': results}

# Example
r = search_books("dune")
# r['total']   => 101026
# r['results'] => [{'book_id':'44767458', 'title':'Dune (Dune, #1)', 'avg_rating':4.29, ...}, ...]
```

**Field notes:**
- Returns exactly 20 results per page.
- `total` is the result count shown in `"N results for…"` header.
- The `avg rating` regex uses `&mdash;` (HTML entity) in the raw HTML — the pattern above matches the decoded text.
- `ratings_count` regex hits the first occurrence of `\d+ rating` in the row, which is always the book's count (not a user review count).
- `cover_url` is a 75px thumbnail (`._SY75_.jpg`). Swap `_SY75_` → `_SX315_` for a larger image.

---

## Author Page

URL: `https://www.goodreads.com/author/show/{author_id}.{Slug}`

Author pages are **not** Next.js — they use classic server-rendered HTML with OG meta tags and microdata.
The author ID and slug can be obtained from a book's `author_url` field.

```python
import re, json
from helpers import http_get

def parse_author(author_id_and_slug):
    # author_id_and_slug e.g. "58.Frank_Patrick_Herbert"
    html = http_get(f"https://www.goodreads.com/author/show/{author_id_and_slug}")

    # Name and basic info from OG/meta tags
    name    = re.search(r"<meta content='([^']+)' property='og:title'>", html)
    img     = re.search(r"<meta content='([^']+)' property='og:image'>", html)
    website = re.search(r"Website\s*</div>\s*<div[^>]*>\s*<a[^>]*href=\"([^\"]+)\"", html)

    # Full biography from hidden span (shown/hidden by "...more" toggle in browser)
    bio_span = re.search(
        r'<span id="freeText(?:author|long)\d+"[^>]*>(.*?)</span>',
        html, re.DOTALL
    )
    bio = re.sub(r'<[^>]+>', '', bio_span.group(1)).strip() if bio_span else None

    # Top books listed on the page (10 rows, same microdata format as search)
    rows = re.findall(
        r'<tr itemscope itemtype="http://schema.org/Book">(.*?)</tr>',
        html, re.DOTALL
    )
    books = []
    for row in rows:
        bid   = re.search(r'<div id="(\d+)" class="u-anchorTarget">', row)
        title = re.search(r"itemprop='name'[^>]*>([^<]+)</span>", row)
        avg   = re.search(r'(\d+\.\d+)\s*avg rating', row)
        cnt   = re.search(r'(\d[\d,]*)\s*rating', row)
        if bid and title:
            books.append({
                'book_id':      bid.group(1),
                'title':        title.group(1).strip(),
                'avg_rating':   float(avg.group(1)) if avg else None,
                'ratings_count':cnt.group(1).replace(',', '') if cnt else None,
                'url':          f"https://www.goodreads.com/book/show/{bid.group(1)}",
            })

    return {
        'name':         name.group(1) if name else None,
        'profile_image':img.group(1) if img else None,
        'bio':          bio,
        'website':      website.group(1) if website else None,
        'top_books':    books,
    }

# Example
author = parse_author("58.Frank_Patrick_Herbert")
# author['name']    => "Frank Patrick Herbert"
# author['bio']     => "Franklin Patrick Herbert Jr. was an American science fiction..."
# len(author['top_books']) => 10
```

**Field notes:**
- Author IDs can be found in a book's `author_url` (from `__NEXT_DATA__` or JSON-LD).
- The slug is optional in the URL — numeric ID alone redirects correctly.
- `profile_image` from OG tag is a large portrait (p8 suffix = 800px). Swap to `p5` for 500px.
- The bio is server-rendered in a `<span id="freeTextauthor{ID}">` or `<span id="freeTextlong{ID}">` — which variant appears depends on length.
- Follower count is **not** present in the static HTML — it requires JS execution to appear.
- Page lists exactly 10 books. To get all books, paginate `/author/list/{author_id}?page=N`.

---

## Listopia List Page

URL: `https://www.goodreads.com/list/show/{list_id}.{Slug}?page={n}`

Returns 100 books per page with rank numbers.

```python
import re, json
from helpers import http_get

def parse_list(list_id_and_slug, page=1):
    url = f"https://www.goodreads.com/list/show/{list_id_and_slug}?page={page}"
    html = http_get(url)

    rows = re.findall(
        r'<tr itemscope itemtype="http://schema.org/Book">(.*?)</tr>',
        html, re.DOTALL
    )

    results = []
    for row in rows:
        rank   = re.search(r'<td[^>]*class="number"[^>]*>(\d+)</td>', row)
        bid    = re.search(r'<div id="(\d+)" class="u-anchorTarget">', row)
        title  = re.search(r"itemprop='name'[^>]*>([^<]+)</span>", row)
        author = re.search(r'class="authorName"[^>]*><span[^>]*>([^<]+)</span>', row)
        avg    = re.search(r'(\d+\.\d+)\s*avg rating', row)
        cnt    = re.search(r'(\d[\d,]*)\s*rating', row)
        if not (bid and title):
            continue
        results.append({
            'rank':         int(rank.group(1)) if rank else None,
            'book_id':      bid.group(1),
            'title':        title.group(1).strip(),
            'author':       author.group(1).strip() if author else None,
            'avg_rating':   float(avg.group(1)) if avg else None,
            'ratings_count':cnt.group(1).replace(',', '') if cnt else None,
            'url':          f"https://www.goodreads.com/book/show/{bid.group(1)}",
        })

    return {'page': page, 'results': results}

# Example
lst = parse_list("1.Best_Books_Ever")
# lst['results'][0] => {'rank': 1, 'book_id': '2767052',
#                       'title': 'The Hunger Games (The Hunger Games, #1)',
#                       'author': 'Suzanne Collins', 'avg_rating': 4.35, ...}
```

**Field notes:**
- 100 rows per page. Ranks are sequential across pages (page 2 starts at rank 101).
- Paginate with `?page=2`, `?page=3` etc.
- List pages do not use `__NEXT_DATA__` — same classic HTML format as author pages.

---

## Open Library API Fallback

Use Open Library when you need structured JSON without HTML parsing, or when you want supplementary data (birth/death dates, ISBNs across editions, subjects).

Open Library's ratings are from its own user base (~400 ratings vs. Goodreads' 800k+ for Dune) — use Goodreads ratings when accuracy matters.

### Search

```python
import json
from urllib.parse import quote_plus
from helpers import http_get

def ol_search(query, limit=10):
    url = f"https://openlibrary.org/search.json?q={quote_plus(query)}&limit={limit}"
    data = json.loads(http_get(url))
    results = []
    for doc in data.get('docs', []):
        cover_id = doc.get('cover_i')
        results.append({
            'ol_key':           doc['key'],           # e.g. "/works/OL893415W"
            'title':            doc.get('title'),
            'author':           (doc.get('author_name') or [''])[0],
            'author_key':       (doc.get('author_key') or [''])[0],
            'first_pub_year':   doc.get('first_publish_year'),
            'edition_count':    doc.get('edition_count'),
            'series':           doc.get('series_name'),
            'cover_url':        f"https://covers.openlibrary.org/b/id/{cover_id}-M.jpg" if cover_id else None,
        })
    return {'total': data.get('numFound'), 'results': results}

r = ol_search("dune frank herbert", limit=5)
# r['results'][0]['ol_key']  => "/works/OL893415W"
# r['results'][0]['title']   => "Dune"
```

### Work (book details)

```python
def ol_work(ol_key):
    # ol_key like "/works/OL893415W" or just "OL893415W"
    key = ol_key if ol_key.startswith('/') else f'/works/{ol_key}'
    data = json.loads(http_get(f"https://openlibrary.org{key}.json"))
    desc = data.get('description', '')
    if isinstance(desc, dict):
        desc = desc.get('value', '')
    return {
        'title':    data.get('title'),
        'subjects': data.get('subjects', []),
        'series':   data.get('series', []),
        'description': desc,
        'covers':   data.get('covers', []),
        'links':    data.get('links', []),
    }

work = ol_work("OL893415W")
# work['title']    => "Dune"
# work['subjects'] => ["Dune (Imaginary place)", "Fiction", ...]
```

### Ratings for a work

```python
def ol_ratings(ol_key):
    key = ol_key if ol_key.startswith('/') else f'/works/{ol_key}'
    data = json.loads(http_get(f"https://openlibrary.org{key}/ratings.json"))
    return data.get('summary', {})

# {'average': 4.30, 'count': 414, 'sortable': 4.21}
```

### Author

```python
def ol_author(author_key):
    # author_key like "OL79034A"
    data = json.loads(http_get(f"https://openlibrary.org/authors/{author_key}.json"))
    bio = data.get('bio', '')
    if isinstance(bio, dict):
        bio = bio.get('value', '')
    return {
        'name':        data.get('name'),
        'birth_date':  data.get('birth_date'),
        'death_date':  data.get('death_date'),
        'bio':         bio,
        'ol_key':      data.get('key'),
    }

author = ol_author("OL79034A")
# author['name']       => "Frank Herbert"
# author['birth_date'] => "8 October 1920"
# author['death_date'] => "11 February 1986"
```

---

## Combining Goodreads + Open Library

```python
# Get full book data: Goodreads for ratings/genres/description, OL for ISBNs/edition details
def get_book_full(goodreads_book_id, ol_work_key=None):
    gr = parse_book(goodreads_book_id)
    result = dict(gr)
    if ol_work_key:
        ol = ol_work(ol_work_key)
        result['ol_subjects']    = ol['subjects']
        result['ol_description'] = ol['description']
        result['ol_covers']      = ol['covers']
    return result
```

---

## Gotchas

- **Goodreads API is gone**: The official API was shut down in December 2020. All data must come from HTML scraping or the unofficial paths documented here.

- **Book ID 5107 redirects**: The URL `goodreads.com/book/show/5107.The_Stand` actually resolves to *The Catcher in the Rye* (ID 5107). The Stand is ID `149267`. Always verify `book['legacyId']` matches the URL ID.

- **Author page ID mismatch**: Author ID `10538` in the URL resolves to Carl Sagan, not Frank Herbert (ID `58`). Always obtain author IDs from the `author_url` field inside a book's data rather than guessing.

- **Two Book entities in `apolloState`**: The `apolloState` contains multiple `Book:` entries — one is a stub (only has `legacyId` and `webUrl`), and one is full. Filter by `legacyId == int(book_id)` AND check that the entry has more than 3 fields.

- **Ratings are on `Work`, not `Book`**: `avg_rating`, `ratingsCount`, and `ratingsCountDist` are in the `Work` entity's `stats` key. The `Book` entity has no rating fields.

- **Author pages are old-style HTML**: Author pages (`/author/show/`) do not use Next.js or `__NEXT_DATA__`. Use OG meta tags and regex for extraction. The follower count only loads via JS — it will be missing from `http_get` responses.

- **Search has no `__NEXT_DATA__`**: Search result pages (`/search`) are classic server-rendered HTML. JSON-LD is absent. Use the `<tr itemscope itemtype="http://schema.org/Book">` microdata rows.

- **`ratings_count` regex order matters**: The pattern `r'(\d[\d,]*)\s*rating'` always matches the book's aggregate rating count first in each search row — this is reliable. Do not use `minirating` span text as it contains nested HTML.

- **Open Library cover URLs return binary JPEG**: `http_get()` will raise a `UnicodeDecodeError` on cover image URLs. Use `urllib.request.urlopen()` directly and read bytes, or just store the URL string without fetching.

- **Open Library ratings are sparse**: OL has ~400 community ratings for Dune vs. Goodreads' 1.6M. Use OL ratings only as a last resort.

- **Search page `&mdash;` entity**: The raw HTML uses `&mdash;` (not `—`) between rating value and count in search and author pages. The regex patterns above match the decoded text because Python's `re` operates on the decoded string after `http_get()` decodes UTF-8.

- **Book slug is optional**: `goodreads.com/book/show/44767458` (no slug) works identically to `goodreads.com/book/show/44767458-dune`. Redirects are transparent.