workbuddy_skills/browser-harness/agent-workspace/domain-skills/itch-io/scraping.md

# itch.io — Scraping & Data Extraction

Field-tested against itch.io on 2026-04-18. All code blocks validated with live requests.

---

## TL;DR — fastest approaches by task

| Task | Method | Notes |
|---|---|---|
| Browse listings (36/page) | `http_get` HTML | Works, no key, no bot block |
| Game detail (name, price, rating) | `http_get` + JSON-LD | `<script type="application/ld+json">` Product block |
| Info table (tags, genre, status) | `http_get` + regex on `game_info_panel_widget` | Always present |
| Top N games from any category | RSS `.xml` feed | Cleaner than HTML for bulk |
| API (key endpoints) | `http_get` + key in path | Free keys at itch.io/docs/api |
| Download/purchase counts | Not public | Owners only via dashboard |

`http_get` works on all itch.io game and browse pages with no extra headers needed.
No Cloudflare, no JS challenge, no CAPTCHA on standard game/browse routes.

---

## Approach 1 (Fastest for listings): RSS feeds — 36 games per call, clean XML

Every browse URL has an `.xml` RSS variant. Returns price, pub/update dates, platforms, thumbnail. No HTML parsing.

```python
import re
from helpers import http_get

def parse_rss(url):
    """
    Parse any itch.io RSS listing feed.
    url examples:
      https://itch.io/games/top-rated.xml
      https://itch.io/games/newest.xml
      https://itch.io/games/featured.xml
      https://itch.io/games/on-sale.xml
      https://itch.io/games/free.xml
      https://itch.io/games/tag-puzzle.xml    # any tag slug works
      https://itch.io/games/top-rated.xml?page=2
    """
    xml = http_get(url)
    items = []
    for m in re.finditer(r'<item>(.*?)</item>', xml, re.DOTALL):
        ix = m.group(1)
        def get(tag, s=ix):
            tm = re.search(rf'<{tag}>(.*?)</{tag}>', s, re.DOTALL)
            return tm.group(1).strip() if tm else None
        items.append({
            'url':         get('guid'),
            'title':       get('plainTitle'),          # clean title, no [tags]
            'price':       get('price'),               # "$0.00", "$7.99", etc.
            'currency':    get('currency'),            # "USD"
            'pub_date':    get('pubDate'),
            'update_date': get('updateDate'),
            'image':       get('imageurl'),            # 315x250 thumbnail
            'platforms': {
                k: get(k) == 'yes'
                for k in ['windows', 'osx', 'linux', 'android', 'html']
                if get(k) is not None
            },
        })
    return items

# Confirmed output:
items = parse_rss("https://itch.io/games/top-rated.xml")
# items[0] -> {
#   'url':         'https://gbpatch.itch.io/our-life',
#   'title':       'Our Life: Beginnings & Always',
#   'price':       '$0.00',
#   'currency':    'USD',
#   'pub_date':    'Fri, 07 Jun 2019 23:47:57 GMT',
#   'update_date': 'Sun, 22 May 2022 15:48:27 GMT',
#   'image':       'https://img.itch.zone/aW1nLzcwMTIxNDMucG5n/315x250%23c/BalGQb.png',
#   'platforms':   {'windows': True, 'osx': True, 'linux': True, 'android': True},
# }
```

**RSS limitations:** no rating score or count. Use HTML scraping (Approach 2) when you need ratings.

---

## Approach 2: HTML listings — ratings, genre, price, 36 games per page

```python
import re
from helpers import http_get

def parse_game_cards(html):
    """
    Extract all game cards from any itch.io browse/listing/search/profile HTML page.
    Works on:
      https://itch.io/games/top-rated
      https://itch.io/games/newest
      https://itch.io/games/featured
      https://itch.io/games/on-sale
      https://itch.io/games/free
      https://itch.io/games/tag-puzzle      (genre/tag path)
      https://itch.io/search?q=platformer   (search — 54 cards per page)
      https://<author>.itch.io              (author profile)
    All accept ?page=N for pagination.
    """
    games = []
    for m in re.finditer(r'data-game_id="(\d+)"', html):
        game_id = m.group(1)
        start = m.start()
        chunk = html[start:start + 3000]

        # Title + URL — attribute order differs between page 1 and pages 2+
        title_m = re.search(
            r'class="title game_link"[^>]*href="([^"]+)"[^>]*>([^<]+)</a>', chunk
        )
        if not title_m:
            title_m = re.search(
                r'href="([^"]+)"[^>]*class="title game_link"[^>]*>([^<]+)</a>', chunk
            )

        rating_m  = re.search(
            r'data-tooltip="([\d.]+) average rating from ([\d,]+) total ratings"', chunk
        )
        genre_m   = re.search(r'class="game_genre">([^<]+)</div>', chunk)
        price_m   = re.search(r'class="price_value">([^<]+)</div>', chunk)
        desc_m    = re.search(r'class="game_text" title="([^"]+)"', chunk)
        img_m     = re.search(r'data-lazy_src="([^"]+)"', chunk)
        platforms = re.findall(r'title="Download for ([^"]+)"', chunk)

        games.append({
            'id':           game_id,
            'url':          title_m.group(1) if title_m else None,
            'title':        title_m.group(2).strip() if title_m else None,
            'rating':       float(rating_m.group(1)) if rating_m else None,
            'rating_count': int(rating_m.group(2).replace(',', '')) if rating_m else None,
            'genre':        genre_m.group(1) if genre_m else None,
            'price':        price_m.group(1) if price_m else 'Free',
            'description':  desc_m.group(1) if desc_m else None,
            'thumbnail':    img_m.group(1) if img_m else None,
            'platforms':    platforms,      # ['Windows', 'macOS', 'Linux', 'Android']
        })
    return games

# Usage:
html = http_get("https://itch.io/games/top-rated")
games = parse_game_cards(html)
# games[0] -> {
#   'id': '434554', 'url': 'https://gbpatch.itch.io/our-life',
#   'title': 'Our Life: Beginnings & Always',
#   'rating': 4.94, 'rating_count': 7191,
#   'genre': 'Visual Novel', 'price': 'Free',
#   'platforms': ['Windows', 'Linux', 'macOS', 'Android'],
# }

# Paid game example:
html = http_get("https://itch.io/games/top-rated?page=5")
games = parse_game_cards(html)
# Returns games where price_m captures '$7.99' when present
```

### CSS selector reference (for browser/JS use)

```
.game_cell                        — one card per game
.game_cell[data-game_id]          — get game ID from attribute
.game_cell .title.game_link       — title text + href
.game_cell .game_rating           — rating container
.game_cell .game_rating[data-tooltip]  — "4.94 average rating from 7,191 total ratings"
.game_cell .star_fill             — inline style width: NN% (rating as percentage of 5)
.game_cell .rating_count          — "(7,191)"
.game_cell .game_genre            — genre text
.game_cell .price_tag .price_value — price e.g. "$7.99" (absent = Free)
.game_cell .game_text             — one-line description (also in title attr)
.game_cell .game_author a         — author name + href
.game_cell img.lazy_loaded        — thumbnail (src in data-lazy_src before JS runs)
```

**Gotcha — attribute order flips on page >= 2.** Page 1 uses `class="..." data-game_id="..."`, page 2+ uses `data-game_id="..." class="..."`. The regex above handles both. If you use a CSS selector engine, `[data-game_id]` is unambiguous.

**Gotcha — ratings absent on some listing types.** The tag/genre browse pages (e.g. `/games/tag-puzzle`) sometimes omit the rating tooltip on the card even when the game has ratings. Fetch the detail page for the authoritative rating.

---

## Approach 3: Game detail page — JSON-LD Product schema

The cleanest source for individual game data. All confirmed fields:

```python
import json, re
from helpers import http_get

def extract_game_detail(url):
    """
    Fetch full metadata for a single itch.io game.
    url format: https://<author>.itch.io/<game-slug>
    """
    html = http_get(url)

    # --- JSON-LD (always present, covers name/description/price/rating) ---
    ld_product = None
    for block in re.findall(
        r'<script[^>]*type="application/ld\+json"[^>]*>(.*?)</script>',
        html, re.DOTALL
    ):
        ld = json.loads(block.strip())
        if ld.get('@type') == 'Product':
            ld_product = ld
            break

    # --- Info panel table (Status, Platforms, Genre, Tags, Author, etc.) ---
    info = {}
    panel_m = re.search(
        r'class="game_info_panel_widget[^"]*"[^>]*><table>(.*?)</table>',
        html, re.DOTALL
    )
    if panel_m:
        for row in re.finditer(
            r'<tr><td>([^<]+)</td><td>(.*?)</td></tr>',
            panel_m.group(1), re.DOTALL
        ):
            key = row.group(1).strip()
            val = re.sub(r'<[^>]+>', '', row.group(2)).strip()
            # Multi-value fields become lists (Tags, Platforms, Genre, Links)
            info[key] = [v.strip() for v in val.split(',')] if ',' in val else val

    # --- Cover image ---
    cover_m = re.search(r'<meta property="og:image" content="([^"]+)"', html)

    offers = (ld_product or {}).get('offers', {})
    agg    = (ld_product or {}).get('aggregateRating', {})

    return {
        'url':          url,
        'name':         (ld_product or {}).get('name'),
        'description':  (ld_product or {}).get('description'),
        'price':        offers.get('price'),          # "0.00" for free, "7.99" for paid
        'currency':     offers.get('priceCurrency'),  # "USD"
        'rating':       agg.get('ratingValue'),       # "4.9" string
        'rating_count': agg.get('ratingCount'),       # int
        'cover':        cover_m.group(1) if cover_m else None,
        'info':         info,
    }

# Free game:
r = extract_game_detail("https://gbpatch.itch.io/our-life")
# {
#   'name': 'Our Life: Beginnings & Always',
#   'description': 'Grow from childhood to adulthood with the lonely boy next door...',
#   'price': None, 'currency': None,   <- no 'offers' block for free games
#   'rating': '4.9', 'rating_count': 7191,
#   'cover': 'https://img.itch.zone/aW1hZ2Uv.../347x500/7HqrvV.jpg',
#   'info': {
#     'Status':    'Released',
#     'Platforms': ['Windows', 'macOS', 'Linux', 'Android'],
#     'Rating':    'Rated 4.9 out of 5 stars(7,191 total ratings)',
#     'Author':    'GBPatch',
#     'Genre':     ['Visual Novel', 'Interactive Fiction'],
#     'Tags':      ['Amare', 'Comedy', 'Dating Sim', 'Gay', 'LGBT', ...],
#     'Links':     'Steam',
#   }
# }

# Paid game:
r = extract_game_detail("https://adamgryu.itch.io/a-short-hike")
# {
#   'name': 'A Short Hike',
#   'price': '7.99', 'currency': 'USD',
#   'rating': '4.9', 'rating_count': 4307,
#   'info': {
#     'Status':          'Released',
#     'Platforms':       ['Windows', 'macOS', 'Linux'],
#     'Release date':    'Jul 30, 2019',
#     'Genre':           ['Adventure', 'Platformer'],
#     'Made with':       'Unity',
#     'Tags':            ['3D', 'Atmospheric', 'Cute', 'Relaxing', 'Short', ...],
#     'Average session': 'About an hour',
#     'Languages':       ['English', 'Spanish; Latin America', 'French', ...],
#     'Inputs':          ['Keyboard', 'Mouse', 'Xbox controller', ...],
#     'Accessibility':   ['Subtitles', 'Configurable controls'],
#     'Links':           ['Steam', 'Homepage', 'Soundtrack', 'Twitter/X'],
#   }
# }
```

**JSON-LD available fields:**

| Field | Free game | Paid game |
|---|---|---|
| `@type` | `Product` | `Product` |
| `name` | yes | yes |
| `description` | yes | yes |
| `aggregateRating.ratingValue` | yes | yes |
| `aggregateRating.ratingCount` | yes | yes |
| `offers.price` | absent | yes ("7.99") |
| `offers.priceCurrency` | absent | yes ("USD") |
| `offers.seller.name` | absent | yes (author name) |
| `offers.seller.url` | absent | yes (author profile URL) |

---

## Pagination

Browse pages: `?page=N`. Detect end of results by HTTP 404 (page too high) or absent `<link rel="next">`.

```python
import re
from helpers import http_get

def paginate_listing(base_url, max_pages=10):
    """
    Scrape multiple pages from any itch.io browse URL.
    base_url: https://itch.io/games/top-rated  (no ?page= suffix)
    Returns flat list of game dicts.
    Stops when HTTP 404 or no <link rel="next"> found.
    """
    all_games = []
    page = 1
    while page <= max_pages:
        url = base_url if page == 1 else f"{base_url}?page={page}"
        try:
            html = http_get(url)
        except Exception:
            break   # 404 = past last page
        all_games.extend(parse_game_cards(html))
        if not re.search(r'<link[^>]+rel="next"[^>]*/>', html):
            break
        page += 1
    return all_games

# Confirmed: page 1 has <link href="?page=2" rel="next"/>
#            page 2 has <link rel="prev" href="/games/top-rated"/> and <link rel="next" href="?page=3"/>
#            past last page returns HTTP 404
# top-rated has at least 200 pages (each 36 games); page 300+ -> 404
```

---

## Browse URL patterns

All confirmed working via `http_get`:

```python
BASE = "https://itch.io/games"

# Sort orders
f"{BASE}/top-rated"      # all-time top rated (rated by community, 0–5 stars)
f"{BASE}/newest"         # most recently published
f"{BASE}/featured"       # itch.io staff picks
f"{BASE}/on-sale"        # discounted games
f"{BASE}/free"           # free games only

# Genre/tag paths (append .xml for RSS)
f"{BASE}/tag-puzzle"     # tag slug — prefix with 'tag-'
f"{BASE}/genre-action"   # genre — prefix with 'genre-' (less common)

# Combine: tag + sort via separate pages (no combined URL that survives http_get)
# Note: https://itch.io/games/top-rated/tag-puzzle -> HTTP 403
# Note: ?tag= query param does NOT filter server-side (returns same games)

# Pagination
f"{BASE}/top-rated?page=2"
f"{BASE}/tag-puzzle?page=3"

# RSS equivalents (36 items, no pagination needed for small sets)
f"{BASE}/top-rated.xml"
f"{BASE}/tag-puzzle.xml"
f"{BASE}/tag-puzzle.xml?page=2"

# Search (54 results/page, no server-side pagination beyond page 1 via http_get)
"https://itch.io/search?q=platformer"

# Author profile
"https://<author-slug>.itch.io"
```

---

## API (requires key)

itch.io has an official REST API. A free key is issued per-account with no rate limit published.
Get one at: `https://itch.io/user/settings/api-keys`

Base URL: `https://itch.io/api/1/<key>/`

```python
import json
from helpers import http_get

ITCH_KEY = "your_api_key_here"   # from https://itch.io/user/settings/api-keys

def api(path):
    return json.loads(http_get(f"https://itch.io/api/1/{ITCH_KEY}/{path}"))

# Authenticated user info
api("me")
# -> {"user": {"id": ..., "username": "...", "url": "...", "display_name": "...", ...}}

# Games owned by authenticated user
api("my-games")
# -> {"games": [{"id": ..., "title": "...", "url": "...", "created_at": "...",
#                "published": true/false, "min_price": 0, ...}, ...]}

# Download keys for a game (owner only)
api("game/434554/download_keys")

# Credentials (for authenticated purchases)
api("game/434554/credentials")
```

**Error structure:** invalid/missing key returns `{"errors": ["invalid key"]}` with HTTP 200.
Non-existent endpoints return HTTP 404.

**No unauthenticated game lookup API.** `https://itch.io/api/1/x/games` -> HTTP 404.
Use HTML scraping or RSS for unauthenticated game data.

---

## Gotchas

1. **Attribute order flips page 1 vs 2+.** On page 1, game cards use `class="game_cell ..." data-game_id="..."`. On pages 2+, the order is `data-game_id="..." class="game_cell ..."`. Always match `data-game_id` independently of class ordering.

2. **Ratings absent on tag/genre listing pages.** The `data-tooltip` with rating is often missing from card HTML on `/games/tag-*` pages even though the game has ratings. Fetch the detail page for `aggregateRating` via JSON-LD.

3. **`price_value` absent = Free.** Paid games have `<div class="price_tag meta_tag" title="Pay $7.99 or more..."><div class="price_value">$7.99</div></div>`. Free games have no such element. Default to `'Free'` when absent.

4. **Free-game JSON-LD has no `offers` block.** Only paid games include the `offers` object. For free games, use absence of `offers` as the signal, not presence of `price: 0`.

5. **`/games/top-rated/tag-puzzle` returns HTTP 403.** Cannot combine sort + tag in a path. Use separate `/games/tag-puzzle` (top-rated is the default sort anyway).

6. **`?tag=` query param is ignored server-side.** `https://itch.io/games/top-rated?tag=puzzle` returns the same games as `?top-rated`. Use `/games/tag-puzzle` path instead.

7. **Download/purchase counts are not public.** No count field appears anywhere in the public HTML, JSON-LD, RSS, or unauthenticated API. Game owners see their stats in the dashboard only.

8. **Search beyond page 1 is AJAX-only.** `https://itch.io/search?q=X&page=2` via `http_get` returns the same 54 results as page 1. To get more search results use the browser and scroll/click "load more".

9. **RSS is capped at 36 items per page.** Paginate with `?page=N`. Very high page numbers (300+) return HTTP 404 on browse pages.

10. **Unicode zero-width space in some titles.** `\u200b` (zero-width space) appears at the start of certain titles (e.g. "​Our Life: Beginnings & Always"). Strip with `.replace('\u200b', '').strip()` or `.strip()` alone won't remove it — use `title.replace('\u200b', '').strip()`.