workbuddy_skills/browser-harness/agent-workspace/domain-skills/world-bank/scraping.md

# World Bank Open Data — Scraping & Data Extraction

`https://api.worldbank.org/v2` — free REST API for global development indicators. No API key, no auth, no browser needed. All data via `http_get`.

## Do this first

**Every response is a 2-element JSON array: `[metadata, data]`.** The metadata element is always at index 0 (pagination info); the data array is at index 1. This is the single biggest gotcha — `json.loads(raw)` gives you a list, not a dict.

```python
from helpers import http_get
import json

raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
d = json.loads(raw)
meta = d[0]   # {"page": 1, "pages": 2, "per_page": 50, "total": 66, ...}
rows = d[1]   # list of data records
```

Always append `?format=json` — default response is XML.

## Common workflows

### Single country, single indicator

```python
from helpers import http_get
import json

raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
d = json.loads(raw)
meta, rows = d[0], d[1]

for r in rows:
    if r["value"] is not None:   # recent years often have null values
        print(r["date"], r["value"])
# Confirmed output (2026-04-18):
# 2024 28750956130731.2
# 2023 27292170793214.4
# 2022 25604848907611.0
# ...
```

### Most recent N values (`mrv` param)

`mrv` (most recent values) skips null years and returns the N most recent non-provisional points.

```python
from helpers import http_get
import json

raw = http_get(
    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
    "?format=json&mrv=5"
)
d = json.loads(raw)
for r in d[1]:
    print(r["date"], r["value"])
# Confirmed output (2026-04-18):
# 2024 28750956130731.2
# 2023 27292170793214.4
# 2022 25604848907611.0
# 2021 23315080560000.0
# 2020 21060473613000.0
```

### Multiple countries, date range

Semicolon-delimit country codes in the URL path. Use `date=YYYY:YYYY` for a range.

```python
from helpers import http_get
import json

raw = http_get(
    "https://api.worldbank.org/v2/country/US;CN;GB/indicator/SP.POP.TOTL"
    "?format=json&date=2000:2023&per_page=100"
)
d = json.loads(raw)
meta, rows = d[0], d[1]
print(f"Total records: {meta['total']}, pages: {meta['pages']}")

for r in rows:
    print(r["country"]["value"], r["date"], r["value"])
# Confirmed: returns 8 records per page (50 default), date range honored exactly
# Countries: ['China', 'United States', 'United Kingdom']
# Dates: ['2000', '2001', ..., '2023']
```

### All countries, latest value only

Use `mrv=1` with `per_page=1000` to get all 266 countries in a single call.

```python
from helpers import http_get
import json

raw = http_get(
    "https://api.worldbank.org/v2/country/all/indicator/NY.GDP.PCAP.CD"
    "?format=json&mrv=1&per_page=1000"
)
d = json.loads(raw)
meta, rows = d[0], d[1]
print(f"Countries returned: {len(rows)}")  # 266 (includes aggregates)

# Filter out regional aggregates — they have no iso2Code or have aggregate ids
countries_only = [r for r in rows if len(r["country"]["id"]) == 2]
for r in sorted(countries_only, key=lambda x: -(x["value"] or 0))[:5]:
    print(r["country"]["value"], r["date"], f"${r['value']:,.0f}")
# Confirmed output (2026-04-18):
# Luxembourg 2024 $135,605
# Norway 2024 $105,056
# ...
```

### Full pagination (fetch all pages)

```python
from helpers import http_get
import json

def fetch_all_pages(base_url):
    """Fetch all pages of a World Bank API endpoint."""
    all_rows = []
    page = 1
    while True:
        url = f"{base_url}&page={page}" if "?" in base_url else f"{base_url}?page={page}"
        d = json.loads(http_get(url))
        meta, rows = d[0], d[1]
        all_rows.extend(rows)
        if page >= meta["pages"]:
            break
        page += 1
    return all_rows

# Example: all US GDP data (66 years, 2 pages)
rows = fetch_all_pages(
    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
    "?format=json&per_page=50"
)
print(f"Total rows: {len(rows)}")  # 66
non_null = [(r["date"], r["value"]) for r in rows if r["value"] is not None]
print(f"Non-null: {len(non_null)}, range: {non_null[-1][0]}–{non_null[0][0]}")
```

### Indicators list (discover available indicators)

```python
from helpers import http_get
import json

raw = http_get("https://api.worldbank.org/v2/indicator?format=json&per_page=50")
d = json.loads(raw)
meta = d[0]
print(f"Total indicators: {meta['total']}, pages: {meta['pages']}")
# Confirmed: 29,511 indicators across 591 pages

for ind in d[1][:3]:
    print(ind["id"], "-", ind["name"])
# 1.0.HCount.1.90usd - Poverty Headcount ($1.90 a day)
# ...
```

### Indicators by topic

```python
from helpers import http_get
import json

# Topic 3 = Economy & Growth
raw = http_get("https://api.worldbank.org/v2/topic/3/indicator?format=json&per_page=50")
d = json.loads(raw)
print(f"Economy & Growth indicators: {d[0]['total']}")  # 306

for ind in d[1][:5]:
    print(ind["id"], "-", ind["name"])
```

### Country metadata

```python
from helpers import http_get
import json

raw = http_get("https://api.worldbank.org/v2/country/US?format=json")
d = json.loads(raw)
c = d[1][0]
print(c["name"], c["capitalCity"], c["region"]["value"], c["incomeLevel"]["value"])
# United States  Washington D.C.  North America  High income

# Filter countries by income level
raw = http_get("https://api.worldbank.org/v2/country?format=json&incomeLevel=LIC&per_page=300")
d = json.loads(raw)
print(f"Low-income countries: {d[0]['total']}")  # 25
```

### Topics list

```python
from helpers import http_get
import json

raw = http_get("https://api.worldbank.org/v2/topics?format=json")
d = json.loads(raw)
for t in d[1]:
    print(t["id"], t["value"])
# 1  Agriculture & Rural Development
# 2  Aid Effectiveness
# 3  Economy & Growth
# ... (21 topics total)
```

### Parallel fetch for multiple indicators (ThreadPoolExecutor)

```python
from helpers import http_get
from concurrent.futures import ThreadPoolExecutor
import json

INDICATORS = {
    "NY.GDP.MKTP.CD": "GDP (current US$)",
    "SP.POP.TOTL": "Population",
    "NY.GDP.PCAP.CD": "GDP per capita",
}

def fetch_indicator(ind_id):
    url = (
        f"https://api.worldbank.org/v2/country/US/indicator/{ind_id}"
        f"?format=json&mrv=5"
    )
    d = json.loads(http_get(url))
    return ind_id, d[1]

with ThreadPoolExecutor(max_workers=3) as ex:
    results = dict(ex.map(lambda i: fetch_indicator(i), INDICATORS))

for ind_id, rows in results.items():
    latest = next((r for r in rows if r["value"] is not None), None)
    if latest:
        print(f"{INDICATORS[ind_id]}: {latest['date']} = {latest['value']:,.2f}")
```

## URL reference

### Base URL

```
https://api.worldbank.org/v2
```

HTTP redirects to HTTPS (302). Always use HTTPS directly.

### Endpoint patterns

| Endpoint | Description |
|---|---|
| `/country/{code}/indicator/{id}` | Single country + indicator time series |
| `/country/{c1};{c2};{c3}/indicator/{id}` | Multi-country (semicolon-delimited) |
| `/country/all/indicator/{id}` | All countries |
| `/country/{code}` | Country metadata |
| `/country` | All countries metadata (filterable) |
| `/indicator` | All indicators list |
| `/indicator/{id}` | Single indicator metadata |
| `/topic/{id}/indicator` | Indicators for a topic |
| `/topics` | All topics |

### Query parameters

| Parameter | Values | Notes |
|---|---|---|
| `format` | `json`, `xml` (default) | Always set `format=json` |
| `per_page` | integer, default 50, max 1000 | Higher is faster for bulk |
| `page` | integer, default 1 | For paginating results |
| `date` | `2020`, `2000:2023` | Single year or colon-separated range |
| `mrv` | integer | N most recent non-null values |
| `gapfill` | `Y` | Forward-fill nulls when used with `mrv` |
| `incomeLevel` | `LIC`, `MIC`, `HIC`, `LMC`, `UMC` | Filter countries by income |
| `region` | `EAS`, `ECS`, `LAC`, `MEA`, `NAC`, `SAS`, `SSF` | Filter countries by region |

### Common indicator IDs (confirmed working, 2026-04-18)

| ID | Name |
|---|---|
| `NY.GDP.MKTP.CD` | GDP (current US$) |
| `NY.GDP.PCAP.CD` | GDP per capita (current US$) |
| `SP.POP.TOTL` | Population, total |
| `SL.UEM.TOTL.ZS` | Unemployment (% of labor force) |
| `FP.CPI.TOTL.ZG` | Inflation, consumer prices (%) |
| `NE.EXP.GNFS.ZS` | Exports of goods and services (% of GDP) |
| `SP.DYN.LE00.IN` | Life expectancy at birth |
| `SE.ADT.LITR.ZS` | Literacy rate, adult total (%) |
| `EG.USE.PCAP.KG.OE` | Energy use per capita (kg of oil equiv.) |

Find more: `https://api.worldbank.org/v2/indicator?format=json&per_page=50&page=N`

### Country codes (ISO2)

Standard ISO 3166-1 alpha-2 codes: `US`, `CN`, `GB`, `DE`, `JP`, `IN`, `BR`, etc.
Special: `all` for all countries.

## Response structure

Every endpoint returns a 2-element array:

```json
[
  {
    "page": 1,
    "pages": 2,
    "per_page": 50,
    "total": 66,
    "sourceid": "2",
    "lastupdated": "2026-04-08"
  },
  [
    {
      "indicator": {"id": "NY.GDP.MKTP.CD", "value": "GDP (current US$)"},
      "country": {"id": "US", "value": "United States"},
      "countryiso3code": "USA",
      "date": "2024",
      "value": 28750956130731.2,
      "unit": "",
      "obs_status": "",
      "decimal": 0
    },
    ...
  ]
]
```

Country metadata endpoint returns same 2-element shape but with country objects (not indicator rows) at index 1.

## Gotchas

- **Response is always a 2-element array, not a dict.** `d = json.loads(raw)` gives a list. `d[0]` is pagination metadata, `d[1]` is the data list. Accessing `d["page"]` raises `TypeError`. This is the most common mistake.

- **`value` can be null.** Recent years (e.g. 2025) and data-sparse countries frequently have `null` values. Always check `if r["value"] is not None` before using. Use `mrv=N` to skip nulls automatically.

- **Always append `?format=json`.** The default response format is XML. Without `format=json`, you get an XML string that fails `json.loads`.

- **`all/indicator/{id}` includes regional aggregates.** The "all countries" endpoint returns 266 entries including aggregated regions like "Africa Eastern and Southern" (`id: "ZH"`). Filter to real countries with `len(r["country"]["id"]) == 2` (ISO2 codes are always 2 chars; aggregate codes are 2-3 chars but with non-standard values).

- **Semicolons in URL path, not query string.** Multi-country requests use `country/US;CN;GB/indicator/...` not `?countries=US,CN,GB`. Commas do not work.

- **HTTP 302 redirects HTTP to HTTPS.** Always use `https://` directly to avoid an extra round trip.

- **`per_page` in metadata is sometimes a string, sometimes an integer.** The API returns `"per_page": "50"` (string) for some endpoints and `"per_page": 50` (int) for others. Don't compare with `==` without casting: `int(meta["per_page"])`.

- **Invalid country codes return an error object, not a 2-element array.** A bad code gives `[{"message": [{"id": "120", "key": "Invalid value", ...}]}]` — a 1-element list with an error dict. Check `if isinstance(d[0], dict) and "message" in d[0]` before accessing `d[1]`.

- **`mrv` + `gapfill=Y` forward-fills the latest value into future years.** If 2024 is the latest data point and `mrv=3`, `gapfill=Y` returns 2025 (the current year) with the 2024 value copied in. Useful for "current" lookups, but the filled date is misleading.

- **No rate limit documented, but 3 req/s sustained is safe.** The API handles bursts (parallel ThreadPoolExecutor with `max_workers=3`) without issue. For crawling thousands of indicators, add `time.sleep(0.5)` between pages.

- **`date` range returns records newest-first.** Results within a date range are sorted descending by year. If you need ascending order, sort after fetching: `sorted(rows, key=lambda r: r["date"])`.

- **Indicator IDs are case-sensitive.** `ny.gdp.mktp.cd` returns an error; use the uppercase dot-separated form `NY.GDP.MKTP.CD`.