339 lines
15 KiB
Markdown
339 lines
15 KiB
Markdown
|
# NASA APIs — Scraping & Data Extraction
|
||
|
|
|
||
|
|
`https://api.nasa.gov` — open NASA data APIs. **Never use the browser.** All endpoints return JSON via `http_get`. DEMO_KEY works for low-volume use; register for a free personal key at https://api.nasa.gov/ to raise limits.
|
||
|
|
|
||
|
|
## Do this first
|
||
|
|
|
||
|
|
**All `api.nasa.gov` endpoints share the same rate-limit pool under DEMO_KEY. EPIC and Exoplanet Archive are on separate domains with no rate limit.**
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Simplest call: today's Astronomy Picture of the Day
|
||
|
|
apod = json.loads(http_get("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY"))
|
||
|
|
print(apod['date'], apod['title'], apod['media_type'])
|
||
|
|
# Confirmed output (2026-04-18): 2026-04-18 PanSTARRS and Planets image
|
||
|
|
```
|
||
|
|
|
||
|
|
Use DEMO_KEY for exploration. Switch to a personal key for any bulk work — DEMO_KEY hits its limit at ~10 req/hour/IP (daily budget around 50; `retry-after` header will show ~22 hours when exhausted).
|
||
|
|
|
||
|
|
## Rate limits
|
||
|
|
|
||
|
|
| Key type | Limit | Resets |
|
||
|
|
|---|---|---|
|
||
|
|
| `DEMO_KEY` | 10 req/hour, ~50/day per IP | Hourly window; daily hard stop with `retry-after` ~22h |
|
||
|
|
| Personal key (free) | 1,000 req/hour | Hourly window |
|
||
|
|
|
||
|
|
Rate limit headers on every `api.nasa.gov` response:
|
||
|
|
- `X-Ratelimit-Limit` — your current window limit (e.g. `10`)
|
||
|
|
- `X-Ratelimit-Remaining` — calls left this window
|
||
|
|
- `Retry-After` — seconds until next window (only on 429)
|
||
|
|
|
||
|
|
**EPIC (`epic.gsfc.nasa.gov`) and Exoplanet Archive (`exoplanetarchive.ipac.caltech.edu`) share no rate-limit pool with `api.nasa.gov`.**
|
||
|
|
|
||
|
|
## Common workflows
|
||
|
|
|
||
|
|
### APOD — single day
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
apod = json.loads(http_get("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY"))
|
||
|
|
print(apod['date']) # '2026-04-18'
|
||
|
|
print(apod['title']) # 'PanSTARRS and Planets'
|
||
|
|
print(apod['media_type']) # 'image' or 'video'
|
||
|
|
print(apod['url']) # full-res or YouTube embed URL
|
||
|
|
print(apod['hdurl']) # HD image URL (absent when media_type='video')
|
||
|
|
print(apod.get('copyright')) # None if public domain
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# url: https://apod.nasa.gov/apod/image/2604/PanstarrsPlanetsPerrotLab1024.jpg
|
||
|
|
# hdurl: https://apod.nasa.gov/apod/image/2604/PanstarrsPlanetsPerrot.jpg
|
||
|
|
# copyright: Luc Perrot
|
||
|
|
```
|
||
|
|
|
||
|
|
### APOD — date range (array response)
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
apods = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/planetary/apod"
|
||
|
|
"?start_date=2024-01-01&end_date=2024-01-07&api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
# Returns a list of 7 dicts — same schema as single-day response
|
||
|
|
for a in apods:
|
||
|
|
print(a['date'], a['media_type'], a['title'][:50])
|
||
|
|
# Confirmed output (7 items):
|
||
|
|
# 2024-01-01 image NGC 1232: A Grand Design Spiral Galaxy
|
||
|
|
# 2024-01-02 image Rocket Transits Rippling Moon
|
||
|
|
# 2024-01-03 image A SAR Arc from New Zealand
|
||
|
|
# 2024-01-04 image Zeta Oph: Runaway Star
|
||
|
|
# 2024-01-05 image Trapezium: At the Heart of Orion
|
||
|
|
# 2024-01-06 video The Snows of Churyumov-Gerasimenko
|
||
|
|
# 2024-01-07 image The Cat's Eye Nebula in Optical and X-ray
|
||
|
|
```
|
||
|
|
|
||
|
|
Optional params: `date=YYYY-MM-DD` (specific day), `count=N` (N random entries), `thumbs=true` (include `thumbnail_url` for video entries).
|
||
|
|
|
||
|
|
### APOD — random sample
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
apods = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/planetary/apod?count=5&api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
for a in apods:
|
||
|
|
print(a['date'], a['title'][:40])
|
||
|
|
# Returns 5 random APOD entries — dates can be any day since 1995-06-16
|
||
|
|
```
|
||
|
|
|
||
|
|
### NEO — Near Earth Objects feed
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/neo/rest/v1/feed"
|
||
|
|
"?start_date=2024-01-01&end_date=2024-01-02&api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
print(data['element_count']) # 32 (total NEOs across both days)
|
||
|
|
neos = data['near_earth_objects'] # dict keyed by date string
|
||
|
|
for date, objects in sorted(neos.items()):
|
||
|
|
for neo in objects:
|
||
|
|
ca = neo['close_approach_data'][0]
|
||
|
|
print(
|
||
|
|
neo['name'],
|
||
|
|
'hazardous:', neo['is_potentially_hazardous_asteroid'],
|
||
|
|
'miss km:', ca['miss_distance']['kilometers'][:12],
|
||
|
|
'vel kph:', ca['relative_velocity']['kilometers_per_hour'][:10]
|
||
|
|
)
|
||
|
|
# Confirmed output (2 days, 32 total NEOs):
|
||
|
|
# 415949 (2001 XY10) hazardous: False miss km: 50452409.34 vel kph: 57205.8951
|
||
|
|
# (22+ more objects per day)
|
||
|
|
```
|
||
|
|
|
||
|
|
NEO object fields:
|
||
|
|
- `id`, `name`, `nasa_jpl_url` — identity
|
||
|
|
- `estimated_diameter` — dict with `kilometers`, `meters`, `miles`, `feet` sub-dicts, each with `min`/`max`
|
||
|
|
- `is_potentially_hazardous_asteroid` — bool
|
||
|
|
- `close_approach_data[0]` — `close_approach_date`, `miss_distance` (au/lunar/km/mi), `relative_velocity` (km/s, km/h, mph), `orbiting_body`
|
||
|
|
|
||
|
|
Date range is capped at **7 days per request**. For longer ranges, paginate with `start_date` / `end_date` in 7-day steps. `links.next` in the response gives the next 7-day window URL.
|
||
|
|
|
||
|
|
### NEO — single asteroid lookup
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Asteroid ID comes from the feed's `id` field
|
||
|
|
neo = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/neo/rest/v1/neo/2415949?api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
print(neo['name'])
|
||
|
|
print(neo['orbital_data']['orbit_class']['orbit_class_description'])
|
||
|
|
# Full orbital history + all close approaches are in `close_approach_data` (long list)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Mars Rover photos — Curiosity by sol
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# sol = Martian solar day since landing
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos"
|
||
|
|
"?sol=1000&api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
photos = data['photos']
|
||
|
|
print(f"Photos on sol 1000: {len(photos)}")
|
||
|
|
p = photos[0]
|
||
|
|
print(p['earth_date']) # '2015-05-30'
|
||
|
|
print(p['img_src']) # direct JPEG URL
|
||
|
|
print(p['camera']['name']) # 'FHAZ'
|
||
|
|
print(p['camera']['full_name']) # 'Front Hazard Avoidance Camera'
|
||
|
|
print(p['rover']['name']) # 'Curiosity'
|
||
|
|
print(p['rover']['status']) # 'active'
|
||
|
|
print(p['rover']['max_sol']) # highest sol with photos
|
||
|
|
|
||
|
|
# Filter by camera
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos"
|
||
|
|
"?sol=1000&camera=navcam&api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
```
|
||
|
|
|
||
|
|
Available cameras for Curiosity: `fhaz`, `rhaz`, `mast`, `chemcam`, `mahli`, `mardi`, `navcam`. Other rovers: `opportunity`, `spirit`, `perseverance`.
|
||
|
|
|
||
|
|
Use `latest_photos` to get the most recent available:
|
||
|
|
```python
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/latest_photos"
|
||
|
|
"?api_key=DEMO_KEY"
|
||
|
|
))
|
||
|
|
photos = data['latest_photos']
|
||
|
|
```
|
||
|
|
|
||
|
|
Add `&page=N` for pagination (25 photos/page by default).
|
||
|
|
|
||
|
|
### EPIC — Earth Polychromatic Imaging Camera
|
||
|
|
|
||
|
|
EPIC images are served from `epic.gsfc.nasa.gov` — **no `api_key` required, no rate limit.**
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Latest available images (natural color)
|
||
|
|
images = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural"))
|
||
|
|
print(f"Latest batch: {len(images)} images") # Confirmed: 4 images on 2026-04-18
|
||
|
|
|
||
|
|
img = images[0]
|
||
|
|
print(img['identifier']) # '20260416162050'
|
||
|
|
print(img['image']) # 'epic_1b_20260416162050'
|
||
|
|
print(img['date']) # '2026-04-16 16:16:01'
|
||
|
|
print(img['centroid_coordinates']) # {'lat': 13.25, 'lon': -75.59}
|
||
|
|
|
||
|
|
# Construct PNG URL from image name + date
|
||
|
|
date_str = img['date'].split(' ')[0] # '2026-04-16'
|
||
|
|
year, month, day = date_str.split('-')
|
||
|
|
png_url = f"https://epic.gsfc.nasa.gov/archive/natural/{year}/{month}/{day}/png/{img['image']}.png"
|
||
|
|
jpg_thumb = f"https://epic.gsfc.nasa.gov/archive/natural/{year}/{month}/{day}/thumbs/{img['image']}.jpg"
|
||
|
|
print(png_url)
|
||
|
|
# Confirmed: https://epic.gsfc.nasa.gov/archive/natural/2026/04/16/png/epic_1b_20260416162050.png
|
||
|
|
```
|
||
|
|
|
||
|
|
```python
|
||
|
|
# Images for a specific date
|
||
|
|
images = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural/date/2024-01-15"))
|
||
|
|
print(len(images)) # 14 images on 2024-01-15
|
||
|
|
|
||
|
|
# Enhanced (color-corrected) images — same API, different path
|
||
|
|
enhanced = json.loads(http_get("https://epic.gsfc.nasa.gov/api/enhanced/date/2024-01-15"))
|
||
|
|
# Enhanced image URL pattern uses 'enhanced' and 'epic_RGB_' prefix:
|
||
|
|
img = enhanced[0]
|
||
|
|
date_str = img['date'].split(' ')[0]
|
||
|
|
year, month, day = date_str.split('-')
|
||
|
|
url = f"https://epic.gsfc.nasa.gov/archive/enhanced/{year}/{month}/{day}/png/{img['image']}.png"
|
||
|
|
# e.g. .../archive/enhanced/2024/01/15/png/epic_RGB_20240115005515.png
|
||
|
|
|
||
|
|
# All available dates
|
||
|
|
all_dates = json.loads(http_get("https://epic.gsfc.nasa.gov/api/natural/all"))
|
||
|
|
print(f"Available dates: {len(all_dates)}") # 3477 dates (2015-06-13 to present)
|
||
|
|
print(all_dates[0]) # {'date': '2026-04-16'} (newest first)
|
||
|
|
print(all_dates[-1]) # {'date': '2015-06-13'} (oldest)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Exoplanet Archive — TAP/ADQL queries
|
||
|
|
|
||
|
|
No API key or rate limit. SQL-like ADQL queries over the full archive.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Short-period planets with known radii
|
||
|
|
planets = json.loads(http_get(
|
||
|
|
"https://exoplanetarchive.ipac.caltech.edu/TAP/sync"
|
||
|
|
"?query=select+pl_name,hostname,pl_orbper+from+ps+where+pl_orbper+%3C+10"
|
||
|
|
"&format=json"
|
||
|
|
))
|
||
|
|
print(f"Rows: {len(planets)}") # 17675 (table 'ps' includes duplicate measurements)
|
||
|
|
print(planets[0])
|
||
|
|
# {'pl_name': 'GJ 1214 b', 'hostname': 'GJ 1214', 'pl_orbper': 1.58040482}
|
||
|
|
```
|
||
|
|
|
||
|
|
```python
|
||
|
|
# Use 'pscomppars' for one row per planet (composite best-estimate params)
|
||
|
|
planets = json.loads(http_get(
|
||
|
|
"https://exoplanetarchive.ipac.caltech.edu/TAP/sync"
|
||
|
|
"?query=select+pl_name,hostname,disc_year,discoverymethod,pl_orbper,pl_rade,pl_masse,pl_eqt,sy_dist"
|
||
|
|
"+from+pscomppars+where+disc_year+%3E+2020+and+pl_rade+is+not+null"
|
||
|
|
"+order+by+disc_year+desc"
|
||
|
|
"&format=json&maxrec=5"
|
||
|
|
))
|
||
|
|
for p in planets:
|
||
|
|
print(p['pl_name'], p['disc_year'], p['discoverymethod'], f"r={p['pl_rade']}Re")
|
||
|
|
# Confirmed output:
|
||
|
|
# KMT-2024-BLG-1870L b 2026 Microlensing r=13.8Re
|
||
|
|
# LHS 1903 b 2026 Transit r=1.382Re
|
||
|
|
# TOI-375 d 2026 Radial Velocity r=13.6Re
|
||
|
|
```
|
||
|
|
|
||
|
|
Key tables:
|
||
|
|
- `ps` — all measurements per planet (multiple rows per planet, all sources)
|
||
|
|
- `pscomppars` — one row per confirmed planet (best composite parameters)
|
||
|
|
|
||
|
|
Key columns: `pl_name`, `hostname`, `disc_year`, `discoverymethod`, `pl_orbper` (orbital period, days), `pl_rade` (radius in Earth radii), `pl_masse` (mass in Earth masses), `pl_eqt` (equilibrium temp K), `sy_dist` (distance in parsec).
|
||
|
|
|
||
|
|
URL-encode operators: `<` = `%3C`, `>` = `%3E`, spaces = `+`.
|
||
|
|
|
||
|
|
## URL reference
|
||
|
|
|
||
|
|
### api.nasa.gov endpoints
|
||
|
|
|
||
|
|
| Endpoint | URL pattern |
|
||
|
|
|---|---|
|
||
|
|
| APOD today | `https://api.nasa.gov/planetary/apod?api_key=KEY` |
|
||
|
|
| APOD by date | `...&date=YYYY-MM-DD` |
|
||
|
|
| APOD range | `...&start_date=YYYY-MM-DD&end_date=YYYY-MM-DD` |
|
||
|
|
| APOD random N | `...&count=N` |
|
||
|
|
| NEO feed | `https://api.nasa.gov/neo/rest/v1/feed?start_date=...&end_date=...&api_key=KEY` |
|
||
|
|
| NEO by ID | `https://api.nasa.gov/neo/rest/v1/neo/{id}?api_key=KEY` |
|
||
|
|
| Mars photos by sol | `https://api.nasa.gov/mars-photos/api/v1/rovers/{rover}/photos?sol=N&api_key=KEY` |
|
||
|
|
| Mars photos by date | `...?earth_date=YYYY-MM-DD&api_key=KEY` |
|
||
|
|
| Mars latest | `https://api.nasa.gov/mars-photos/api/v1/rovers/{rover}/latest_photos?api_key=KEY` |
|
||
|
|
|
||
|
|
### EPIC (epic.gsfc.nasa.gov — no key, no rate limit)
|
||
|
|
|
||
|
|
| Endpoint | URL |
|
||
|
|
|---|---|
|
||
|
|
| Latest natural images | `https://epic.gsfc.nasa.gov/api/natural` |
|
||
|
|
| Natural by date | `https://epic.gsfc.nasa.gov/api/natural/date/YYYY-MM-DD` |
|
||
|
|
| Enhanced latest | `https://epic.gsfc.nasa.gov/api/enhanced` |
|
||
|
|
| Enhanced by date | `https://epic.gsfc.nasa.gov/api/enhanced/date/YYYY-MM-DD` |
|
||
|
|
| All available dates | `https://epic.gsfc.nasa.gov/api/natural/all` |
|
||
|
|
| PNG image | `https://epic.gsfc.nasa.gov/archive/natural/YYYY/MM/DD/png/{image}.png` |
|
||
|
|
| Thumbnail (JPEG) | `https://epic.gsfc.nasa.gov/archive/natural/YYYY/MM/DD/thumbs/{image}.jpg` |
|
||
|
|
| Enhanced PNG | `https://epic.gsfc.nasa.gov/archive/enhanced/YYYY/MM/DD/png/{image}.png` |
|
||
|
|
|
||
|
|
### Exoplanet Archive (no key, no rate limit)
|
||
|
|
|
||
|
|
```
|
||
|
|
https://exoplanetarchive.ipac.caltech.edu/TAP/sync?query=<ADQL>&format=json&maxrec=<N>
|
||
|
|
```
|
||
|
|
|
||
|
|
## Gotchas
|
||
|
|
|
||
|
|
- **DEMO_KEY limit is effectively 10/hour per IP, not 30** — The `X-Ratelimit-Limit` header shows `10` in practice. When the daily budget (~50 req) is exhausted, the `retry-after` header is set to ~80,000 seconds (about 22 hours). Register a free personal key at https://api.nasa.gov/ to get 1,000/hour.
|
||
|
|
|
||
|
|
- **All `api.nasa.gov` paths share one rate-limit pool** — APOD, NEO, Mars Rover, and all other `api.nasa.gov` paths draw from the same DEMO_KEY bucket. Calling any one of them depletes the limit for all others.
|
||
|
|
|
||
|
|
- **EPIC and Exoplanet Archive are fully free** — `epic.gsfc.nasa.gov` returns no rate-limit headers and is not throttled. `exoplanetarchive.ipac.caltech.edu/TAP/sync` is similarly unrestricted. Use these freely without fear of exhausting DEMO_KEY.
|
||
|
|
|
||
|
|
- **NEO date range max is 7 days** — Requests spanning more than 7 days return HTTP 400. Paginate with 7-day windows and use `links.next` from the response to get the next URL.
|
||
|
|
|
||
|
|
- **APOD earliest date is 1995-06-16** — Requesting `date` before `1995-06-16` returns HTTP 400 with an error message. No upper date bound other than today.
|
||
|
|
|
||
|
|
- **APOD `hdurl` is absent for video entries** — When `media_type` is `video`, the response has `url` (a YouTube embed URL) but no `hdurl`. Always check `media_type` before accessing `hdurl`.
|
||
|
|
|
||
|
|
- **Mars Rover `sol` vs `earth_date`** — Both are valid filter params. `sol` is the Martian solar day since rover landing. `earth_date` uses `YYYY-MM-DD`. You cannot mix them in one request.
|
||
|
|
|
||
|
|
- **Mars Rover pagination defaults to 25 photos/page** — Large sols (Curiosity sol 1000 has many photos) require `&page=2`, `&page=3`, etc. There is no total count in the response; keep paginating until you get an empty `photos` list.
|
||
|
|
|
||
|
|
- **EPIC image name encodes type in the prefix** — Natural images use `epic_1b_` prefix; enhanced color-corrected images use `epic_RGB_` prefix. The API returns the correct filename in `img['image']`; don't guess the prefix.
|
||
|
|
|
||
|
|
- **EPIC `/api/natural/all` returns newest-first** — The list of 3,477+ available dates starts from today and goes back to 2015-06-13. Not all days have images (gaps during spacecraft maintenance).
|
||
|
|
|
||
|
|
- **Exoplanet `ps` table has multiple rows per planet** — Different publications report different measurements for the same planet. Use `pscomppars` for one-row-per-planet composite parameters. `ps` is useful when you need all reported values or want to filter by specific reference.
|
||
|
|
|
||
|
|
- **Exoplanet null values come back as `None` in JSON** — Many fields like `pl_masse` are `null` for planets without mass measurements. Always guard with `if row['pl_masse'] is not None`.
|
||
|
|
|
||
|
|
- **`http_get` in helpers.py uses stdlib `urllib`** — On some macOS Python 3.11 installs, SSL certificate verification fails (`CERTIFICATE_VERIFY_FAILED`). If you hit this, run `curl` via `subprocess` as a fallback, or install certifi and patch the default SSL context. The harness's browser CDP connection is not affected; only `http_get` is.
|