421 lines
18 KiB
Markdown
421 lines
18 KiB
Markdown
|
# PubMed / NCBI — Scraping & Data Extraction
|
||
|
|
|
||
|
|
`https://pubmed.ncbi.nlm.nih.gov` — 37 M+ biomedical citations. **Never use the browser for PubMed.** All data is reachable via `http_get` using the NCBI E-utilities REST API. No API key required; a free key raises the rate limit from 3 to 10 req/s.
|
||
|
|
|
||
|
|
## Do this first
|
||
|
|
|
||
|
|
**ESearch → ESummary is the fastest pipeline for most tasks — two calls, JSON responses, no XML parsing.**
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Step 1: search → get PMIDs
|
||
|
|
search = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
|
||
|
|
"?db=pubmed&term=deep+learning+radiology&retmax=10&retmode=json"
|
||
|
|
))
|
||
|
|
pmids = search['esearchresult']['idlist'] # e.g. ['41999029', '41998456', ...]
|
||
|
|
count = search['esearchresult']['count'] # total hits across all pages
|
||
|
|
|
||
|
|
# Step 2: fetch lightweight metadata for all PMIDs in one call
|
||
|
|
summary = json.loads(http_get(
|
||
|
|
f"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi"
|
||
|
|
f"?db=pubmed&id={','.join(pmids)}&retmode=json"
|
||
|
|
))
|
||
|
|
result = summary['result']
|
||
|
|
for uid in result['uids']:
|
||
|
|
art = result[uid]
|
||
|
|
print(uid, art['pubdate'], art['source'])
|
||
|
|
print(" ", art['title'][:80])
|
||
|
|
print(" authors:", [a['name'] for a in art['authors'][:3]])
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# 41999029 2026 Apr 18 Med Sci Monit
|
||
|
|
# Use of Deep Learning Models in the Diagnosis of Proptosis Through Orbi
|
||
|
|
# authors: ['Kesimal U', 'Akkaya HE', 'Polat Ö']
|
||
|
|
# 41998456 2026 Apr 17 Sci Rep
|
||
|
|
# ...
|
||
|
|
```
|
||
|
|
|
||
|
|
Use **EFetch XML** when you need: full abstract text, MeSH terms, complete author names (not just "Last I"), structured abstract labels, or the DOI from within the article record.
|
||
|
|
|
||
|
|
## Common workflows
|
||
|
|
|
||
|
|
### Search PubMed (ESearch)
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
|
||
|
|
"?db=pubmed"
|
||
|
|
"&term=large+language+models+clinical"
|
||
|
|
"&retmax=5"
|
||
|
|
"&retmode=json"
|
||
|
|
"&sort=pub+date" # newest first; default is relevance
|
||
|
|
"&datetype=pdat" # filter by publication date
|
||
|
|
"&mindate=2024/01/01&maxdate=2024/12/31" # YYYY/MM/DD format
|
||
|
|
))
|
||
|
|
result = data['esearchresult']
|
||
|
|
print("Total hits:", result['count']) # '24160' — note: string, not int
|
||
|
|
print("PMIDs:", result['idlist'])
|
||
|
|
print("Query translation:", result['querytranslation'])
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# Total hits: 24160
|
||
|
|
# PMIDs: ['41996895', '41996722', '41996006', '41995888', '41995759']
|
||
|
|
# Query translation: "large language models"[MeSH Terms] OR ...
|
||
|
|
```
|
||
|
|
|
||
|
|
#### ESearch field tags (append to term)
|
||
|
|
|
||
|
|
```
|
||
|
|
machine learning[MeSH Terms] MeSH controlled vocabulary
|
||
|
|
Hinton GE[Author] author last + initials
|
||
|
|
attention is all you need[Title] title words
|
||
|
|
Nature[Journal] journal name
|
||
|
|
2024[pdat] publication year
|
||
|
|
```
|
||
|
|
|
||
|
|
Boolean operators: `AND`, `OR`, `NOT`. Phrase search: `"exact phrase"[Title]`.
|
||
|
|
|
||
|
|
#### Sort options (`sort=`)
|
||
|
|
|
||
|
|
| Value | Effect |
|
||
|
|
|---|---|
|
||
|
|
| *(omit)* | Relevance (default) |
|
||
|
|
| `pub+date` | Most recent publication first |
|
||
|
|
| `Author` | First author alphabetical |
|
||
|
|
| `JournalName` | Journal alphabetical |
|
||
|
|
|
||
|
|
### Lightweight metadata — ESummary (JSON, no XML)
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi"
|
||
|
|
"?db=pubmed&id=41999029,41998456,41997837&retmode=json"
|
||
|
|
))
|
||
|
|
result = data['result']
|
||
|
|
for uid in result['uids']:
|
||
|
|
art = result[uid]
|
||
|
|
# Key fields available:
|
||
|
|
title = art['title'] # full title string
|
||
|
|
source = art['source'] # abbreviated journal name
|
||
|
|
fulljournalname = art['fulljournalname']
|
||
|
|
pubdate = art['pubdate'] # e.g. '2026 Apr 18'
|
||
|
|
epubdate = art['epubdate'] # e-pub ahead of print date (may be empty)
|
||
|
|
authors = art['authors'] # list of {'name': 'Last I', 'authtype': ...}
|
||
|
|
volume = art['volume']
|
||
|
|
issue = art['issue']
|
||
|
|
pages = art['pages']
|
||
|
|
pubtype = art['pubtype'] # list: ['Journal Article', 'Review', ...]
|
||
|
|
# Extract DOI from elocationid or articleids:
|
||
|
|
doi_field = art['elocationid'] # e.g. 'doi: 10.12659/MSM.951157'
|
||
|
|
article_ids = {x['idtype']: x['value'] for x in art['articleids']}
|
||
|
|
doi = article_ids.get('doi')
|
||
|
|
pmc_id = article_ids.get('pmc') # PMC ID if open access
|
||
|
|
print(uid, pubdate, source)
|
||
|
|
print(" ", title[:70])
|
||
|
|
print(" doi:", doi, "| pmc:", pmc_id)
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# 41999029 2026 Apr 18 Med Sci Monit
|
||
|
|
# Use of Deep Learning Models in the Diagnosis of Proptosis Through Orbi
|
||
|
|
# doi: 10.12659/MSM.951157 | pmc: None
|
||
|
|
```
|
||
|
|
|
||
|
|
### Full article metadata — EFetch XML
|
||
|
|
|
||
|
|
Use this for full abstracts, complete author names, MeSH terms, structured abstract sections.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json, xml.etree.ElementTree as ET
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
raw = http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi"
|
||
|
|
"?db=pubmed&id=41999029,36328784&retmode=xml&rettype=abstract"
|
||
|
|
)
|
||
|
|
root = ET.fromstring(raw)
|
||
|
|
|
||
|
|
for art in root.findall('.//PubmedArticle'):
|
||
|
|
mc = art.find('MedlineCitation')
|
||
|
|
pmid = mc.find('PMID').text
|
||
|
|
article = mc.find('Article')
|
||
|
|
|
||
|
|
# Title — use itertext() to handle embedded tags like <i>, <sub>
|
||
|
|
title = ''.join(article.find('ArticleTitle').itertext()).strip()
|
||
|
|
|
||
|
|
# Abstract — plain or structured (BACKGROUND / METHODS / RESULTS / CONCLUSION)
|
||
|
|
abstract_el = article.find('Abstract')
|
||
|
|
if abstract_el is not None:
|
||
|
|
sections = []
|
||
|
|
for t in abstract_el.findall('AbstractText'):
|
||
|
|
label = t.get('Label', '') # e.g. 'BACKGROUND', 'METHODS'
|
||
|
|
text = ''.join(t.itertext()).strip()
|
||
|
|
sections.append(f"[{label}] {text}" if label else text)
|
||
|
|
abstract = ' '.join(sections)
|
||
|
|
else:
|
||
|
|
abstract = '' # ~15% of articles have no abstract
|
||
|
|
|
||
|
|
# Journal + year
|
||
|
|
journal = article.find('Journal')
|
||
|
|
j_title = journal.find('Title').text if journal is not None else ''
|
||
|
|
pub_date = journal.find('.//PubDate') if journal is not None else None
|
||
|
|
if pub_date is not None:
|
||
|
|
year_el = pub_date.find('Year')
|
||
|
|
medline_el = pub_date.find('MedlineDate') # fallback for old/seasonal dates
|
||
|
|
season_el = pub_date.find('Season') # e.g. 'Jul-Aug', 'Oct-Dec'
|
||
|
|
year = (year_el.text if year_el is not None
|
||
|
|
else medline_el.text[:4] if medline_el is not None else '')
|
||
|
|
|
||
|
|
# DOI
|
||
|
|
doi_el = next(
|
||
|
|
(e for e in article.findall('ELocationID') if e.get('EIdType') == 'doi'),
|
||
|
|
None
|
||
|
|
)
|
||
|
|
doi = doi_el.text if doi_el is not None else ''
|
||
|
|
|
||
|
|
# Authors — handle CollectiveName (consortium/group authors)
|
||
|
|
author_list = article.find('AuthorList')
|
||
|
|
authors = []
|
||
|
|
if author_list is not None:
|
||
|
|
for a in author_list.findall('Author'):
|
||
|
|
collective = a.find('CollectiveName')
|
||
|
|
last = a.find('LastName')
|
||
|
|
fore = a.find('ForeName')
|
||
|
|
initials = a.find('Initials')
|
||
|
|
if collective is not None:
|
||
|
|
authors.append(collective.text)
|
||
|
|
elif last is not None:
|
||
|
|
full = last.text
|
||
|
|
if fore is not None:
|
||
|
|
full += f", {fore.text}"
|
||
|
|
authors.append(full)
|
||
|
|
|
||
|
|
# MeSH controlled vocabulary terms
|
||
|
|
mesh_list = mc.find('MeshHeadingList')
|
||
|
|
mesh_terms = []
|
||
|
|
if mesh_list is not None:
|
||
|
|
mesh_terms = [
|
||
|
|
mh.find('DescriptorName').text
|
||
|
|
for mh in mesh_list.findall('MeshHeading')
|
||
|
|
if mh.find('DescriptorName') is not None
|
||
|
|
]
|
||
|
|
|
||
|
|
print(f"PMID={pmid} ({year}) {j_title}")
|
||
|
|
print(f" Title: {title[:70]}")
|
||
|
|
print(f" Authors: {authors[:3]}")
|
||
|
|
print(f" DOI: {doi}")
|
||
|
|
print(f" MeSH: {mesh_terms[:4]}")
|
||
|
|
print(f" Abstract: {abstract[:120]}")
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# PMID=41999029 (2026) Medical science monitor : international medical...
|
||
|
|
# Title: Use of Deep Learning Models in the Diagnosis of Proptosis Thro
|
||
|
|
# Authors: ['Kesimal, Uğur', 'Akkaya, Habip Eser', 'Polat, Önder']
|
||
|
|
# DOI: 10.12659/MSM.951157
|
||
|
|
# MeSH: ['Humans', 'Deep Learning', 'Exophthalmos', 'Magnetic Resonance Imaging']
|
||
|
|
# Abstract: BACKGROUND Proptosis is a common manifestation of orbital disease...
|
||
|
|
# PMID=36328784 (...)
|
||
|
|
# Abstract: [OBJECTIVES] Physical inactivity and sedentary behaviour... ← structured
|
||
|
|
```
|
||
|
|
|
||
|
|
### Large result sets — usehistory + WebEnv
|
||
|
|
|
||
|
|
When `count` exceeds `retmax` (max 10 000), use server-side history to paginate EFetch without re-running ESearch on every page.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json, xml.etree.ElementTree as ET
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Step 1: ESearch with usehistory=y — NCBI holds result set on server
|
||
|
|
search = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
|
||
|
|
"?db=pubmed&term=CRISPR+gene+editing&retmax=0&retmode=json&usehistory=y"
|
||
|
|
))
|
||
|
|
webenv = search['esearchresult']['webenv'] # server-side session token
|
||
|
|
query_key = search['esearchresult']['querykey'] # result set ID within session
|
||
|
|
total = int(search['esearchresult']['count'])
|
||
|
|
print(f"Total: {total}, WebEnv: {webenv[:30]}..., query_key: {query_key}")
|
||
|
|
# Confirmed output (2026-04-18):
|
||
|
|
# Total: 24160, WebEnv: MCID_69e4203757db89391008d6f1..., query_key: 1
|
||
|
|
|
||
|
|
# Step 2: EFetch pages using WebEnv (no re-searching)
|
||
|
|
batch_size = 200
|
||
|
|
for start in range(0, min(total, 1000), batch_size): # cap at 1000 for demo
|
||
|
|
raw = http_get(
|
||
|
|
f"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi"
|
||
|
|
f"?db=pubmed&query_key={query_key}&WebEnv={webenv}"
|
||
|
|
f"&retstart={start}&retmax={batch_size}&retmode=xml&rettype=abstract"
|
||
|
|
)
|
||
|
|
root = ET.fromstring(raw)
|
||
|
|
articles = root.findall('.//PubmedArticle')
|
||
|
|
print(f" Fetched {len(articles)} articles (start={start})")
|
||
|
|
# process articles here...
|
||
|
|
```
|
||
|
|
|
||
|
|
### EInfo — list available NCBI databases
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?retmode=json"
|
||
|
|
))
|
||
|
|
dbs = data['einforesult']['dblist']
|
||
|
|
print(f"Total databases: {len(dbs)}") # Confirmed: 39 (2026-04-18)
|
||
|
|
print(dbs[:10])
|
||
|
|
# ['pubmed', 'protein', 'nuccore', 'ipg', 'nucleotide', 'structure',
|
||
|
|
# 'genome', 'annotinfo', 'assembly', 'bioproject']
|
||
|
|
```
|
||
|
|
|
||
|
|
Get PubMed-specific metadata (field list, link list):
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi?db=pubmed&retmode=json"
|
||
|
|
))
|
||
|
|
db_info = data['einforesult']['dbinfo'][0]
|
||
|
|
print("DB name:", db_info['dbname'])
|
||
|
|
print("Record count:", db_info['count']) # total PubMed records
|
||
|
|
link_names = [l['name'] for l in db_info.get('linklist', [])]
|
||
|
|
print(f"Link types ({len(link_names)}):", link_names[:5])
|
||
|
|
# Confirmed (2026-04-18):
|
||
|
|
# DB name: pubmed
|
||
|
|
# Record count: 37620453
|
||
|
|
# Link types (48): ['pubmed_assembly', 'pubmed_bioproject', ...]
|
||
|
|
```
|
||
|
|
|
||
|
|
### ELink — cross-database linking
|
||
|
|
|
||
|
|
ELink connects a PubMed record to associated data in other NCBI databases. The `pubmed_pubmed` "related articles" linkname relies on a similarity server that is intermittently unavailable (returns `"Couldn't resolve #exLinkSrv2, the address table is empty."`). Use the non-similarity links below instead.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import json
|
||
|
|
from helpers import http_get
|
||
|
|
|
||
|
|
# Link a PMID to its free full-text in PMC (if open access)
|
||
|
|
# linkname=pubmed_pmc — may also hit the server outage; check error field
|
||
|
|
data = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi"
|
||
|
|
"?dbfrom=pubmed&id=38325330&linkname=pubmed_pmc&retmode=json"
|
||
|
|
))
|
||
|
|
error = data.get('ERROR', '')
|
||
|
|
if error:
|
||
|
|
print("ELink error:", error) # 'Couldn't resolve #exLinkSrv2...' — NCBI server issue
|
||
|
|
else:
|
||
|
|
for ls in data.get('linksets', []):
|
||
|
|
for lsdb in ls.get('linksetdbs', []):
|
||
|
|
print(lsdb['linkname'], "→", lsdb['links'][:5])
|
||
|
|
```
|
||
|
|
|
||
|
|
Available ELink linknames from pubmed (48 total):
|
||
|
|
|
||
|
|
| linkname | Target |
|
||
|
|
|---|---|
|
||
|
|
| `pubmed_pmc` | Free full text in PMC |
|
||
|
|
| `pubmed_pubmed_citedin` | Articles citing this paper |
|
||
|
|
| `pubmed_pubmed_refs` | References cited by this paper |
|
||
|
|
| `pubmed_gene` | Related Gene records |
|
||
|
|
| `pubmed_clinvar` | Clinical variants associated with publication |
|
||
|
|
| `pubmed_gds` | Related GEO datasets |
|
||
|
|
|
||
|
|
**Practical alternative**: If ELink is down, extract DOI from EFetch/ESummary and use `https://doi.org/{doi}` directly for the full-text link.
|
||
|
|
|
||
|
|
## URL and parameter reference
|
||
|
|
|
||
|
|
### E-utilities base URLs
|
||
|
|
|
||
|
|
```
|
||
|
|
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi # search → PMIDs
|
||
|
|
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi # PMIDs → JSON summary
|
||
|
|
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi # PMIDs → full XML
|
||
|
|
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi # cross-db links
|
||
|
|
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/einfo.fcgi # DB metadata
|
||
|
|
```
|
||
|
|
|
||
|
|
### ESearch parameters
|
||
|
|
|
||
|
|
| Parameter | Values | Notes |
|
||
|
|
|---|---|---|
|
||
|
|
| `db` | `pubmed` | Always `pubmed` for PubMed |
|
||
|
|
| `term` | query string | Supports field tags like `[Author]`, `[Title]`, `[MeSH Terms]` |
|
||
|
|
| `retmax` | integer, max 10000 | Results returned per call |
|
||
|
|
| `retmode` | `json` | JSON output |
|
||
|
|
| `sort` | `pub+date`, `Author`, `JournalName` | Default is relevance |
|
||
|
|
| `datetype` | `pdat` (pub), `edat` (entrez), `mdat` (modified) | |
|
||
|
|
| `mindate`, `maxdate` | `YYYY/MM/DD` or `YYYY` | Requires `datetype` |
|
||
|
|
| `usehistory` | `y` | Store results on server; returns `webenv` + `querykey` |
|
||
|
|
|
||
|
|
### EFetch parameters
|
||
|
|
|
||
|
|
| Parameter | Values | Notes |
|
||
|
|
|---|---|---|
|
||
|
|
| `db` | `pubmed` | |
|
||
|
|
| `id` | `38000000,37999999` | Comma-separated PMIDs; max ~200 per call |
|
||
|
|
| `query_key` + `WebEnv` | from ESearch `usehistory=y` | Alternative to `id` for large sets |
|
||
|
|
| `retstart` | integer | Offset for pagination with WebEnv |
|
||
|
|
| `retmax` | integer, max 10000 | Batch size |
|
||
|
|
| `retmode` | `xml` | Use XML for EFetch (JSON not available for full records) |
|
||
|
|
| `rettype` | `abstract` | Returns abstract + core metadata |
|
||
|
|
|
||
|
|
### PubMed article URL construction
|
||
|
|
|
||
|
|
```python
|
||
|
|
pmid = "41999029"
|
||
|
|
pubmed_url = f"https://pubmed.ncbi.nlm.nih.gov/{pmid}/"
|
||
|
|
doi = "10.12659/MSM.951157"
|
||
|
|
doi_url = f"https://doi.org/{doi}" # resolves to publisher page
|
||
|
|
pmc_id = "PMC9876543" # from ESummary articleids
|
||
|
|
pmc_url = f"https://www.ncbi.nlm.nih.gov/pmc/articles/{pmc_id}/"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Gotchas
|
||
|
|
|
||
|
|
- **`count` is a string, not int.** `search['esearchresult']['count']` returns `'24160'`, not `24160`. Always cast with `int()` before arithmetic.
|
||
|
|
|
||
|
|
- **EFetch retmode must be `xml` for full records.** Unlike ESearch and ESummary, EFetch with `retmode=json` returns flat text (the MEDLINE citation text format), not structured JSON. Parse EFetch responses with `xml.etree.ElementTree`.
|
||
|
|
|
||
|
|
- **`ArticleTitle` may contain embedded XML tags.** Titles with italics (`<i>Staphylococcus aureus</i>`) or math (`<sub>2</sub>`) are mixed-content nodes. Always use `''.join(el.itertext())` instead of `el.text`, which silently drops everything after the first child tag.
|
||
|
|
|
||
|
|
- **~15% of articles have no abstract.** `article.find('Abstract')` returns `None` for short communications, editorials, letters, and older records. Always guard with `if abstract_el is not None`.
|
||
|
|
|
||
|
|
- **Author names vary in structure — always handle `CollectiveName`.** Consortium papers list a group name (`'GeKeR Study Group'`, `'Breast Cancer Association Consortium'`) under `<CollectiveName>` instead of `<LastName>/<ForeName>`. Individual authors have `<LastName>` + optionally `<ForeName>` and `<Initials>`. Check `CollectiveName` first; falling through to `LastName` without the check produces `None` errors.
|
||
|
|
- Confirmed real examples (2026-04-18): PMID 37586835 (`GeKeR Study Group`), PMID 36328784 (`Breast Cancer Association Consortium`)
|
||
|
|
|
||
|
|
- **PubDate has three possible structures.** Most articles have `<Year>` + optional `<Month>` + optional `<Day>`. Seasonal journals use `<Season>` (e.g. `Jul-Aug`, `Oct-Dec`) instead of `<Month>`. A minority of older records use `<MedlineDate>` (e.g. `1995 Fall`) with no `<Year>`. Safe extraction pattern:
|
||
|
|
```python
|
||
|
|
pub_date = journal.find('.//PubDate')
|
||
|
|
year_el = pub_date.find('Year') if pub_date is not None else None
|
||
|
|
medline_el = pub_date.find('MedlineDate') if pub_date is not None else None
|
||
|
|
year = (year_el.text if year_el is not None
|
||
|
|
else medline_el.text[:4] if medline_el is not None else '')
|
||
|
|
```
|
||
|
|
|
||
|
|
- **Batch EFetch: keep IDs to ~200 per call.** The API accepts comma-separated IDs in `id=`, but very large batches (500+) occasionally time out or return truncated XML. For >200 articles, iterate in chunks or use `usehistory` + `WebEnv`.
|
||
|
|
|
||
|
|
- **ELink `pubmed_pubmed` (related articles) is intermittently broken.** The NCBI similarity server returns `"Couldn't resolve #exLinkSrv2, the address table is empty."` — this is a persistent server-side issue as of 2026-04-18, not a rate-limit error. Other linknames (`pubmed_gene`, `pubmed_pmc`, `pubmed_clinvar`) fail with the same error. Use the DOI as a fallback link to publisher full text.
|
||
|
|
|
||
|
|
- **Rate limits: 3 req/s without API key, 10 req/s with free key.** Exceeding 3 req/s returns HTTP 429. Insert `time.sleep(0.34)` between sequential calls without a key. Get a free API key at https://www.ncbi.nlm.nih.gov/account/ and append `&api_key=YOUR_KEY` to all URLs.
|
||
|
|
|
||
|
|
- **`retmax` upper bound is 10 000 for ESearch.** To retrieve more than 10 000 PMIDs for a search, use `usehistory=y` and page through EFetch with `retstart` offsets. EFetch itself also accepts `retmax` up to 10 000 per call.
|
||
|
|
|
||
|
|
- **`retmax=0` in ESearch returns only the count, not IDs — useful for counting.** Combine with `usehistory=y` to store the result for later paging without fetching IDs upfront:
|
||
|
|
```python
|
||
|
|
search = json.loads(http_get(
|
||
|
|
"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
|
||
|
|
"?db=pubmed&term=cancer&retmax=0&retmode=json&usehistory=y"
|
||
|
|
))
|
||
|
|
total = int(search['esearchresult']['count']) # e.g. 4800000
|
||
|
|
webenv = search['esearchresult']['webenv']
|
||
|
|
```
|
||
|
|
|
||
|
|
- **ESummary `authors` field uses abbreviated names (`Last I`), not full names.** Use EFetch XML to get `ForeName` (e.g. `'Kesimal, Uğur'` vs ESummary `'Kesimal U'`). For bulk tasks where full names are not needed, ESummary is faster.
|
||
|
|
|
||
|
|
- **`querytranslation` shows how NCBI interpreted your term.** The ESearch response includes `esearchresult.querytranslation` — a MeSH-expanded version of your query. Inspect it to verify the search matched what you intended.
|