重开仓库内容:改推五个技能(browser-harness / humanizer / humanizer-zh / product-planning / session-mechanism)
按授权清空原有内容后重新提交(原 oil-ui-pro 一并移出,可从历史恢复)。 browser-harness 剔除 .venv 等运行环境;根 .gitignore 补记 .venv/ 与 node_modules/。
This commit is contained in:
1 parent
9a152a1952
commit
777f7fe5d0
240 files changed
+52518
-4409
No files matched your search
@@ -0,0 +1,537 @@
|
||||
# Macrotrends — Data Extraction
|
||||
|
||||
`https://www.macrotrends.net` — long-term historical financial and economic charts. Three access patterns depending on page type; all work with plain `http_get`, no browser required.
|
||||
|
||||
All results validated against live site on 2026-04-18.
|
||||
|
||||
## Do this first: pick your access pattern
|
||||
|
||||
| Goal | Pattern | Latency | Variable |
|
||||
|------|---------|---------|----------|
|
||||
| Stock OHLCV price history | Direct iframe PHP | ~190ms | `dataDaily` |
|
||||
| Stock market cap (daily) | Direct iframe PHP | ~200ms | `chartData` |
|
||||
| Stock fundamentals (PE, revenue, margins) | Direct iframe PHP | ~140ms | `chartData` |
|
||||
| S&P 500 / composite index charts | `chart_iframe_comp.php` | ~90ms | `originalData` |
|
||||
| Economic indicators (rates, yields, CPI) | `/economic-data/` JSON API | ~150ms | `data[]` array |
|
||||
| Gold, commodity prices | Either path (both work) | ~150ms | `data[]` or `originalData` |
|
||||
|
||||
**Never use the browser for Macrotrends read-only tasks.** All endpoints are accessible via `http_get` with the default `Mozilla/5.0` UA. For pages that occasionally 403, switch to a Chrome UA (see gotchas).
|
||||
|
||||
---
|
||||
|
||||
## Pattern 1: Stock price history (OHLCV)
|
||||
|
||||
Construct the iframe URL directly — no need to fetch the main page first.
|
||||
|
||||
```python
|
||||
import json, re
|
||||
from helpers import http_get
|
||||
|
||||
def get_stock_ohlcv(ticker: str, years_back: int = None) -> list[dict]:
|
||||
"""
|
||||
Returns daily OHLCV records for any US stock.
|
||||
|
||||
ticker: uppercase ticker symbol, e.g. 'AAPL', 'MSFT', 'TSLA', 'NVDA'
|
||||
years_back: number of years of history (1=~250 records, 15=~3772 records).
|
||||
Omit (None) to get ALL available history (AAPL goes back to 1980).
|
||||
"""
|
||||
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t={ticker}"
|
||||
if years_back:
|
||||
url += f"&yb={years_back}"
|
||||
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+dataDaily\s*=\s*\[', html)
|
||||
if not m:
|
||||
raise ValueError(f"No dataDaily found for ticker {ticker!r}")
|
||||
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
# Usage
|
||||
records = get_stock_ohlcv('AAPL', years_back=15)
|
||||
# [{'d': '2011-04-18', 'o': '9.771', 'h': '9.9547', 'l': '9.593', 'c': '9.9433', 'v': '18.275'}, ...]
|
||||
|
||||
latest = records[-1]
|
||||
# {'d': '2026-04-17', 'o': '266.96', 'h': '272.3', 'l': '266.72', 'c': '270.23',
|
||||
# 'v': '55.211', 'ma50': '260.554', 'ma200': '251.828'}
|
||||
|
||||
print(f"{latest['d']}: close=${latest['c']} vol={latest['v']}M shares")
|
||||
```
|
||||
|
||||
### dataDaily field reference
|
||||
|
||||
| Field | Meaning | Type |
|
||||
|-------|---------|------|
|
||||
| `d` | Date (YYYY-MM-DD) | str |
|
||||
| `o` | Open price (adjusted for splits) | str/float |
|
||||
| `h` | High | str/float |
|
||||
| `l` | Low | str/float |
|
||||
| `c` | Close | str/float |
|
||||
| `v` | Volume in **millions of shares** | str/float |
|
||||
| `ma50` | 50-day moving average | str/float (appears on recent records only) |
|
||||
| `ma200` | 200-day moving average | str/float (appears on recent records only) |
|
||||
|
||||
**Note:** All price values are strings — cast with `float()`. Volume is millions: `55.211` = 55.2M shares traded.
|
||||
|
||||
### Confirmed tickers (2026-04-18)
|
||||
|
||||
All tested with direct iframe URL, no page fetch needed:
|
||||
|
||||
```python
|
||||
# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc.
|
||||
# 3772 records for yb=15 (goes back to 2011-04-18)
|
||||
# AAPL full history: 11428 records back to 1980-12-12
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern 2: Stock fundamentals (PE ratio, revenue, market cap, margins)
|
||||
|
||||
Different PHP files depending on metric. Construct directly.
|
||||
|
||||
### Market cap (daily, in billions USD)
|
||||
|
||||
```python
|
||||
import json, re
|
||||
from helpers import http_get
|
||||
|
||||
def get_market_cap(ticker: str, years_back: int = 15) -> list[dict]:
|
||||
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t={ticker}&yb={years_back}"
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+chartData\s*=\s*\[', html)
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
data = get_market_cap('AAPL')
|
||||
# [{'date': '2026-04-15', 'v1': 3929.35}, {'date': '2026-04-16', 'v1': 3884.67}, ...]
|
||||
# v1 = market cap in billions USD
|
||||
```
|
||||
|
||||
### PE ratio, revenue, current ratio (quarterly/annual fundamentals)
|
||||
|
||||
```python
|
||||
import json, re
|
||||
from helpers import http_get
|
||||
|
||||
def get_fundamental(ticker: str, metric_type: str, statement: str,
|
||||
freq: str = 'Q', years_back: int = 15) -> list[dict]:
|
||||
"""
|
||||
freq: 'Q' = quarterly, 'A' = annual
|
||||
"""
|
||||
url = (
|
||||
f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
|
||||
f"fundamental_iframe.php?t={ticker}&type={metric_type}&statement={statement}"
|
||||
f"&freq={freq}&sub=&yb={years_back}"
|
||||
)
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+chartData\s*=\s*\[', html)
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
# PE ratio
|
||||
pe = get_fundamental('AAPL', 'pe-ratio', 'price-ratios')
|
||||
# [{'date': '2025-09-30', 'v1': 254.146, 'v2': 7.46, 'v3': 34.07}, ...]
|
||||
# v1 = stock price, v2 = quarterly EPS, v3 = PE ratio
|
||||
|
||||
# Revenue
|
||||
rev = get_fundamental('AAPL', 'revenue', 'income-statement')
|
||||
# [{'date': '2025-12-31', 'v1': 435.617, 'v2': 143.756, 'v3': 15.65}, ...]
|
||||
# v1 = TTM revenue ($B), v2 = quarterly revenue ($B), v3 = YoY growth %
|
||||
|
||||
# Total assets
|
||||
assets = get_fundamental('AAPL', 'total-assets', 'balance-sheet')
|
||||
|
||||
# Current ratio
|
||||
ratio = get_fundamental('AAPL', 'current-ratio', 'ratios')
|
||||
```
|
||||
|
||||
### Profit margins
|
||||
|
||||
```python
|
||||
def get_profit_margins(ticker: str, years_back: int = 15) -> list[dict]:
|
||||
url = (
|
||||
f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
|
||||
f"fundamental_metric.php?t={ticker}&chart=profit-margin&sub=&yb={years_back}"
|
||||
)
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+chartData\s*=\s*\[', html)
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
margins = get_profit_margins('AAPL')
|
||||
# [{'date': '2025-12-31', 'v1': 47.33, 'v2': 32.38, 'v3': 27.04}, ...]
|
||||
# v1 = gross margin %, v2 = operating margin %, v3 = net margin %
|
||||
```
|
||||
|
||||
### Dividend yield
|
||||
|
||||
```python
|
||||
def get_dividend_yield(ticker: str, years_back: int = 15) -> list[dict]:
|
||||
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t={ticker}&yb={years_back}"
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+chartData\s*=\s*\[', html)
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
dy = get_dividend_yield('AAPL')
|
||||
# [{'date': '2026-04-17', 'c': 270.23, 'ttm_d': 1.03848, 'ttm_dy': 0.3843}, ...]
|
||||
# c = stock price, ttm_d = TTM dividend ($), ttm_dy = TTM yield (%)
|
||||
```
|
||||
|
||||
### Stock metric URL reference
|
||||
|
||||
| Metric | PHP file | Extra params |
|
||||
|--------|----------|-------------|
|
||||
| Stock price OHLCV | `stock_price_history.php` | — |
|
||||
| Market cap (daily) | `market_cap.php` | — |
|
||||
| Dividend yield | `dividend_yield.php` | — |
|
||||
| Stock splits (price history) | `stock_splits.php` | — |
|
||||
| PE ratio | `fundamental_iframe.php` | `type=pe-ratio&statement=price-ratios` |
|
||||
| Revenue | `fundamental_iframe.php` | `type=revenue&statement=income-statement` |
|
||||
| Total assets | `fundamental_iframe.php` | `type=total-assets&statement=balance-sheet` |
|
||||
| Current ratio | `fundamental_iframe.php` | `type=current-ratio&statement=ratios` |
|
||||
| Profit margins | `fundamental_metric.php` | `chart=profit-margin` |
|
||||
|
||||
Base URL prefix: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/`
|
||||
|
||||
All take `?t={TICKER}&yb={N}` (or `&sub=&yb={N}` for the fundamental ones).
|
||||
|
||||
---
|
||||
|
||||
## Pattern 3: Index and composite charts (S&P 500, Shiller PE, etc.)
|
||||
|
||||
These pages embed chart data via `chart_iframe_comp.php`. The variable is `originalData`.
|
||||
|
||||
```python
|
||||
import json, re
|
||||
from helpers import http_get
|
||||
|
||||
def extract_index_chart(page_id: int, url_slug: str) -> list[dict]:
|
||||
"""
|
||||
page_id: the numeric ID from the page URL, e.g. 2577
|
||||
url_slug: last segment of the page URL, e.g. 'sp500-pe-ratio-price-to-earnings-chart'
|
||||
"""
|
||||
url = f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={page_id}&url={url_slug}"
|
||||
html = http_get(url)
|
||||
m = re.search(r'var\s+originalData\s*=\s*\[', html)
|
||||
if not m:
|
||||
raise ValueError("originalData not found — this page may use a different pattern")
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0: ei = j; break
|
||||
return json.loads(html[si:ei+1])
|
||||
|
||||
# S&P 500 PE ratio (1180 monthly records, 1927-2026)
|
||||
pe_data = extract_index_chart(2577, 'sp500-pe-ratio-price-to-earnings-chart')
|
||||
# [{'date': '1927-12-01', 'close': '15.9099'}, ..., {'date': '2026-03-01', 'close': '27.8925'}]
|
||||
# 'close' is the PE ratio value
|
||||
|
||||
# Gold prices (1336 monthly records, 1915-2026)
|
||||
gold_data = extract_index_chart(1333, 'historical-gold-prices-100-year-chart')
|
||||
# [{'id': 'GOLDAMGBD228NLBM', 'date': '1915-01-01', 'close': '629.36', 'close1': '19.250'}, ...]
|
||||
# 'close' = inflation-adjusted price, 'close1' = nominal USD price
|
||||
|
||||
print(f"Latest S&P PE: {pe_data[-1]}") # {'date': '2026-03-01', 'close': '27.8925'}
|
||||
print(f"Latest gold: {gold_data[-1]}") # {'id': ..., 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
|
||||
```
|
||||
|
||||
### Detecting which pattern a page uses
|
||||
|
||||
```python
|
||||
def get_page_pattern(page_url: str) -> str:
|
||||
html = http_get(page_url)
|
||||
if 'chart_iframe_comp.php' in html:
|
||||
return 'index_chart' # use extract_index_chart()
|
||||
elif 'generateChart' in html and 'highchartsURL' in html:
|
||||
return 'economic_api' # use get_economic_data()
|
||||
elif '/production/stocks/desktop/PRODUCTION/' in html:
|
||||
return 'stock_iframe' # use get_stock_ohlcv() etc.
|
||||
return 'unknown'
|
||||
```
|
||||
|
||||
### To get the ID and slug from a page
|
||||
|
||||
```python
|
||||
import re
|
||||
from helpers import http_get
|
||||
|
||||
page_url = "https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart"
|
||||
html = http_get(page_url)
|
||||
|
||||
# Option A: parse from the iframe src in the HTML
|
||||
m = re.search(r'chart_iframe_comp\.php\?id=(\d+)&url=([^"&]+)', html)
|
||||
if m:
|
||||
page_id, url_slug = int(m.group(1)), m.group(2)
|
||||
|
||||
# Option B: derive from the page URL (works when slug matches)
|
||||
import urllib.parse
|
||||
parts = page_url.rstrip('/').split('/')
|
||||
page_id = int(parts[-2]) # 2577
|
||||
url_slug = parts[-1] # 'sp500-pe-ratio-price-to-earnings-chart'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern 4: Economic indicator API
|
||||
|
||||
Pages that use `generateChart()` in their JS load data from `/economic-data/{pageID}/{freq}`.
|
||||
This endpoint requires a `Referer` header matching the page URL.
|
||||
|
||||
```python
|
||||
import json, datetime, gzip, urllib.request
|
||||
from helpers import http_get
|
||||
|
||||
def get_economic_data(page_id: int, referer_url: str, freq: str = 'D') -> dict:
|
||||
"""
|
||||
page_id: numeric ID from the page URL (e.g. 2015 for Fed Funds Rate)
|
||||
referer_url: the full page URL — required as Referer header
|
||||
freq: 'D' = daily, 'M' = monthly (not all support both)
|
||||
|
||||
Returns {'data': [[ts_ms, value], ...], 'metadata': {...}}
|
||||
"""
|
||||
url = f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json, */*",
|
||||
"Accept-Encoding": "gzip",
|
||||
"Referer": referer_url,
|
||||
}
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=20) as r:
|
||||
raw = r.read()
|
||||
if r.headers.get("Content-Encoding") == "gzip":
|
||||
raw = gzip.decompress(raw)
|
||||
result = json.loads(raw)
|
||||
if result is None:
|
||||
raise ValueError(f"pageID={page_id} does not support freq={freq!r}")
|
||||
return result
|
||||
|
||||
# Fed Funds Rate (daily, 25319 records)
|
||||
ffr = get_economic_data(2015, "https://www.macrotrends.net/2015/fed-funds-rate-historical-chart", freq='D')
|
||||
print(ffr['metadata']['name']) # 'Fed Funds Interest Rate'
|
||||
print(ffr['metadata']['label']) # '%'
|
||||
|
||||
# Convert timestamps to dates
|
||||
for ts_ms, value in ffr['data'][-3:]:
|
||||
dt = datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC)
|
||||
print(f"{dt.strftime('%Y-%m-%d')}: {value}%")
|
||||
# 2026-04-13: 3.64%
|
||||
# 2026-04-14: 3.64%
|
||||
# 2026-04-15: 3.64%
|
||||
|
||||
# 10-Year Treasury yield (daily, 16074 records)
|
||||
t10 = get_economic_data(2016, "https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart", freq='D')
|
||||
# Last: 2026-04-15: 4.29%
|
||||
|
||||
# Gold prices (monthly, 1336 records, 1915-present) — template=5
|
||||
gold = get_economic_data(1333, "https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart", freq='M')
|
||||
# metadata: {'name': 'Gold Prices', 'currency': '$', 'label': ''}
|
||||
|
||||
# US Unemployment Rate (monthly, 938 records)
|
||||
unemp = get_economic_data(1316, "https://www.macrotrends.net/1316/us-national-unemployment-rate", freq='M')
|
||||
# metadata: {'name': 'U.S. Unemployment Rate', 'label': '%'}
|
||||
|
||||
# Debt-to-GDP ratio (monthly, 712 records)
|
||||
debt_gdp = get_economic_data(1381, "https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart", freq='M')
|
||||
```
|
||||
|
||||
### metadata fields
|
||||
|
||||
```python
|
||||
{
|
||||
'name': 'Fed Funds Interest Rate', # chart title
|
||||
'tableHeaderName': 'Fed Funds Interest Rate',
|
||||
'currency': '', # '$' for dollar-denominated series
|
||||
'label': '%', # units label
|
||||
'chartType': 'line',
|
||||
'mobileChartType': 'line',
|
||||
'lineWidth': 2,
|
||||
'positiveColor': '#2caffe',
|
||||
'negativeColor': '',
|
||||
'decimals': '',
|
||||
'chartScale': 'linear',
|
||||
'seriesUnits': ''
|
||||
}
|
||||
```
|
||||
|
||||
### Available frequency codes
|
||||
|
||||
| Code | Meaning | Notes |
|
||||
|------|---------|-------|
|
||||
| `D` | Daily | Most series support this |
|
||||
| `M` | Monthly | Returns `null` if not available |
|
||||
| `Q` | Quarterly | Usually `null` — use `M` instead |
|
||||
| `A` | Annual | Usually `null` — use `M` instead |
|
||||
| `DEFAULT` | Default (usually monthly) | Same data as `M` for most series |
|
||||
| `INDEXMONTHLY` | Monthly index close | Some commodity/index series |
|
||||
| `INDEXDAILY` | Daily index | Some series |
|
||||
| `DAILYEXCHANGERATE` | Daily FX rate | Currency pairs |
|
||||
| `10YD` | 10-year daily | Specialized series |
|
||||
|
||||
Try `D` first, fall back to `M` if you get `null`.
|
||||
|
||||
### Known economic page IDs
|
||||
|
||||
| ID | URL slug | Description |
|
||||
|----|----------|-------------|
|
||||
| 1316 | us-national-unemployment-rate | U.S. Unemployment Rate (monthly, back to 1948) |
|
||||
| 1333 | historical-gold-prices-100-year-chart | Gold Prices (monthly, back to 1915) |
|
||||
| 1381 | debt-to-gdp-ratio-historical-chart | U.S. Debt to GDP Ratio |
|
||||
| 2015 | fed-funds-rate-historical-chart | Fed Funds Interest Rate (daily, back to 1954) |
|
||||
| 2016 | 10-year-treasury-bond-rate-yield-chart | 10-Year Treasury Yield (daily, back to 1962) |
|
||||
| 2577 | sp500-pe-ratio-price-to-earnings-chart | S&P 500 PE Ratio (uses `chart_iframe_comp.php`) |
|
||||
|
||||
---
|
||||
|
||||
## Generic extraction helper
|
||||
|
||||
One function that handles all three embedded-JS patterns:
|
||||
|
||||
```python
|
||||
import json, re
|
||||
from helpers import http_get
|
||||
|
||||
def extract_chart_var(html: str, var_name: str) -> list:
|
||||
"""Extract a JS array variable from Macrotrends iframe HTML."""
|
||||
m = re.search(rf'var\s+{re.escape(var_name)}\s*=\s*\[', html)
|
||||
if not m:
|
||||
return []
|
||||
si = html.index('[', m.start())
|
||||
bc = 0
|
||||
for j, ch in enumerate(html[si:], si):
|
||||
if ch == '[': bc += 1
|
||||
elif ch == ']':
|
||||
bc -= 1
|
||||
if bc == 0:
|
||||
return json.loads(html[si:j+1])
|
||||
return []
|
||||
|
||||
# Works for dataDaily, chartData, or originalData:
|
||||
html = http_get("https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t=AAPL&yb=15")
|
||||
daily = extract_chart_var(html, 'dataDaily')
|
||||
|
||||
html2 = http_get("https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=2577&url=sp500-pe-ratio-price-to-earnings-chart")
|
||||
pe_data = extract_chart_var(html2, 'originalData')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## URL construction guide
|
||||
|
||||
### Stock pages
|
||||
|
||||
```python
|
||||
STOCK_BASE = "https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
|
||||
|
||||
# Price history OHLCV
|
||||
f"{STOCK_BASE}stock_price_history.php?t={ticker}" # all history
|
||||
f"{STOCK_BASE}stock_price_history.php?t={ticker}&yb={years}" # last N years
|
||||
|
||||
# Market cap
|
||||
f"{STOCK_BASE}market_cap.php?t={ticker}&yb={years}"
|
||||
|
||||
# Fundamentals
|
||||
f"{STOCK_BASE}fundamental_iframe.php?t={ticker}&type={type}&statement={stmt}&freq={freq}&sub=&yb={years}"
|
||||
# type/statement combos: pe-ratio/price-ratios, revenue/income-statement,
|
||||
# total-assets/balance-sheet, current-ratio/ratios
|
||||
|
||||
# Metrics
|
||||
f"{STOCK_BASE}fundamental_metric.php?t={ticker}&chart={metric}&sub=&yb={years}"
|
||||
# metrics: profit-margin
|
||||
|
||||
# Dividend yield
|
||||
f"{STOCK_BASE}dividend_yield.php?t={ticker}&yb={years}"
|
||||
```
|
||||
|
||||
### Economic / index pages
|
||||
|
||||
```python
|
||||
# From numeric ID + URL slug (read from page source or page URL)
|
||||
f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={id}&url={slug}"
|
||||
|
||||
# Economic indicator JSON API (requires Referer header)
|
||||
f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate limits and anti-bot
|
||||
|
||||
- **No rate limiting observed** at any tested volume. 10 rapid requests to the same stock iframe completed in 1.8s with no throttling, CAPTCHA, or 429 errors.
|
||||
- **Default UA works** (`Mozilla/5.0`) for most endpoints. The iframe PHP files never 403'd.
|
||||
- **Chrome UA needed** for some main HTML pages (not data endpoints): use when fetching `/stocks/charts/...` or `/2015/...` wrapper pages if you get 403. Switch to:
|
||||
```python
|
||||
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
|
||||
```
|
||||
- **Referer required** for `/economic-data/{id}/{freq}` — send the page URL as `Referer`. Without it, the request is allowed but you get a 403 on some pages.
|
||||
- **No cookies, sessions, or auth tokens** needed for any endpoint.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
**Main page URL ≠ data page:** Some URLs redirect to different content. `/1316/us-national-debt-by-year` redirects to `/1316/us-national-unemployment-rate`. Always check the final URL with `r.url` if the returned data looks wrong. Use the final URL as the Referer.
|
||||
|
||||
**yb parameter controls history depth:**
|
||||
- `yb=1` → ~250 records (last year)
|
||||
- `yb=15` → ~3772 records (last 15 years)
|
||||
- omit → full history (AAPL: 11428 records to 1980; default for most queries)
|
||||
|
||||
**Two iframe patterns for economic pages:** Pages at `macrotrends.net/NNNN/slug` use either `chart_iframe_comp.php` (→ `originalData`) or `generateChart` + `/economic-data/` API. Check the main page HTML to detect which:
|
||||
```python
|
||||
if 'chart_iframe_comp.php' in html: # use extract_index_chart()
|
||||
elif 'highchartsURL' in html: # use get_economic_data()
|
||||
```
|
||||
|
||||
**Gold data has two price columns:**
|
||||
```python
|
||||
{'id': 'GOLDAMGBD228NLBM', 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
|
||||
# 'close' = inflation-adjusted price (base year adjusts over time)
|
||||
# 'close1' = nominal USD price (the raw market price)
|
||||
```
|
||||
|
||||
**Economic API frequency codes:** Only `D` and `M` consistently return data across most series. `A` and `Q` return `null` for most economic indicators. Always try `D` first.
|
||||
|
||||
**chartData fields vary by metric:**
|
||||
- `market_cap.php` → `{'date', 'v1'}` (v1 = market cap in $B)
|
||||
- `fundamental_iframe.php` type=pe-ratio → `{'date', 'v1', 'v2', 'v3'}` (stock price, EPS, PE)
|
||||
- `fundamental_iframe.php` type=revenue → `{'date', 'v1', 'v2', 'v3'}` (TTM revenue, quarterly revenue, YoY%)
|
||||
- `fundamental_metric.php` chart=profit-margin → `{'date', 'v1', 'v2', 'v3'}` (gross%, operating%, net%)
|
||||
- `dividend_yield.php` → `{'date', 'c', 'ttm_d', 'ttm_dy'}` (price, dividend, yield%)
|
||||
|
||||
**Bracket matching required for large arrays:** The `var dataDaily = [...]` in stock iframes is ~450KB with 3772 OHLCV records. The `re.DOTALL` greedy approach works but is slow; bracket-counting (`bc` pattern above) is O(n) and fast.
|
||||
|
||||
**No public API for ticker lookup:** To find the company slug for a URL, check the search endpoint: `https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?v=YYYYMMDD` — but the stock price iframe only needs the ticker symbol (`?t=AAPL`), not the slug.
|
||||
Reference in new issue
Block a user