309 lines
12 KiB
Markdown
309 lines
12 KiB
Markdown
|
# TradingView — Scraping & Data Extraction
|
|||
|
|
|
|||
|
|
`https://www.tradingview.com` — charting platform with multiple internal REST APIs. Stock/crypto/forex screener and symbol search work without auth. Use `http_get` or raw `urllib` for all workflows except JS-rendered chart pages.
|
|||
|
|
|
|||
|
|
## Do this first
|
|||
|
|
|
|||
|
|
**Use the scanner API for bulk screener data — one POST, no browser, full column control.**
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import json, urllib.request
|
|||
|
|
|
|||
|
|
def tv_scan(payload, market="america"):
|
|||
|
|
data = json.dumps(payload).encode()
|
|||
|
|
req = urllib.request.Request(
|
|||
|
|
f"https://scanner.tradingview.com/{market}/scan",
|
|||
|
|
data=data,
|
|||
|
|
headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
|
|||
|
|
)
|
|||
|
|
with urllib.request.urlopen(req, timeout=20) as r:
|
|||
|
|
return json.loads(r.read())
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**No auth, no Referer, no cookies required for the scanner.** Responses arrive in ~200ms.
|
|||
|
|
|
|||
|
|
## Common workflows
|
|||
|
|
|
|||
|
|
### Top stocks by market cap (screener)
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import json, urllib.request
|
|||
|
|
|
|||
|
|
payload = {
|
|||
|
|
"filter": [],
|
|||
|
|
"options": {"lang": "en"},
|
|||
|
|
"columns": ["name", "close", "change", "volume", "market_cap_basic"],
|
|||
|
|
"sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
|
|||
|
|
"range": [0, 10] # [start, end] — half-open, so this returns rows 0–9
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
data = json.dumps(payload).encode()
|
|||
|
|
req = urllib.request.Request(
|
|||
|
|
"https://scanner.tradingview.com/america/scan",
|
|||
|
|
data=data,
|
|||
|
|
headers={"Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}
|
|||
|
|
)
|
|||
|
|
with urllib.request.urlopen(req, timeout=20) as r:
|
|||
|
|
resp = json.loads(r.read())
|
|||
|
|
|
|||
|
|
# resp["totalCount"] = 19549 (all US-listed instruments)
|
|||
|
|
# resp["data"] is a list of {"s": "NASDAQ:NVDA", "d": [col0, col1, ...]}
|
|||
|
|
# "d" values align positionally with "columns" in the payload
|
|||
|
|
|
|||
|
|
cols = payload["columns"]
|
|||
|
|
for item in resp["data"]:
|
|||
|
|
row = dict(zip(cols, item["d"]))
|
|||
|
|
symbol = item["s"] # e.g. "NASDAQ:AAPL"
|
|||
|
|
print(symbol, row["close"], row["change"], row["market_cap_basic"])
|
|||
|
|
# NASDAQ:NVDA 201.68 1.68 4900823822021.0
|
|||
|
|
# NASDAQ:AAPL 270.23 2.59 3967284528489.0
|
|||
|
|
# ...
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Critical**: `"d"` is a plain positional array — index 0 = columns[0], index 1 = columns[1], etc. There are no keys in the row data itself.
|
|||
|
|
|
|||
|
|
### Pagination
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# Page 1: range [0, 20]
|
|||
|
|
# Page 2: range [20, 40]
|
|||
|
|
payload["range"] = [20, 40]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Filtering stocks
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
payload = {
|
|||
|
|
"filter": [
|
|||
|
|
{"left": "market_cap_basic", "operation": "greater", "right": 10_000_000_000},
|
|||
|
|
{"left": "volume", "operation": "greater", "right": 5_000_000},
|
|||
|
|
{"left": "change", "operation": "in_range", "right": [2, 10]},
|
|||
|
|
{"left": "exchange", "operation": "equal", "right": "NASDAQ"},
|
|||
|
|
{"left": "sector", "operation": "equal", "right": "Electronic Technology"},
|
|||
|
|
],
|
|||
|
|
"columns": ["name", "close", "change", "volume", "market_cap_basic",
|
|||
|
|
"description", "sector", "industry"],
|
|||
|
|
"sort": {"sortBy": "market_cap_basic", "sortOrder": "desc"},
|
|||
|
|
"range": [0, 20]
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Valid filter operations: `greater`, `less`, `equal`, `in_range` (right = [min, max]), `match` (substring on `name`).
|
|||
|
|
|
|||
|
|
Sector names use TradingView taxonomy (not GICS). Confirmed working values:
|
|||
|
|
- `"Electronic Technology"` — NVDA, AAPL, TSM
|
|||
|
|
- `"Technology Services"` — MSFT, GOOGL, META
|
|||
|
|
- `"Finance"`, `"Health Technology"`, `"Consumer Non-Durables"`
|
|||
|
|
|
|||
|
|
### Full list of tested valid column names
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# Price & volume
|
|||
|
|
"name" # ticker (e.g. "AAPL")
|
|||
|
|
"description" # full name ("Apple Inc.")
|
|||
|
|
"close" # last price
|
|||
|
|
"open", "high", "low"
|
|||
|
|
"volume"
|
|||
|
|
"change" # % change today
|
|||
|
|
"change_abs" # absolute price change
|
|||
|
|
"change|1M" # 1-month % change (also: |6M, |1Y)
|
|||
|
|
"High.1M", "High.6M" # period high
|
|||
|
|
"High.All", "Low.All" # all-time high/low
|
|||
|
|
"price_52_week_high" # confirmed works
|
|||
|
|
"price_52_week_low" # confirmed works
|
|||
|
|
"premarket_change" # pre-market %
|
|||
|
|
"postmarket_change" # after-hours %
|
|||
|
|
"gap" # overnight gap %
|
|||
|
|
"change_from_open_abs" # intraday move from open
|
|||
|
|
"average_volume_10d_calc" # 10-day avg volume
|
|||
|
|
"relative_volume_10d_calc" # relative volume vs 10-day avg
|
|||
|
|
"relative_volume_intraday|5" # intraday relative vol (5m bars)
|
|||
|
|
|
|||
|
|
# Fundamentals
|
|||
|
|
"market_cap_basic" # market cap in USD
|
|||
|
|
"earnings_per_share_diluted_ttm" # EPS TTM
|
|||
|
|
"price_earnings_ttm" # P/E TTM
|
|||
|
|
"P/E" # P/E (snapshot)
|
|||
|
|
"dividends_yield" # dividend yield %
|
|||
|
|
"beta_1_year" # beta
|
|||
|
|
"float_shares_outstanding" # float shares
|
|||
|
|
|
|||
|
|
# Technical ratings & indicators
|
|||
|
|
"Recommend.All" # composite rating: -1 (strong sell) to +1 (strong buy)
|
|||
|
|
"RSI" # RSI 14
|
|||
|
|
"MACD.macd" # MACD line
|
|||
|
|
|
|||
|
|
# Classification
|
|||
|
|
"sector", "industry", "country", "exchange"
|
|||
|
|
"type" # "stock", "fund", "dr" (depository receipt), etc.
|
|||
|
|
|
|||
|
|
# NOTE: "52_week_high" / "52_week_low" are INVALID — use "price_52_week_high" / "price_52_week_low"
|
|||
|
|
# NOTE: "EPS_diluted_net" is INVALID — use "earnings_per_share_diluted_ttm"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Bad columns return HTTP 400 with `{"error": "Unknown field \"X\""}`.
|
|||
|
|
|
|||
|
|
### Other scanner markets
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# market argument options (confirmed working):
|
|||
|
|
# "america" — US equities (19,549 instruments)
|
|||
|
|
# "crypto" — crypto across exchanges (56,455 instruments)
|
|||
|
|
# "forex" — FX pairs (6,401 instruments)
|
|||
|
|
# "futures" — futures (53,947 instruments)
|
|||
|
|
|
|||
|
|
# Crypto example
|
|||
|
|
payload = {
|
|||
|
|
"filter": [],
|
|||
|
|
"columns": ["name", "close", "change", "volume", "market_cap_calc"],
|
|||
|
|
"sort": {"sortBy": "market_cap_calc", "sortOrder": "desc"},
|
|||
|
|
"range": [0, 10]
|
|||
|
|
}
|
|||
|
|
resp = tv_scan(payload, market="crypto")
|
|||
|
|
# Returns BTC, ETH, etc. across Binance, Bybit, OKX...
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Symbol search (requires Origin header)
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import json, urllib.request
|
|||
|
|
|
|||
|
|
def symbol_search(query, exchange="", type_filter="", limit=50):
|
|||
|
|
url = (
|
|||
|
|
f"https://symbol-search.tradingview.com/symbol_search/v3/"
|
|||
|
|
f"?text={query}&hl=1&exchange={exchange}&lang=en"
|
|||
|
|
f"&search_type={type_filter or 'undefined'}&domain=production"
|
|||
|
|
)
|
|||
|
|
req = urllib.request.Request(url, headers={
|
|||
|
|
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
|
|||
|
|
"Origin": "https://www.tradingview.com", # REQUIRED — 403 without this
|
|||
|
|
})
|
|||
|
|
with urllib.request.urlopen(req, timeout=15) as r:
|
|||
|
|
return json.loads(r.read())
|
|||
|
|
|
|||
|
|
result = symbol_search("AAPL")
|
|||
|
|
# result["symbols_remaining"] = 137
|
|||
|
|
# result["symbols"] = list of up to 50 matches
|
|||
|
|
# result["symbols"][0] keys:
|
|||
|
|
# symbol, description, type, exchange, country, currency_code,
|
|||
|
|
# cusip, isin, cik_code, logoid, provider_id, source_id,
|
|||
|
|
# is_primary_listing, typespecs
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Gotcha**: `symbol-search.tradingview.com` requires `Origin: https://www.tradingview.com`. Referer alone is not enough. The scanner API does NOT need Origin or Referer.
|
|||
|
|
|
|||
|
|
Filter by exchange and type:
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# Exact match on NASDAQ:AAPL
|
|||
|
|
result = symbol_search("AAPL", exchange="NASDAQ", type_filter="stock")
|
|||
|
|
# Returns 1 result — exact symbol only when exchange is specified
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### News headlines for a symbol
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import json, urllib.request
|
|||
|
|
|
|||
|
|
def get_news(symbol, limit=20):
|
|||
|
|
# symbol format: "NASDAQ:AAPL", "NYSE:TSLA"
|
|||
|
|
url = (
|
|||
|
|
f"https://news-headlines.tradingview.com/v2/view/headlines/symbol"
|
|||
|
|
f"?symbol={symbol}&client=web&streaming=false&lang=en&limit={limit}"
|
|||
|
|
)
|
|||
|
|
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
|||
|
|
with urllib.request.urlopen(req, timeout=15) as r:
|
|||
|
|
data = json.loads(r.read())
|
|||
|
|
return data["items"] # list of news items
|
|||
|
|
|
|||
|
|
items = get_news("NASDAQ:AAPL", limit=10)
|
|||
|
|
# item keys: id, title, provider, sourceLogoId, published (unix ts),
|
|||
|
|
# source, urgency, link, permission, relatedSymbols, storyPath
|
|||
|
|
# example:
|
|||
|
|
# items[0]["title"] = "Apple Clears Major Legal Hurdle..."
|
|||
|
|
# items[0]["published"] = 1776472317 (unix timestamp)
|
|||
|
|
# items[0]["link"] = "https://stocktwits.com/..."
|
|||
|
|
# items[0]["relatedSymbols"] = [{"symbol": "NASDAQ:AAPL", "logoid": "apple"}]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
No auth or special headers needed. Returns up to 200 items per request.
|
|||
|
|
|
|||
|
|
### Published trading ideas feed
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import json, urllib.request
|
|||
|
|
|
|||
|
|
def get_ideas(sort="trending", page=1, symbol=None):
|
|||
|
|
# Valid sort values (others return 400):
|
|||
|
|
# "trending", "recent", "latest_popular", "week_popular",
|
|||
|
|
# "suggested", "recent_extended", "picked_time"
|
|||
|
|
url = f"https://www.tradingview.com/api/v1/ideas/?lang=en&sort={sort}&page={page}"
|
|||
|
|
if symbol:
|
|||
|
|
url += f"&symbol={symbol}" # e.g. "NASDAQ:AAPL"
|
|||
|
|
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
|||
|
|
with urllib.request.urlopen(req, timeout=15) as r:
|
|||
|
|
return json.loads(r.read())
|
|||
|
|
|
|||
|
|
data = get_ideas("trending")
|
|||
|
|
# data["count"] = 1000 (always 1000 — soft cap)
|
|||
|
|
# data["page_size"] = 20
|
|||
|
|
# data["page_count"] = 50
|
|||
|
|
# data["next"] = "https://www.tradingview.com/api/v1/ideas/?page=2"
|
|||
|
|
# data["results"] = list of idea objects
|
|||
|
|
|
|||
|
|
idea = data["results"][0]
|
|||
|
|
# idea keys: id, name, description, created_at, chart_url, views_count,
|
|||
|
|
# likes_count, comments_count, is_video, is_education, is_hot,
|
|||
|
|
# symbol (dict with name/exchange/type/interval/direction),
|
|||
|
|
# user (dict with username/is_pro/badges), image (big/middle URLs)
|
|||
|
|
# idea["symbol"]["direction"]: 1=long, 2=short, 0=neutral
|
|||
|
|
|
|||
|
|
# Filter by symbol:
|
|||
|
|
aapl_ideas = get_ideas(symbol="NASDAQ:AAPL")
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## API summary table
|
|||
|
|
|
|||
|
|
| Endpoint | Auth | Headers needed | Speed |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| `scanner.tradingview.com/{market}/scan` | None | None | ~200ms |
|
|||
|
|
| `symbol-search.tradingview.com/symbol_search/v3/` | None | `Origin: https://www.tradingview.com` | ~150ms |
|
|||
|
|
| `symbol-search.tradingview.com/symbol_search/` (v1) | None | `Origin: https://www.tradingview.com` | ~100ms |
|
|||
|
|
| `news-headlines.tradingview.com/v2/view/headlines/symbol` | None | None | ~400ms |
|
|||
|
|
| `www.tradingview.com/api/v1/ideas/` | None | None | ~300ms |
|
|||
|
|
| `data.tradingview.com/quotes/` | None | None | **Dead** — connection refused |
|
|||
|
|
| `economic-calendar.tradingview.com/events` | Yes | — | HTTP 403 |
|
|||
|
|
|
|||
|
|
## Gotchas
|
|||
|
|
|
|||
|
|
**Scanner `range` is half-open**: `[0, 10]` returns rows 0–9 (10 rows total). `[10, 20]` for the next page.
|
|||
|
|
|
|||
|
|
**Column order is critical**: The `"d"` array in each result row is positional — it exactly mirrors your `"columns"` array. Always zip them: `dict(zip(columns, item["d"]))`.
|
|||
|
|
|
|||
|
|
**`data.tradingview.com/quotes/` is dead**: The URL `https://data.tradingview.com/quotes/?symbols=NASDAQ:AAPL` closes the connection without a response. Use the scanner API instead for real-time quotes.
|
|||
|
|
|
|||
|
|
**Scanner needs no Referer**: `scanner.tradingview.com` works with just `User-Agent`. The symbol-search subdomain checks `Origin` (CORS enforcement on the server side).
|
|||
|
|
|
|||
|
|
**Symbol search highlights**: The v3 endpoint wraps matched text in `<em>` tags (e.g. `"<em>AAPL</em>"`). Strip them: `re.sub(r'</?em>', '', symbol["symbol"])`.
|
|||
|
|
|
|||
|
|
**Ideas sort validation**: Only specific values work. `"sort=popular"` returns 400. Use `"trending"`, `"recent"`, `"latest_popular"`, `"week_popular"`, `"suggested"`.
|
|||
|
|
|
|||
|
|
**Ideas count cap**: The API always reports `count=1000` regardless of actual corpus size. With `page_size=20`, max pages is 50.
|
|||
|
|
|
|||
|
|
**Scanner server is AWS CloudFront** (`X-Amz-Cf-Pop` header) with a custom `Server: tv` — no Cloudflare. No anti-bot on the scanner subdomain. Main `www.tradingview.com` is a React SPA with `window.initData = {}` (empty — no embedded data). All data is loaded via API calls after hydration.
|
|||
|
|
|
|||
|
|
**Rate limits**: No 429s observed in testing. 5 concurrent scanner calls complete in ~1s. Symbol search returns `symbols_remaining` in the response (counts against some quota — varies 90–180 across calls but never blocks). Observed no blocking after 15 rapid calls in a row.
|
|||
|
|
|
|||
|
|
**Sector names**: Use TradingView's own taxonomy, not GICS. "Technology" does not exist — use `"Electronic Technology"` (hardware/semis) or `"Technology Services"` (software/internet).
|
|||
|
|
|
|||
|
|
## When to use the browser
|
|||
|
|
|
|||
|
|
The charting UI (`/chart/`), symbol detail pages (`/symbols/NASDAQ-AAPL/`), and the ideas page (`/ideas/`) are React SPAs — their visible data comes from the APIs above, not embedded HTML. Use browser + JS extraction only if you need visual chart screenshots or data from auth-gated pages (watchlists, portfolio, paper trading).
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# Only if you need a chart screenshot:
|
|||
|
|
goto_url("https://www.tradingview.com/chart/?symbol=NASDAQ:AAPL")
|
|||
|
|
wait_for_load()
|
|||
|
|
wait(3) # chart renders asynchronously after readyState
|
|||
|
|
capture_screenshot("/tmp/aapl_chart.png", full=False)
|
|||
|
|
```
|