364 lines
16 KiB
Markdown
364 lines
16 KiB
Markdown
|
# Quora — Data Extraction
|
||
|
|
|
||
|
|
`https://www.quora.com` — Q&A platform. One reliable access path: `http_get` with a Chrome UA against question, answer, topic, and profile pages. Quora SSR-renders all public data into `window.ansFrontendGlobals.data.inlineQueryResults` via `.push()` calls. No browser needed for read-only tasks.
|
||
|
|
|
||
|
|
## Do this first: pick your access path
|
||
|
|
|
||
|
|
| Goal | Best approach | Latency |
|
||
|
|
|------|--------------|---------|
|
||
|
|
| Question metadata + first ~3 ranked answers | `http_get` question page + parse push payloads | ~600ms |
|
||
|
|
| Single answer (full text + upvotes + views) | `http_get` answer permalink | ~400ms |
|
||
|
|
| Answer count for a question | question page, payload with `answerCount` | same request as above |
|
||
|
|
| Topic metadata (id, name, follower count) | `http_get` topic page + parse push payloads | ~400ms |
|
||
|
|
| User profile (name, follower/following, credential) | `http_get` profile page + parse push payloads | ~500ms |
|
||
|
|
| Keyword search results | NOT available via http_get — server returns no result data | N/A |
|
||
|
|
|
||
|
|
**Never use a browser for read-only Quora tasks.** All question, answer, topic, and profile data is server-rendered. Browser is only needed for authenticated actions (posting, upvoting, following) or for getting more than the first ~3 answers on a question page (the rest load via XHR pagination).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## UA requirement: Chrome or Firefox — NOT bare Mozilla/5.0
|
||
|
|
|
||
|
|
```
|
||
|
|
bare "Mozilla/5.0" -> HTTP 403
|
||
|
|
Googlebot UA -> HTTP 403
|
||
|
|
Chrome UA -> HTTP 200 (confirmed working)
|
||
|
|
Firefox UA -> HTTP 200 (confirmed working)
|
||
|
|
```
|
||
|
|
|
||
|
|
Use this header bundle for all requests:
|
||
|
|
|
||
|
|
```python
|
||
|
|
import urllib.request, gzip, json, re
|
||
|
|
|
||
|
|
CHROME_UA = (
|
||
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||
|
|
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||
|
|
"Chrome/123.0.0.0 Safari/537.36"
|
||
|
|
)
|
||
|
|
|
||
|
|
def quora_get(url):
|
||
|
|
"""Fetch any public Quora page. Returns HTML string.
|
||
|
|
Requires Chrome/Firefox UA — bare Mozilla/5.0 returns 403.
|
||
|
|
"""
|
||
|
|
req = urllib.request.Request(url, headers={
|
||
|
|
"User-Agent": CHROME_UA,
|
||
|
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||
|
|
"Accept-Encoding": "gzip",
|
||
|
|
"Accept-Language": "en-US,en;q=0.9",
|
||
|
|
})
|
||
|
|
with urllib.request.urlopen(req, timeout=20) as r:
|
||
|
|
data = r.read()
|
||
|
|
if r.headers.get("Content-Encoding") == "gzip":
|
||
|
|
data = gzip.decompress(data)
|
||
|
|
return data.decode()
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## The data format: `ansFrontendGlobals.data.inlineQueryResults`
|
||
|
|
|
||
|
|
Quora SSR embeds all page data as a series of `.push("...")` calls inside `<script>` blocks. Each call pushes a JSON-encoded string (with escaped quotes) into `inlineQueryResults`. There are no JSON-LD blocks, no `__NEXT_DATA__`, no React hydration state — only these push calls.
|
||
|
|
|
||
|
|
```python
|
||
|
|
def extract_quora_payloads(html):
|
||
|
|
"""Extract and parse all push() payloads from a Quora page.
|
||
|
|
Returns list of dicts (already decoded from double JSON encoding).
|
||
|
|
"""
|
||
|
|
raw_payloads = re.findall(r'\.push\("((?:[^"\\]|\\.)*)"\)', html)
|
||
|
|
results = []
|
||
|
|
for raw in raw_payloads:
|
||
|
|
try:
|
||
|
|
# Two levels of encoding: outer JS string escape, inner JSON
|
||
|
|
inner = json.loads('"' + raw + '"') # decode JS string escaping
|
||
|
|
results.append(json.loads(inner)) # decode actual JSON
|
||
|
|
except Exception:
|
||
|
|
pass
|
||
|
|
return results
|
||
|
|
```
|
||
|
|
|
||
|
|
A question page returns **16 payloads**. A profile or topic page returns **3 payloads**. The payloads that matter are identified by their `data` keys, not by position (positions are stable across requests for the same page type, but best to key on content).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Path 1: Question page — metadata + first answers (fastest)
|
||
|
|
|
||
|
|
```python
|
||
|
|
def quora_question(url):
|
||
|
|
"""
|
||
|
|
Scrape a Quora question page.
|
||
|
|
Returns:
|
||
|
|
question: {qid, id, title, url, slug, topics}
|
||
|
|
answers: list of answer dicts (first ~3 ranked answers only)
|
||
|
|
answer_count: total answer count (all answers, not just loaded)
|
||
|
|
related_questions: list of question title strings
|
||
|
|
Only the first ~3 highest-ranked answers are SSR'd.
|
||
|
|
The rest require XHR pagination (browser or session cookies needed).
|
||
|
|
"""
|
||
|
|
html = quora_get(url)
|
||
|
|
payloads = extract_quora_payloads(html)
|
||
|
|
|
||
|
|
def spans_to_text(json_str):
|
||
|
|
"""Quora stores all text as serialized span objects."""
|
||
|
|
try:
|
||
|
|
doc = json.loads(json_str)
|
||
|
|
parts = []
|
||
|
|
for sec in doc.get('sections', []):
|
||
|
|
for span in sec.get('spans', []):
|
||
|
|
if span.get('text'):
|
||
|
|
parts.append(span['text'])
|
||
|
|
parts.append('\n')
|
||
|
|
return ''.join(parts).strip()
|
||
|
|
except Exception:
|
||
|
|
return json_str
|
||
|
|
|
||
|
|
def author_display_name(author_dict):
|
||
|
|
names = author_dict.get('names', [])
|
||
|
|
if names:
|
||
|
|
n = names[0]
|
||
|
|
return f"{n.get('givenName', '')} {n.get('familyName', '')}".strip()
|
||
|
|
return None
|
||
|
|
|
||
|
|
result = {'question': {}, 'answers': [], 'answer_count': None, 'related_questions': []}
|
||
|
|
|
||
|
|
for payload in payloads:
|
||
|
|
data = payload.get('data', payload)
|
||
|
|
|
||
|
|
# Question metadata — keyed by presence of 'qid' inside 'question'
|
||
|
|
if 'question' in data and isinstance(data['question'], dict):
|
||
|
|
q = data['question']
|
||
|
|
if q.get('qid') and not result['question']:
|
||
|
|
result['question'] = {
|
||
|
|
'qid': q.get('qid'),
|
||
|
|
'id': q.get('id'),
|
||
|
|
'title': spans_to_text(q.get('title', '')),
|
||
|
|
'url': q.get('url'),
|
||
|
|
'slug': q.get('slug'),
|
||
|
|
'topics': [t['name'] for t in q.get('navigationTopics', [])],
|
||
|
|
}
|
||
|
|
|
||
|
|
# Total answer count — keyed by 'answerCount'
|
||
|
|
if 'answerCount' in data:
|
||
|
|
result['answer_count'] = data['answerCount']
|
||
|
|
rq = (data.get('bottomRelatedQuestionsInfo') or {}).get('relatedQuestions', [])
|
||
|
|
result['related_questions'] = [spans_to_text(r['title']) for r in rq]
|
||
|
|
|
||
|
|
# Answer nodes — keyed by node.__typename == 'QuestionAnswerItem2'
|
||
|
|
node = data.get('node', {})
|
||
|
|
if isinstance(node, dict) and node.get('__typename') == 'QuestionAnswerItem2':
|
||
|
|
answer = node.get('answer', {})
|
||
|
|
if answer.get('aid'):
|
||
|
|
a_author = answer.get('author') or {}
|
||
|
|
cred = answer.get('authorCredential') or {}
|
||
|
|
result['answers'].append({
|
||
|
|
'aid': answer.get('aid'),
|
||
|
|
'index': node.get('index'),
|
||
|
|
'author_name': author_display_name(a_author),
|
||
|
|
'author_profile': a_author.get('profileUrl'),
|
||
|
|
'author_uid': a_author.get('uid'),
|
||
|
|
'author_credential': cred.get('translatedString'),
|
||
|
|
'num_upvotes': answer.get('numUpvotes'),
|
||
|
|
'num_views': answer.get('numViews'),
|
||
|
|
'num_shares': answer.get('numShares'),
|
||
|
|
'num_comments': answer.get('numDisplayComments'),
|
||
|
|
'creation_time_us': answer.get('creationTime'), # microseconds since epoch
|
||
|
|
'viewer_has_access': answer.get('viewerHasAccess'),
|
||
|
|
'perma_url': answer.get('permaUrl'),
|
||
|
|
'text': spans_to_text(answer.get('content', '{}')),
|
||
|
|
})
|
||
|
|
|
||
|
|
return result
|
||
|
|
```
|
||
|
|
|
||
|
|
### Example output
|
||
|
|
|
||
|
|
```python
|
||
|
|
result = quora_question("https://www.quora.com/What-is-the-meaning-of-life")
|
||
|
|
|
||
|
|
# result['question']:
|
||
|
|
# {
|
||
|
|
# 'qid': 2861,
|
||
|
|
# 'id': 'UXVlc3Rpb25AMDoyODYx',
|
||
|
|
# 'title': 'What is the meaning of life?',
|
||
|
|
# 'url': '/What-is-the-meaning-of-life',
|
||
|
|
# 'slug': 'What-is-the-meaning-of-life',
|
||
|
|
# 'topics': ['Philosophy', 'The Big Unanswered Questions', 'Meaning of Life', ...]
|
||
|
|
# }
|
||
|
|
|
||
|
|
# result['answer_count']: 413
|
||
|
|
|
||
|
|
# result['answers'][0]:
|
||
|
|
# {
|
||
|
|
# 'aid': 2779675,
|
||
|
|
# 'index': 1,
|
||
|
|
# 'author_name': 'Shubhankar Srivastava',
|
||
|
|
# 'author_profile': '/profile/Shubhankar-Srivastava',
|
||
|
|
# 'author_uid': 5381038,
|
||
|
|
# 'author_credential': 'works at D. E. Shaw',
|
||
|
|
# 'num_upvotes': 589,
|
||
|
|
# 'num_views': 24085,
|
||
|
|
# 'num_shares': 0,
|
||
|
|
# 'num_comments': 8,
|
||
|
|
# 'creation_time_us': 1373364681312036, # divide by 1e6 for seconds
|
||
|
|
# 'viewer_has_access': True,
|
||
|
|
# 'perma_url': '/What-is-the-meaning-of-life/answer/Shubhankar-Srivastava',
|
||
|
|
# 'text': 'Every morning in Africa, a deer wakes up...'
|
||
|
|
# }
|
||
|
|
```
|
||
|
|
|
||
|
|
### Convert creation_time_us to datetime
|
||
|
|
|
||
|
|
```python
|
||
|
|
from datetime import datetime, timezone
|
||
|
|
ts_sec = result['answers'][0]['creation_time_us'] / 1_000_000
|
||
|
|
dt = datetime.fromtimestamp(ts_sec, tz=timezone.utc)
|
||
|
|
# datetime(2013, 7, 9, 9, 31, 21, tzinfo=timezone.utc)
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Path 2: Single answer permalink
|
||
|
|
|
||
|
|
Fetching `quora.com/{question-slug}/answer/{author-slug}` directly returns only that one answer's full data in 3 payloads instead of 16. Use this when you already know the answer URL.
|
||
|
|
|
||
|
|
```python
|
||
|
|
def quora_answer(answer_url):
|
||
|
|
"""
|
||
|
|
Fetch a single answer by its permalink.
|
||
|
|
URL format: https://www.quora.com/{question-slug}/answer/{author-profile-slug}
|
||
|
|
Returns answer dict with: aid, num_upvotes, num_views, text, author info.
|
||
|
|
"""
|
||
|
|
html = quora_get(answer_url)
|
||
|
|
payloads = extract_quora_payloads(html)
|
||
|
|
|
||
|
|
for payload in payloads:
|
||
|
|
data = payload.get('data', {})
|
||
|
|
if 'answer' in data and isinstance(data['answer'], dict):
|
||
|
|
a = data['answer']
|
||
|
|
author = a.get('author') or {}
|
||
|
|
names = author.get('names', [{}])
|
||
|
|
n = names[0] if names else {}
|
||
|
|
return {
|
||
|
|
'aid': a.get('aid'),
|
||
|
|
'num_upvotes': a.get('numUpvotes'),
|
||
|
|
'num_views': a.get('numViews'),
|
||
|
|
'author_name': f"{n.get('givenName','')} {n.get('familyName','')}".strip(),
|
||
|
|
'author_uid': author.get('uid'),
|
||
|
|
'text': _spans_to_text(a.get('content', '{}')),
|
||
|
|
}
|
||
|
|
return {}
|
||
|
|
|
||
|
|
# Example:
|
||
|
|
# quora_answer("https://www.quora.com/What-is-the-meaning-of-life/answer/Pararth-Shah")
|
||
|
|
# -> {'aid': 4734237, 'num_upvotes': 234, 'num_views': 100643, 'author_name': 'Pararth Shah', ...}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Path 3: Topic page
|
||
|
|
|
||
|
|
```python
|
||
|
|
def quora_topic(topic_url):
|
||
|
|
"""
|
||
|
|
Fetch topic metadata from a Quora topic page.
|
||
|
|
URL format: https://www.quora.com/topic/{topic-slug}
|
||
|
|
Returns: tid, name, num_followers, url, is_following, has_leaderboard.
|
||
|
|
NOTE: The topic page itself only renders topic metadata, NOT the question feed.
|
||
|
|
Question feed requires browser (XHR-loaded via React).
|
||
|
|
"""
|
||
|
|
html = quora_get(topic_url)
|
||
|
|
payloads = extract_quora_payloads(html)
|
||
|
|
|
||
|
|
for payload in payloads:
|
||
|
|
data = payload.get('data', {})
|
||
|
|
if 'topic' in data and isinstance(data['topic'], dict):
|
||
|
|
t = data['topic']
|
||
|
|
return {
|
||
|
|
'tid': t.get('tid'),
|
||
|
|
'id': t.get('id'),
|
||
|
|
'name': t.get('name'),
|
||
|
|
'url': t.get('url'),
|
||
|
|
'num_followers': t.get('numFollowers'),
|
||
|
|
'is_following': t.get('isFollowing'),
|
||
|
|
'has_leaderboard': t.get('hasLeaderboard'),
|
||
|
|
'photo_url': t.get('photoUrl'),
|
||
|
|
'is_locked': t.get('isLocked'),
|
||
|
|
}
|
||
|
|
return {}
|
||
|
|
|
||
|
|
# Example:
|
||
|
|
# quora_topic("https://www.quora.com/topic/Python-programming-language")
|
||
|
|
# -> {'tid': 13292, 'name': 'Python Programming Language', 'num_followers': 10, ...}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Path 4: User profile page
|
||
|
|
|
||
|
|
```python
|
||
|
|
def quora_profile(profile_url):
|
||
|
|
"""
|
||
|
|
Fetch user profile data from https://www.quora.com/profile/{username}
|
||
|
|
Returns: uid, name, credential, follower_count, following_count, profile_image_url.
|
||
|
|
"""
|
||
|
|
html = quora_get(profile_url)
|
||
|
|
payloads = extract_quora_payloads(html)
|
||
|
|
|
||
|
|
for payload in payloads:
|
||
|
|
data = payload.get('data', {})
|
||
|
|
if 'user' in data and isinstance(data['user'], dict):
|
||
|
|
u = data['user']
|
||
|
|
names = u.get('names', [{}])
|
||
|
|
n = names[0] if names else {}
|
||
|
|
cred = u.get('profileCredential') or {}
|
||
|
|
return {
|
||
|
|
'uid': u.get('uid'),
|
||
|
|
'id': u.get('id'),
|
||
|
|
'name': f"{n.get('givenName','')} {n.get('familyName','')}".strip(),
|
||
|
|
'profile_url': u.get('profileUrl'),
|
||
|
|
'follower_count': u.get('followerCount'),
|
||
|
|
'following_count': u.get('followingCount'),
|
||
|
|
'profile_image': u.get('profileImageUrl'),
|
||
|
|
'credential': cred.get('experience'),
|
||
|
|
'is_verified': u.get('isVerified'),
|
||
|
|
'is_anon': u.get('isAnon'),
|
||
|
|
'is_ai_account': u.get('isAiAccount'),
|
||
|
|
'deactivated': u.get('deactivated'),
|
||
|
|
}
|
||
|
|
return {}
|
||
|
|
|
||
|
|
# Example:
|
||
|
|
# quora_profile("https://www.quora.com/profile/Pararth-Shah")
|
||
|
|
# -> {'uid': 4683832, 'name': 'Pararth Shah', 'follower_count': 5154,
|
||
|
|
# 'following_count': 83, 'credential': 'Unfinished symphony.', ...}
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Gotchas
|
||
|
|
|
||
|
|
- **Bare Mozilla/5.0 UA returns HTTP 403** — Always use a full Chrome or Firefox UA string. The default `http_get` helper's `"User-Agent": "Mozilla/5.0"` will be blocked. Do not use `http_get` directly; use the `quora_get` wrapper above.
|
||
|
|
|
||
|
|
- **Googlebot UA returns HTTP 403** — Quora blocks crawler UAs. Only real browser UAs work.
|
||
|
|
|
||
|
|
- **Double JSON encoding** — Each `.push()` argument is a JavaScript string literal containing JSON. To parse: first `json.loads('"' + raw + '"')` to decode the JS string escaping (converts `\\"` to `"`), then `json.loads(inner)` to parse the actual JSON object. Skipping either step produces parse errors.
|
||
|
|
|
||
|
|
- **All text fields are serialized span objects** — `question.title`, `answer.content`, `user.descriptionQtextDocument.legacyJson`, etc. are all JSON strings containing a `{"sections": [{"spans": [...]}]}` document, not plain text. Always parse through `spans_to_text()`.
|
||
|
|
|
||
|
|
- **Question page only SSR's the first ~3 answers** — The `answers` list in the result will contain at most 3 entries (the top-ranked answers). The `answer_count` field shows the true total (e.g. 413). To get more answers you need browser-based XHR pagination (Quora sends additional answers via GraphQL calls that require session auth in practice).
|
||
|
|
|
||
|
|
- **`viewer_has_access: false` still includes full content** — Even when `viewerHasAccess` is `False` (answers from Quora+ Spaces / tribe-only content), the `content` field is still present in the SSR payload and the full text is readable. The flag only controls client-side gating in the browser.
|
||
|
|
|
||
|
|
- **`creation_time_us` is microseconds, not milliseconds** — Divide by `1_000_000` (not `1_000`) to get a Unix timestamp in seconds. Confirmed: `1373364681312036 / 1e6 = 1373364681.3` (July 2013).
|
||
|
|
|
||
|
|
- **`numFollowers` on topic pages may be 0 even for major topics** — The field reflects the logged-in user's follow state for some topics and appears to undercount. Treat as approximate.
|
||
|
|
|
||
|
|
- **Search pages do not yield result data** — `https://www.quora.com/search?q=...` returns 3 payloads with viewer/network info only — no search results in the SSR payload. Search results are loaded client-side and are not accessible via http_get.
|
||
|
|
|
||
|
|
- **Profile pages do not include the user's answer list** — The profile page SSR payload returns user metadata only. The list of a user's answers is loaded via XHR pagination. To get answers for a specific question, use the question URL directly.
|
||
|
|
|
||
|
|
- **IDs are base64-encoded Relay global IDs** — `id: "UXVlc3Rpb25AMDoyODYx"` decodes to `"Question@0:2861"`. The numeric `qid`/`uid`/`aid`/`tid` fields are more useful for constructing URLs and deduplication. Use `qid` and `aid` as stable identifiers.
|
||
|
|
|
||
|
|
- **`permaUrl` may be an absolute URL for Spaces answers** — Most answers have `permaUrl: "/Question-slug/answer/Author-Name"` (relative). Answers posted in a Quora Space have a full absolute URL like `"https://spacename.quora.com/Question-slug"`. Handle both forms.
|
||
|
|
|
||
|
|
- **No public REST or GraphQL API** — Quora's internal `graphql/gql_para_POST` endpoint requires a valid `quora-formkey` header derived from the session, making it inaccessible without a real authenticated session. The SSR push-payload approach is the only reliable unauthenticated path.
|