440 lines
18 KiB
Markdown
440 lines
18 KiB
Markdown
|
# Capterra — Scraping & Data Extraction
|
|||
|
|
|
|||
|
|
Field-tested against capterra.com on 2026-04-18. All code blocks validated with live requests.
|
|||
|
|
|
|||
|
|
## Do this first
|
|||
|
|
|
|||
|
|
**Use `User-Agent: ClaudeBot` — Capterra explicitly allows it in robots.txt and returns clean, pre-rendered Markdown instead of JavaScript-heavy HTML. No browser needed.**
|
|||
|
|
|
|||
|
|
Capterra serves a fully structured Markdown representation of every page to AI bots (`ClaudeBot`, `GPTBot`, `PerplexityBot`, `Anthropic-AI` are all listed as `Allow: /` in robots.txt). The Markdown format is far easier to parse than HTML.
|
|||
|
|
|
|||
|
|
With the default `Mozilla/5.0` UA (or any realistic browser UA), Capterra returns HTTP 403 with `Cf-Mitigated: challenge` — Cloudflare blocks all browser UA requests. There is no bypass via HTTP; those pages require a real browser session.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re, json
|
|||
|
|
|
|||
|
|
# Works everywhere:
|
|||
|
|
html = http_get(
|
|||
|
|
"https://www.capterra.com/p/135003/Slack/reviews/",
|
|||
|
|
headers={"User-Agent": "ClaudeBot"}
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
# Extract overall rating and review count from the Markdown header line "4.7 (24059)"
|
|||
|
|
m = re.search(r'^([\d.]+)\s+\(([\d,]+)\)$', html, re.MULTILINE)
|
|||
|
|
print(m.group(1), m.group(2)) # 4.7 24059
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Fastest approach: product summary in one call
|
|||
|
|
|
|||
|
|
All key metrics — overall rating, review count, sub-ratings, pagination — come from the `/reviews/` endpoint in a single request.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re, json
|
|||
|
|
|
|||
|
|
def get_product_summary(product_id, slug):
|
|||
|
|
"""
|
|||
|
|
Returns overall rating, review count, sub-ratings.
|
|||
|
|
product_id: Capterra numeric ID (e.g. 135003)
|
|||
|
|
slug: URL slug (e.g. 'Slack')
|
|||
|
|
"""
|
|||
|
|
url = f"https://www.capterra.com/p/{product_id}/{slug}/reviews/"
|
|||
|
|
html = http_get(url, headers={"User-Agent": "ClaudeBot"})
|
|||
|
|
|
|||
|
|
result = {"product_id": product_id, "slug": slug}
|
|||
|
|
|
|||
|
|
# Overall rating + review count from header line "4.7 (24059)"
|
|||
|
|
m = re.search(r'^([\d.]+)\s+\(([\d,]+)\)$', html, re.MULTILINE)
|
|||
|
|
if m:
|
|||
|
|
result["overall_rating"] = float(m.group(1))
|
|||
|
|
result["review_count"] = int(m.group(2).replace(",", ""))
|
|||
|
|
|
|||
|
|
# Page size and total pages from "Showing 1-25 of 24059 Reviews"
|
|||
|
|
showing = re.search(r"Showing\s+(\d+)[-–](\d+)\s+of\s+([\d,]+)\s+Reviews", html)
|
|||
|
|
if showing:
|
|||
|
|
result["per_page"] = int(showing.group(2))
|
|||
|
|
result["total_pages"] = (int(showing.group(3).replace(",", "")) + 24) // 25
|
|||
|
|
|
|||
|
|
# Sub-ratings: "Ease of use\n\n4.6" and "Customer Service\n\n4.4"
|
|||
|
|
lines = html.split("\n")
|
|||
|
|
for i, line in enumerate(lines):
|
|||
|
|
for label, key in [("Ease of use", "ease_of_use"), ("Customer Service", "customer_service")]:
|
|||
|
|
if line.strip() == label:
|
|||
|
|
for j in range(i + 1, min(i + 5, len(lines))):
|
|||
|
|
try:
|
|||
|
|
val = float(lines[j].strip())
|
|||
|
|
if 0 < val <= 5.0:
|
|||
|
|
result[key] = val
|
|||
|
|
break
|
|||
|
|
except ValueError:
|
|||
|
|
pass
|
|||
|
|
|
|||
|
|
return result
|
|||
|
|
|
|||
|
|
summary = get_product_summary(135003, "Slack")
|
|||
|
|
print(json.dumps(summary, indent=2))
|
|||
|
|
# {
|
|||
|
|
# "product_id": 135003,
|
|||
|
|
# "slug": "Slack",
|
|||
|
|
# "overall_rating": 4.7,
|
|||
|
|
# "review_count": 24059,
|
|||
|
|
# "per_page": 25,
|
|||
|
|
# "total_pages": 963,
|
|||
|
|
# "ease_of_use": 4.6,
|
|||
|
|
# "customer_service": 4.4
|
|||
|
|
# }
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Common workflows
|
|||
|
|
|
|||
|
|
### Get reviews (paginated)
|
|||
|
|
|
|||
|
|
25 reviews per page. Use `?page=N` for pagination.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re
|
|||
|
|
|
|||
|
|
def get_reviews_page(product_id, slug, page=1):
|
|||
|
|
"""
|
|||
|
|
Returns up to 25 reviews for one page.
|
|||
|
|
Total pages = ceil(review_count / 25).
|
|||
|
|
"""
|
|||
|
|
url = f"https://www.capterra.com/p/{product_id}/{slug}/reviews/?page={page}"
|
|||
|
|
html = http_get(url, headers={"User-Agent": "ClaudeBot"})
|
|||
|
|
|
|||
|
|
# Total review count from header
|
|||
|
|
m = re.search(r'^([\d.]+)\s+\(([\d,]+)\)$', html, re.MULTILINE)
|
|||
|
|
total = int(m.group(2).replace(",", "")) if m else 0
|
|||
|
|
|
|||
|
|
# Showing X-Y of Z
|
|||
|
|
showing = re.search(r"Showing\s+(\d+)[-–](\d+)\s+of\s+([\d,]+)\s+Reviews", html)
|
|||
|
|
|
|||
|
|
# Split by review title markers "### "Title""
|
|||
|
|
blocks = re.split(r'\n### "', html)
|
|||
|
|
reviews = []
|
|||
|
|
|
|||
|
|
for block in blocks[1:]:
|
|||
|
|
r = {}
|
|||
|
|
|
|||
|
|
# Title (up to closing quote)
|
|||
|
|
t = re.match(r'([^"]+)"', block)
|
|||
|
|
if t:
|
|||
|
|
r["title"] = t.group(1).strip()
|
|||
|
|
|
|||
|
|
# Date
|
|||
|
|
d = re.search(
|
|||
|
|
r"(January|February|March|April|May|June|July|August|September|October|November|December)\s+\d+,\s+\d{4}",
|
|||
|
|
block
|
|||
|
|
)
|
|||
|
|
if d:
|
|||
|
|
r["date"] = d.group(0)
|
|||
|
|
|
|||
|
|
# Overall rating for this review (first float 1.0–5.0 between blank lines)
|
|||
|
|
rm = re.search(r"\n\n([\d.]+)\n\n", block)
|
|||
|
|
if rm:
|
|||
|
|
val = float(rm.group(1))
|
|||
|
|
if 1.0 <= val <= 5.0:
|
|||
|
|
r["rating"] = val
|
|||
|
|
|
|||
|
|
# Pros
|
|||
|
|
pros = re.search(r"\nPros\n\n(.+?)(?=\n\nCons|\n\nReview Source|\n\nSwitched|\Z)", block, re.DOTALL)
|
|||
|
|
if pros:
|
|||
|
|
r["pros"] = pros.group(1).strip()
|
|||
|
|
|
|||
|
|
# Cons
|
|||
|
|
cons = re.search(r"\nCons\n\n(.+?)(?=\n\nReview Source|\n\nSwitched|\n\n##|\Z)", block, re.DOTALL)
|
|||
|
|
if cons:
|
|||
|
|
r["cons"] = cons.group(1).strip()
|
|||
|
|
|
|||
|
|
if r.get("title"):
|
|||
|
|
reviews.append(r)
|
|||
|
|
|
|||
|
|
return {
|
|||
|
|
"total": total,
|
|||
|
|
"page": page,
|
|||
|
|
"showing": f"{showing.group(1)}-{showing.group(2)} of {showing.group(3)}" if showing else None,
|
|||
|
|
"reviews": reviews,
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# Page 1
|
|||
|
|
result = get_reviews_page(135003, "Slack", page=1)
|
|||
|
|
print(f"Total reviews: {result['total']}, this page: {len(result['reviews'])}")
|
|||
|
|
# Total reviews: 24059, this page: 25
|
|||
|
|
|
|||
|
|
print(result["reviews"][0])
|
|||
|
|
# {'title': 'Love, love, love Slack!', 'date': 'April 14, 2026', 'rating': 5.0,
|
|||
|
|
# 'pros': '...', 'cons': '...'}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Scrape all reviews in bulk (parallel)
|
|||
|
|
|
|||
|
|
10 pages in ~2s with 5 workers. No rate limiting observed during testing.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re
|
|||
|
|
from concurrent.futures import ThreadPoolExecutor
|
|||
|
|
|
|||
|
|
UA = {"User-Agent": "ClaudeBot"}
|
|||
|
|
|
|||
|
|
def _fetch_page(args):
|
|||
|
|
product_id, slug, page = args
|
|||
|
|
url = f"https://www.capterra.com/p/{product_id}/{slug}/reviews/?page={page}"
|
|||
|
|
html = http_get(url, headers=UA)
|
|||
|
|
blocks = re.split(r'\n### "', html)
|
|||
|
|
reviews = []
|
|||
|
|
for block in blocks[1:]:
|
|||
|
|
r = {}
|
|||
|
|
t = re.match(r'([^"]+)"', block)
|
|||
|
|
if t: r["title"] = t.group(1).strip()
|
|||
|
|
d = re.search(r"(January|February|March|April|May|June|July|August|September|October|November|December)\s+\d+,\s+\d{4}", block)
|
|||
|
|
if d: r["date"] = d.group(0)
|
|||
|
|
rm = re.search(r"\n\n([\d.]+)\n\n", block)
|
|||
|
|
if rm:
|
|||
|
|
val = float(rm.group(1))
|
|||
|
|
if 1.0 <= val <= 5.0: r["rating"] = val
|
|||
|
|
pros = re.search(r"\nPros\n\n(.+?)(?=\n\nCons|\n\nReview Source|\n\nSwitched|\Z)", block, re.DOTALL)
|
|||
|
|
if pros: r["pros"] = pros.group(1).strip()
|
|||
|
|
cons = re.search(r"\nCons\n\n(.+?)(?=\n\nReview Source|\n\nSwitched|\n\n##|\Z)", block, re.DOTALL)
|
|||
|
|
if cons: r["cons"] = cons.group(1).strip()
|
|||
|
|
if r.get("title"): reviews.append(r)
|
|||
|
|
return reviews
|
|||
|
|
|
|||
|
|
def get_all_reviews(product_id, slug, max_pages=None, workers=5):
|
|||
|
|
"""Fetch all reviews in parallel. max_pages=None fetches everything."""
|
|||
|
|
# First: get total pages
|
|||
|
|
summary_html = http_get(
|
|||
|
|
f"https://www.capterra.com/p/{product_id}/{slug}/reviews/",
|
|||
|
|
headers=UA
|
|||
|
|
)
|
|||
|
|
m = re.search(r'^([\d.]+)\s+\(([\d,]+)\)$', summary_html, re.MULTILINE)
|
|||
|
|
total = int(m.group(2).replace(",", "")) if m else 0
|
|||
|
|
total_pages = (total + 24) // 25
|
|||
|
|
pages = range(1, (max_pages or total_pages) + 1)
|
|||
|
|
|
|||
|
|
tasks = [(product_id, slug, p) for p in pages]
|
|||
|
|
all_reviews = []
|
|||
|
|
with ThreadPoolExecutor(max_workers=workers) as ex:
|
|||
|
|
for batch in ex.map(_fetch_page, tasks):
|
|||
|
|
all_reviews.extend(batch)
|
|||
|
|
return all_reviews
|
|||
|
|
|
|||
|
|
# Fetch first 50 reviews (2 pages) in parallel
|
|||
|
|
reviews = get_all_reviews(135003, "Slack", max_pages=2, workers=2)
|
|||
|
|
print(f"Fetched {len(reviews)} reviews")
|
|||
|
|
# Fetched 50 reviews
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Get a product's full overview (rating breakdown, sentiment, pricing)
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re, json
|
|||
|
|
|
|||
|
|
def get_product_overview(product_id, slug):
|
|||
|
|
"""Rating breakdown, sentiment, starting price from the product page."""
|
|||
|
|
url = f"https://www.capterra.com/p/{product_id}/{slug}/"
|
|||
|
|
html = http_get(url, headers={"User-Agent": "ClaudeBot"})
|
|||
|
|
|
|||
|
|
result = {}
|
|||
|
|
|
|||
|
|
# Overall rating and review count from the reviews section
|
|||
|
|
# Appears as "\n4.7\n\nBased on 24,059 reviews\n"
|
|||
|
|
m = re.search(r'\n([\d.]+)\n\nBased on ([\d,]+) reviews\n', html)
|
|||
|
|
if m:
|
|||
|
|
result["overall_rating"] = float(m.group(1))
|
|||
|
|
result["review_count"] = int(m.group(2).replace(",", ""))
|
|||
|
|
|
|||
|
|
# Rating breakdown: "5(17268)\n\n4(5708)\n\n3(907)\n\n2(128)\n\n1(48)"
|
|||
|
|
breakdown = re.findall(r'\b([1-5])\((\d+)\)', html)
|
|||
|
|
if breakdown:
|
|||
|
|
result["rating_breakdown"] = {int(s): int(c) for s, c in breakdown if 1 <= int(s) <= 5}
|
|||
|
|
|
|||
|
|
# Sentiment: "Positive\n\n96%\n\nNeutral\n\n4%\n\nNegative\n\n1%"
|
|||
|
|
for label, key in [("Positive", "sentiment_positive"), ("Neutral", "sentiment_neutral"), ("Negative", "sentiment_negative")]:
|
|||
|
|
sm = re.search(rf'{label}\s*\n+\s*(\d+)%', html)
|
|||
|
|
if sm:
|
|||
|
|
result[key] = int(sm.group(1))
|
|||
|
|
|
|||
|
|
# Starting price ("Starting price\n\n$8.75\n\nPer User")
|
|||
|
|
pm = re.search(r'Starting price\s*\n+\$?([\d.]+)', html)
|
|||
|
|
if pm:
|
|||
|
|
result["starting_price_usd"] = float(pm.group(1))
|
|||
|
|
|
|||
|
|
# Categories ("What is X used for?" links)
|
|||
|
|
cats = re.findall(r'\[([^\]]+)\]\(https://www\.capterra\.com/([a-z-]+-software)/\)', html[:3000])
|
|||
|
|
if cats:
|
|||
|
|
result["categories"] = [name for name, _ in cats]
|
|||
|
|
|
|||
|
|
# Sub-ratings from product page
|
|||
|
|
for label, key in [("Value for money", "value_for_money"), ("Features", "features_rating")]:
|
|||
|
|
sub = re.search(rf'{label}\s*\n+\s*([\d.]+)', html)
|
|||
|
|
if sub:
|
|||
|
|
try:
|
|||
|
|
val = float(sub.group(1))
|
|||
|
|
if 0 < val <= 5.0:
|
|||
|
|
result[key] = val
|
|||
|
|
except ValueError:
|
|||
|
|
pass
|
|||
|
|
|
|||
|
|
return result
|
|||
|
|
|
|||
|
|
overview = get_product_overview(135003, "Slack")
|
|||
|
|
print(json.dumps(overview, indent=2))
|
|||
|
|
# {
|
|||
|
|
# "overall_rating": 4.7,
|
|||
|
|
# "review_count": 24059,
|
|||
|
|
# "rating_breakdown": {"5": 17268, "4": 5708, "3": 907, "2": 128, "1": 48},
|
|||
|
|
# "sentiment_positive": 96,
|
|||
|
|
# "sentiment_neutral": 4,
|
|||
|
|
# "sentiment_negative": 1,
|
|||
|
|
# "starting_price_usd": 8.75,
|
|||
|
|
# "categories": ["Team Communication", "Collaboration", "Remote Work"]
|
|||
|
|
# }
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Browse a software category
|
|||
|
|
|
|||
|
|
Each category page returns up to 40 products on page 1, then ~24–25 per subsequent page. Pagination works via `?page=N`.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re
|
|||
|
|
|
|||
|
|
def get_category_products(category_slug, page=1):
|
|||
|
|
"""
|
|||
|
|
List products in a Capterra category.
|
|||
|
|
category_slug examples: 'project-management-software', 'crm-software', 'accounting-software'
|
|||
|
|
Full list: https://www.capterra.com/categories/
|
|||
|
|
"""
|
|||
|
|
url = f"https://www.capterra.com/{category_slug}/"
|
|||
|
|
if page > 1:
|
|||
|
|
url = f"https://www.capterra.com/{category_slug}/?page={page}"
|
|||
|
|
html = http_get(url, headers={"User-Agent": "ClaudeBot"})
|
|||
|
|
|
|||
|
|
# Ratings: [4.6 (5732)](https://www.capterra.com/p/147657/monday-com/reviews/)
|
|||
|
|
raw = re.findall(
|
|||
|
|
r'\[([\d.]+)\s+\(([\d,]+)\)\]\(https://www\.capterra\.com/p/(\d+)/([^/]+)/reviews/\)',
|
|||
|
|
html
|
|||
|
|
)
|
|||
|
|
# Product names from "Learn more about X" links
|
|||
|
|
names = {pid: name for name, pid in re.findall(
|
|||
|
|
r'\[Learn more about ([^\]]+)\]\(https://www\.capterra\.com/p/(\d+)/[^/]+/\)', html
|
|||
|
|
)}
|
|||
|
|
|
|||
|
|
items, seen = [], set()
|
|||
|
|
for rating, review_count, pid, slug in raw:
|
|||
|
|
if pid not in seen:
|
|||
|
|
seen.add(pid)
|
|||
|
|
items.append({
|
|||
|
|
"product_id": int(pid),
|
|||
|
|
"name": names.get(pid, slug),
|
|||
|
|
"slug": slug,
|
|||
|
|
"overall_rating": float(rating),
|
|||
|
|
"review_count": int(review_count.replace(",", "")),
|
|||
|
|
"product_url": f"https://www.capterra.com/p/{pid}/{slug}/",
|
|||
|
|
"reviews_url": f"https://www.capterra.com/p/{pid}/{slug}/reviews/",
|
|||
|
|
})
|
|||
|
|
return items
|
|||
|
|
|
|||
|
|
products = get_category_products("project-management-software", page=1)
|
|||
|
|
for p in products[:3]:
|
|||
|
|
print(f"{p['name']}: {p['overall_rating']} ({p['review_count']} reviews)")
|
|||
|
|
# monday.com: 4.6 (5732 reviews)
|
|||
|
|
# Jira: 4.4 (15325 reviews)
|
|||
|
|
# Celoxis: 4.4 (327 reviews)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Get all 1000+ software categories
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from helpers import http_get
|
|||
|
|
import re
|
|||
|
|
|
|||
|
|
def get_all_categories():
|
|||
|
|
"""Returns list of {name, slug} for all ~1003 Capterra software categories."""
|
|||
|
|
html = http_get("https://www.capterra.com/categories/", headers={"User-Agent": "ClaudeBot"})
|
|||
|
|
cats = re.findall(r'\[([^\]]+)\]\(https://www\.capterra\.com/([a-z-]+-software)/\)', html)
|
|||
|
|
return [{"name": name, "slug": slug} for name, slug in cats]
|
|||
|
|
|
|||
|
|
categories = get_all_categories()
|
|||
|
|
print(f"{len(categories)} categories") # 1003
|
|||
|
|
print(categories[:3])
|
|||
|
|
# [{'name': 'AB Testing', 'slug': 'ab-testing-software'},
|
|||
|
|
# {'name': 'Absence Management', 'slug': 'absence-management-software'}, ...]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## URL patterns
|
|||
|
|
|
|||
|
|
| Page type | URL pattern |
|
|||
|
|
|-----------|-------------|
|
|||
|
|
| Product overview | `https://www.capterra.com/p/{id}/{Slug}/` |
|
|||
|
|
| Product reviews | `https://www.capterra.com/p/{id}/{Slug}/reviews/` |
|
|||
|
|
| Reviews page N | `https://www.capterra.com/p/{id}/{Slug}/reviews/?page={N}` |
|
|||
|
|
| Reviews (alt) | `https://www.capterra.com/reviews/{id}/{Slug}/` |
|
|||
|
|
| Category listing | `https://www.capterra.com/{category}-software/` |
|
|||
|
|
| Category page N | `https://www.capterra.com/{category}-software/?page={N}` |
|
|||
|
|
| All categories | `https://www.capterra.com/categories/` |
|
|||
|
|
| Product pricing | `https://www.capterra.com/p/{id}/{Slug}/pricing/` |
|
|||
|
|
| Product alternatives | `https://www.capterra.com/p/{id}/{Slug}/alternatives/` |
|
|||
|
|
| Compare A vs B | `https://www.capterra.com/compare/{id_a}-{id_b}/{Slug_a}-vs-{Slug_b}` |
|
|||
|
|
|
|||
|
|
**Finding a product's ID:** Look in the URL of any product listing in a category page. The pattern `https://www.capterra.com/p/{id}/{Slug}/reviews/` appears in every category listing as the link target for each rating badge. The slug is case-sensitive in practice (e.g. `Slack`, not `slack`).
|
|||
|
|
|
|||
|
|
Product IDs are stable numeric identifiers. Note that the same software vendor may have multiple product IDs under different names/versions. Always find the ID from a category search rather than guessing.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Anti-bot measures
|
|||
|
|
|
|||
|
|
- **Cloudflare is active on all routes** (`Server: cloudflare`, `CF-RAY` present in all response headers).
|
|||
|
|
- **Browser UAs (Chrome, Firefox, Safari) return HTTP 403** with `Cf-Mitigated: challenge` regardless of how complete the headers are. There is no HTTP-only bypass.
|
|||
|
|
- **`ClaudeBot` UA bypasses Cloudflare** and receives clean pre-rendered Markdown. Capterra explicitly allows it in `robots.txt` via `User-agent: ClaudeBot / Allow: /`. This is a deliberate AI-accessibility feature.
|
|||
|
|
- **Other AI bot UAs that also work**: `GPTBot`, `PerplexityBot` (also in `robots.txt` Allow list). `Anthropic-AI` was tested and returns 403 — only `ClaudeBot` is the correct UA.
|
|||
|
|
- **The search endpoint (`/search/?q=...`) returns empty results** via ClaudeBot — the query parameter is not passed through. Use category browsing or direct product URLs instead.
|
|||
|
|
- **No CAPTCHA observed** during testing with ClaudeBot.
|
|||
|
|
- **No rate limiting observed**: 10 parallel requests across 5 workers completed in ~2s with all 200 responses. Sequential batches of 5 pages at 0.15–0.95s per request also worked cleanly.
|
|||
|
|
- **The Markdown response has no JSON-LD, no `__NEXT_DATA__`** — these are HTML-only structures. The Markdown format is simpler to parse.
|
|||
|
|
- **Disallowed paths** (from robots.txt): `/search`, `/ppc/clicks/`, `/sem-b/`, `/sem-compare-b/`, `/workspace/`, `/auth/login`. These 403 even with ClaudeBot.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Gotchas
|
|||
|
|
|
|||
|
|
- **Old Capterra product IDs may be invalid.** The URL `https://www.capterra.com/p/56703/Slack/` (ID 56703) returns 404 even with ClaudeBot — this is a stale or merged product ID. Slack's current ID is 135003, found in the team-communication-software category listing. Always discover IDs by crawling category pages rather than hard-coding them.
|
|||
|
|
|
|||
|
|
- **Slug is case-sensitive.** `Slack` works; `slack` returns 404. The slug is always in the category listing data.
|
|||
|
|
|
|||
|
|
- **Response is Markdown, not HTML.** `http_get` returns pre-rendered Markdown with no HTML tags, no JSON-LD, and no `__NEXT_DATA__`. Do not attempt `BeautifulSoup` parsing. Use `re` on the text directly.
|
|||
|
|
|
|||
|
|
- **`http_get` default UA is `Mozilla/5.0`** — this returns 403 from Capterra. Always pass `headers={"User-Agent": "ClaudeBot"}` explicitly.
|
|||
|
|
|
|||
|
|
- **Reviews page vs product page**: The `/reviews/` page has a clean rating header (`4.7 (24059)`) on line 10. The product overview page (`/p/{id}/{Slug}/`) has the same number buried deeper in the page as `\n4.7\n\nBased on 24,059 reviews\n`. For rating extraction, the reviews page is simpler and more reliable.
|
|||
|
|
|
|||
|
|
- **Category page 1 is larger than subsequent pages**: Page 1 includes editorial content (author bio, top-picks editorial) which can double the page size. Subsequent pages are ~20–30KB and contain only listings.
|
|||
|
|
|
|||
|
|
- **Reviewer name is present in the text but not cleanly delimited**: The Markdown format for reviewer attribution uses plain text lines above the review body. It's easier to skip reviewer name extraction than to parse the ambiguous formatting.
|
|||
|
|
|
|||
|
|
- **Sub-rating labels in reviews page**: "Ease of use" (lowercase 'u') and "Customer Service" (capitalized 'S') — match exactly. The product overview page may show additional sub-ratings like "Features" and "Value for money".
|
|||
|
|
|
|||
|
|
- **`rating_breakdown` pattern caveat**: The pattern `[1-5]\(\d+\)` on the product page can also match feature ratings. To isolate the 5-star breakdown, find it within the "Filter by rating" section, which appears as a block like `5(17268)\n\n4(5708)\n\n3(907)\n\n2(128)\n\n1(48)`.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## When to use the browser instead
|
|||
|
|
|
|||
|
|
The browser is not needed for any common Capterra task — the ClaudeBot flow handles all of them. Use the browser only if:
|
|||
|
|
|
|||
|
|
- You need to interact with a page element (e.g. submit a review, use the "fit-finder" wizard).
|
|||
|
|
- You need to access a Capterra page that is explicitly blocked in robots.txt (e.g. `/workspace/`, `/auth/login/`).
|
|||
|
|
- You need to simulate a logged-in user session with Capterra credentials.
|
|||
|
|
|
|||
|
|
For read-only scraping of product data, reviews, and category listings, `http_get` with `ClaudeBot` UA is both faster and more reliable than a browser.
|