重开仓库内容：改推五个技能（browser-harness / humanizer / humanizer-zh / product-planning / session-mechanism）
按授权清空原有内容后重新提交（原 oil-ui-pro 一并移出，可从历史恢复）。 browser-harness 剔除 .venv 等运行环境；根 .gitignore 补记 .venv/ 与 node_modules/。
This commit is contained in:
admin committed
1 parent
9a152a1952
commit
777f7fe5d0
240 files changed
+52518
-4409
No files matched your search
@@ -0,0 +1,232 @@
|
||||
# Facebook Groups — mining feeds for posts + external URLs
|
||||
|
||||
Field-tested against a logged-in Jay account on 2026-04-18.
|
||||
**Requires:** Browser Harness driving a real Chrome that is (a) signed into
|
||||
Facebook and (b) already a member of the target group. Non-member or logged-out
|
||||
views serve a stripped landing page with no post content.
|
||||
|
||||
## What this skill is for
|
||||
|
||||
1. Pull the N most recent posts from a named FB group
|
||||
2. Harvest every external URL that members have shared
|
||||
3. Hand that URL list to `http_get` or another downstream extractor for structured scraping at scale
|
||||
4. Cache post text + author + timestamp for downstream keyword matching
|
||||
|
||||
It is NOT for: replying in groups, DMing members, or any write action.
|
||||
|
||||
## URL patterns
|
||||
|
||||
| What | URL |
|
||||
|------|-----|
|
||||
| Group main feed | `https://www.facebook.com/groups/{id_or_slug}` |
|
||||
| Group "Discussion" tab (canonical feed) | `https://www.facebook.com/groups/{id_or_slug}/?sorting_setting=CHRONOLOGICAL` |
|
||||
| Single post (permalink) | `https://www.facebook.com/groups/{id_or_slug}/posts/{post_id}/` |
|
||||
| User's joined-groups feed | `https://www.facebook.com/groups/feed/` |
|
||||
| List of YOUR groups | `https://www.facebook.com/groups/joins/` |
|
||||
|
||||
The `?sorting_setting=CHRONOLOGICAL` flag matters — without it, FB inserts an
|
||||
algorithmic ranking that hides older posts and shows the same handful of "popular"
|
||||
items every visit, which kills monitoring use cases.
|
||||
|
||||
## DOM anchors (verified 2026-04-18)
|
||||
|
||||
FB rewrites class names every few weeks but ARIA roles and stable URL patterns
|
||||
hold up well. Anchor on those, not on hashed CSS classes.
|
||||
|
||||
| Anchor | Selector | Notes |
|
||||
|--------|----------|-------|
|
||||
| Each post container | `div[role="article"]` | Stable. One per visible post. |
|
||||
| Post permalink | `a[href*="/groups/"][href*="/posts/"], a[href*="/groups/"][href*="/permalink/"]` | First match per article = the post link |
|
||||
| Post body text | `div[data-ad-preview="message"], div[data-ad-comet-preview="message"]` | One of these is the visible body |
|
||||
| Post author | `h3 a, h4 a` (first inside the article) | Falls back to `strong a` |
|
||||
| Post timestamp | `a[href*="/posts/"] abbr, a[role="link"] > span > span` (relative time text) | Hover gets the absolute time but the relative string is fine for sorting |
|
||||
| External link (FB redirector) | `a[href^="https://l.facebook.com/l.php?u="]` | Decode the `u=` param to get the real URL |
|
||||
| "See more" button on long posts | `div[role="button"]:has(span:contains("See more"))` (use XPath fallback if `:has` is unsupported) | Click before reading body or posts get truncated |
|
||||
|
||||
If selectors stop returning results, run the self-inspection block at the bottom
|
||||
of this file and update this table — that's the workflow, not a fallback.
|
||||
|
||||
|
||||
## Scrolling the feed (lazy load)
|
||||
|
||||
FB virtualizes the feed: scrolled-past posts get unmounted from the DOM. So
|
||||
"scroll then collect" misses old posts. Pattern that works: **collect-as-you-go.**
|
||||
|
||||
```python
|
||||
seen = {} # post_url -> dict
|
||||
TARGET = 50 # how many posts to collect
|
||||
MAX_SCROLLS = 30
|
||||
|
||||
for i in range(MAX_SCROLLS):
|
||||
new_posts = js("""
|
||||
Array.from(document.querySelectorAll('div[role="article"]')).map(el => {
|
||||
const link = el.querySelector('a[href*="/groups/"][href*="/posts/"], a[href*="/groups/"][href*="/permalink/"]');
|
||||
const body = el.querySelector('div[data-ad-preview="message"], div[data-ad-comet-preview="message"]');
|
||||
const author = el.querySelector('h3 a, h4 a, strong a');
|
||||
const time = el.querySelector('abbr, a[role="link"] > span > span');
|
||||
const externals = Array.from(el.querySelectorAll('a[href^="https://l.facebook.com/l.php?u="]'))
|
||||
.map(a => a.href);
|
||||
return {
|
||||
url: link?.href || null,
|
||||
author: author?.innerText || null,
|
||||
time: time?.innerText || null,
|
||||
body: body?.innerText?.slice(0, 4000) || null,
|
||||
externals: externals,
|
||||
};
|
||||
}).filter(p => p.url)
|
||||
""") or []
|
||||
for p in new_posts:
|
||||
seen.setdefault(p["url"], p)
|
||||
if len(seen) >= TARGET:
|
||||
break
|
||||
scroll(640, 400, dy=900) # scroll near middle of viewport
|
||||
wait(2.5) # FB needs ~2s to render new batch + a little buffer
|
||||
```
|
||||
|
||||
`wait(2.5)` is the floor. Faster than that and you'll see empty post containers
|
||||
because React hasn't hydrated them yet.
|
||||
|
||||
|
||||
## Decoding the external-URL redirector
|
||||
|
||||
Every external link gets wrapped in `https://l.facebook.com/l.php?u={URL-encoded real URL}&h=...`.
|
||||
You want the real URL, not the redirector.
|
||||
|
||||
```python
|
||||
from urllib.parse import urlparse, parse_qs, unquote
|
||||
def decode_fb_link(href):
|
||||
if not href.startswith("https://l.facebook.com/l.php"):
|
||||
return href
|
||||
q = parse_qs(urlparse(href).query)
|
||||
return unquote(q["u"][0]) if "u" in q else href
|
||||
```
|
||||
|
||||
## Handoff for the public outbound URLs
|
||||
|
||||
Once you have the harvested external list, those URLs are outside FB's walled
|
||||
garden — public, scrapable by ordinary HTTP clients or downstream extractors.
|
||||
Typed extraction is useful here because the sources are heterogeneous.
|
||||
|
||||
```python
|
||||
# After the scroll loop:
|
||||
external_urls = []
|
||||
for p in seen.values():
|
||||
for raw in p["externals"]:
|
||||
external_urls.append(decode_fb_link(raw))
|
||||
external_urls = sorted(set(external_urls))
|
||||
print(f"harvested {len(external_urls)} unique external URLs")
|
||||
|
||||
# Hand off to a downstream extractor in the calling conversation with whatever
|
||||
# schema matches the task, such as product/listing name, price, location, year,
|
||||
# and key features.
|
||||
```
|
||||
|
||||
For simple or static pages, `http_get(url)` from Harness itself is fine — it
|
||||
does a plain HTTP fetch without a browser and is the fastest option for bulk.
|
||||
|
||||
|
||||
## Rate-limit discipline
|
||||
|
||||
FB notices automation patterns at the account level, not the IP level. Driving
|
||||
a real logged-in session means Jay's account is the one getting rate-limited if
|
||||
you get greedy. Keep these floors:
|
||||
|
||||
- **≥2 seconds between scrolls** in the collect loop (the `wait(2.5)` above)
|
||||
- **≥3 seconds between groups** if you're sweeping multiple
|
||||
- **No more than ~6 groups per hour** for sustained monitoring
|
||||
- **Don't open the same group more than every 15 minutes** — repeated visits
|
||||
within a short window is a heuristic that triggers checkpoints
|
||||
|
||||
Symptoms of over-pacing: article containers start rendering with empty bodies,
|
||||
`/groups/{id}/` redirects to `/checkpoint/`, or the account briefly gets asked
|
||||
to re-verify a phone or confirm a login from a new device. If that happens,
|
||||
**stop immediately** and let Jay deal with the UI — don't try to auto-resolve.
|
||||
|
||||
## Self-inspection block (run this when selectors stop working)
|
||||
|
||||
Paste this into a Harness stdin block to see what anchors currently exist in the
|
||||
visible feed. Run it on a group you're a member of.
|
||||
|
||||
```python
|
||||
print(js("""
|
||||
({
|
||||
articles: document.querySelectorAll('div[role="article"]').length,
|
||||
body_preview_a: document.querySelectorAll('div[data-ad-preview="message"]').length,
|
||||
body_preview_b: document.querySelectorAll('div[data-ad-comet-preview="message"]').length,
|
||||
external_redirectors: document.querySelectorAll('a[href^="https://l.facebook.com/l.php?u="]').length,
|
||||
permalink_posts: document.querySelectorAll('a[href*="/groups/"][href*="/posts/"]').length,
|
||||
permalink_permalinks: document.querySelectorAll('a[href*="/groups/"][href*="/permalink/"]').length,
|
||||
})
|
||||
"""))
|
||||
# If any count is 0, the selector drifted. Open DevTools, right-click a visible
|
||||
# post, inspect, find the new stable attribute (aria-*, data-*), and update the
|
||||
# DOM anchors table above.
|
||||
```
|
||||
|
||||
|
||||
## Full example — mine one group, emit JSON for downstream tools
|
||||
|
||||
```bash
|
||||
cd ~/Developer/browser-harness && uv run browser-harness <<'PY'
|
||||
import json, sys
|
||||
from urllib.parse import urlparse, parse_qs, unquote
|
||||
|
||||
GROUP = "riceLakeBoating" # slug or numeric id
|
||||
TARGET = 50 # how many posts to collect
|
||||
MAX_SCROLLS = 30
|
||||
|
||||
goto_url(f"https://www.facebook.com/groups/{GROUP}/?sorting_setting=CHRONOLOGICAL")
|
||||
wait_for_load()
|
||||
wait(2)
|
||||
|
||||
# Abort if FB bounced us
|
||||
info = page_info()
|
||||
if "/checkpoint/" in info["url"] or "/login" in info["url"]:
|
||||
sys.exit("AUTH_WALL — stop and have Jay re-verify the account.")
|
||||
|
||||
seen = {}
|
||||
for _ in range(MAX_SCROLLS):
|
||||
batch = js("""
|
||||
Array.from(document.querySelectorAll('div[role="article"]')).map(el => {
|
||||
const link = el.querySelector('a[href*="/groups/"][href*="/posts/"], a[href*="/groups/"][href*="/permalink/"]');
|
||||
const body = el.querySelector('div[data-ad-preview="message"], div[data-ad-comet-preview="message"]');
|
||||
const author = el.querySelector('h3 a, h4 a, strong a');
|
||||
const time = el.querySelector('abbr, a[role="link"] > span > span');
|
||||
const externals = Array.from(el.querySelectorAll('a[href^="https://l.facebook.com/l.php?u="]')).map(a => a.href);
|
||||
return { url: link?.href, author: author?.innerText, time: time?.innerText,
|
||||
body: body?.innerText?.slice(0, 4000), externals };
|
||||
}).filter(p => p.url)
|
||||
""") or []
|
||||
for p in batch:
|
||||
seen.setdefault(p["url"], p)
|
||||
if len(seen) >= TARGET:
|
||||
break
|
||||
scroll(640, 400, dy=900)
|
||||
wait(2.5)
|
||||
|
||||
def decode(u):
|
||||
if not u.startswith("https://l.facebook.com/l.php"): return u
|
||||
q = parse_qs(urlparse(u).query)
|
||||
return unquote(q["u"][0]) if "u" in q else u
|
||||
|
||||
posts = list(seen.values())
|
||||
all_externals = sorted({decode(x) for p in posts for x in p["externals"]})
|
||||
capture_screenshot(f"/tmp/fb-group-{GROUP}.png", full=True)
|
||||
print(json.dumps({
|
||||
"group": GROUP,
|
||||
"post_count": len(posts),
|
||||
"posts": posts,
|
||||
"external_urls": all_externals,
|
||||
}, ensure_ascii=False))
|
||||
PY
|
||||
```
|
||||
|
||||
The JSON on stdout is the handoff payload — parse it in the calling agent and
|
||||
route `external_urls` into the downstream extractor that matches the task
|
||||
(competitor inventory, pricing intel, boat listings, etc).
|
||||
|
||||
## Gotchas log (append when you hit something new)
|
||||
|
||||
- **2026-04-18:** Fresh install verified. People-search URL requires login;
|
||||
page search `/search/pages/?q=` works the same way. Groups feed defaults to
|
||||
algorithmic sort — always append `?sorting_setting=CHRONOLOGICAL`.
|
||||
Reference in new issue
Block a user