feat(cluster): 集群化落地 —— Manager/Worker 拆分 + 归属租约 + 跨机验证(T08)

背景:把平台从「单机单进程」改造成「1 组 Manager + N 台 Worker + 共享归属状态」,
硬约束 = 全程兼容单例模式(deployMode 默认 local;生产切换前 47 一行未动)。

主要改动
1) 数据模型 v7(SQLite 与 PG 两方言同步):新增 dsh_hosts 注册表 +
   dsh_instances.{host_id,epoch,heartbeat_at,lease_until};claimInstance 原子抢占
   (UPDATE … WHERE host_id IS NULL OR lease_until < now)+ pinInstanceHost 钉住归属。
2) 租约与 fencing:src/supervisor/lease.ts(acquire/renew/release + stillHolder 判据 +
   ttl > 2×renew 硬校验);心跳里续租,失权即向 worker 下发更高 epoch(self-fencing)。
   ⚠️ release 只清租约(lease_until),**保留 host_id** —— host_id 是「用户数据在哪台」的锚点。
3) Worker agent(src/worker/agent.ts,子命令 dshs worker):实例生命周期 + 文件面 /fs/*
   + 幂等键(operationId)+ 鉴权(timingSafeEqual);Worker 不写控制面数据
   (apiKey/uid 由 Manager 随 launch 投递,R5 收窄)。
4) 远端 Spawner + LeasedSpawner:按 host 路由(**粘性优先**:有历史归属且那台 up 就留在原地,
   否则按容量选最空的)+ 容量准入 + deployMode=cluster 装配(systemd drop-in,可回滚)。
5) bwrap 修正:**所有挂载点的中间目录统一前置 + 去重 + 由外到内**(「就近创建」会在嵌套前缀下
   遮掉已绑挂载点 ⇒ bwrap: Can't chdir);且**只能用 --tmpfs**,用 --perms 会让 47 的
   bwrap 0.4.0 直接拒启动(沙箱全挂)。
6) 跨机隧道 src/worker/tunnel.ts:SSH ControlMaster + 动态 -R 转发;**自愈由 agent 本地
   20s 定时器驱动**(不能只放 /healthz —— 心跳本身经隧道进来,断了就没人触发它)。
7) 文件面按归属路由(RemoteUserFs):实例与文件必须落在同一台机器,否则实例看不到自己的文件。
8) 观测面:dshs doctor / dshs cluster status。

验证(本次均已实跑)
- test/lease.test.mjs:SQLite 10/10 == PG 10/10
- 组件级端到端 5 个:verify-cluster-{agent,lease,fs,migrate,live}.mjs
- 真跨机(47 Manager / 106 Worker,跨云 + 反向隧道)verify-cluster-cross.mjs 九步全绿
- 域名形态访问 verify-cluster-domain.mjs(<user>.域名 → Manager → 远端实例;越权 403)
- 冒烟 scripts/smoke-*:6/8,失败项与改动前基线完全相同(无回归)
- 生产切换与回滚剧本见 dsh-server-docs/交接单/T08-集群化落地-兼容单例模式.md §16
This commit is contained in:
admin committed 2026-09-15 18:47:02 +08:00
1 parent 68c0a320ed
commit c70d5d860e
47 files changed
+5367 -14

No files matched your search

+73
View File
@@ -57,18 +57,32 @@ import {
setUserRole as setUserRoleSync,
setUserUid as setUserUidSync,
toggleCredentialKey as toggleCredentialKeySync,
// 集群化(v7;T08 S2)
claimInstance as claimInstanceSync,
findDshHost as findDshHostSync,
listDshHosts as listDshHostsSync,
listExpiredInstanceLeases as listExpiredInstanceLeasesSync,
listInstancesByHost as listInstancesByHostSync,
pinInstanceHost as pinInstanceHostSync,
releaseInstanceLease as releaseInstanceLeaseSync,
renewInstanceLease as renewInstanceLeaseSync,
setDshHostStatus as setDshHostStatusSync,
upsertDshHost as upsertDshHostSync,
upsertBusinessPlugin as upsertBusinessPluginSync,
upsertDomain as upsertDomainSync,
upsertInstance as upsertInstanceSync,
} from './repo.js'
import type {
BusinessPlugin,
ClaimResult,
CredentialKey,
CredentialKeyMeta,
CredentialLandingRow,
CreateSessionInput,
CreateUserInput,
Domain,
DshHost,
DshHostStatus,
DshInstance,
DshInstanceRole,
DshInstanceStatus,
@@ -76,6 +90,7 @@ import type {
SessionRow,
SessionUser,
UpsertBusinessPluginInput,
UpsertDshHostInput,
UpsertDshInstanceInput,
User,
UserRole,
@@ -321,6 +336,64 @@ export class SqliteAdapter implements DbAdapter {
deleteUserInstancesSync(this.db, userId)
}
// ── 集群化:worker 注册表 + 归属/租约(v7;T08 S2)────────────────────────
// local 模式不会走到这些方法(`LocalSpawner` 不写库),它们只是让
// **SQLite 侧与 PG 侧行为一致** —— 测试与单机试跑都需要。
async upsertDshHost(input: UpsertDshHostInput): Promise<DshHost> {
try {
return upsertDshHostSync(this.db, input)
} catch (e) {
mapSqliteError(e)
}
}
async findDshHost(id: string): Promise<DshHost | undefined> {
return findDshHostSync(this.db, id)
}
async listDshHosts(): Promise<DshHost[]> {
return listDshHostsSync(this.db)
}
async setDshHostStatus(
id: string,
status: DshHostStatus,
usedMb?: number,
heartbeatAt?: number,
): Promise<boolean> {
return setDshHostStatusSync(this.db, id, status, usedMb, heartbeatAt)
}
async claimInstance(
userId: string,
hostId: string,
ttlMs: number,
meta?: { folder?: string; patch?: string },
): Promise<ClaimResult> {
return claimInstanceSync(this.db, userId, hostId, ttlMs, meta)
}
async renewInstanceLease(userId: string, hostId: string, epoch: number, ttlMs: number): Promise<boolean> {
return renewInstanceLeaseSync(this.db, userId, hostId, epoch, ttlMs)
}
async releaseInstanceLease(userId: string, hostId: string, epoch: number): Promise<boolean> {
return releaseInstanceLeaseSync(this.db, userId, hostId, epoch)
}
async pinInstanceHost(userId: string, hostId: string): Promise<void> {
pinInstanceHostSync(this.db, userId, hostId)
}
async listExpiredInstanceLeases(now: number): Promise<DshInstance[]> {
return listExpiredInstanceLeasesSync(this.db, now)
}
async listInstancesByHost(hostId: string): Promise<DshInstance[]> {
return listInstancesByHostSync(this.db, hostId)
}
async close(): Promise<void> {
this.db.close()
}