feat(cluster): 集群化落地 —— Manager/Worker 拆分 + 归属租约 + 跨机验证(T08)
背景:把平台从「单机单进程」改造成「1 组 Manager + N 台 Worker + 共享归属状态」,
硬约束 = 全程兼容单例模式(deployMode 默认 local;生产切换前 47 一行未动)。
主要改动
1) 数据模型 v7(SQLite 与 PG 两方言同步):新增 dsh_hosts 注册表 +
dsh_instances.{host_id,epoch,heartbeat_at,lease_until};claimInstance 原子抢占
(UPDATE … WHERE host_id IS NULL OR lease_until < now)+ pinInstanceHost 钉住归属。
2) 租约与 fencing:src/supervisor/lease.ts(acquire/renew/release + stillHolder 判据 +
ttl > 2×renew 硬校验);心跳里续租,失权即向 worker 下发更高 epoch(self-fencing)。
⚠️ release 只清租约(lease_until),**保留 host_id** —— host_id 是「用户数据在哪台」的锚点。
3) Worker agent(src/worker/agent.ts,子命令 dshs worker):实例生命周期 + 文件面 /fs/*
+ 幂等键(operationId)+ 鉴权(timingSafeEqual);Worker 不写控制面数据
(apiKey/uid 由 Manager 随 launch 投递,R5 收窄)。
4) 远端 Spawner + LeasedSpawner:按 host 路由(**粘性优先**:有历史归属且那台 up 就留在原地,
否则按容量选最空的)+ 容量准入 + deployMode=cluster 装配(systemd drop-in,可回滚)。
5) bwrap 修正:**所有挂载点的中间目录统一前置 + 去重 + 由外到内**(「就近创建」会在嵌套前缀下
遮掉已绑挂载点 ⇒ bwrap: Can't chdir);且**只能用 --tmpfs**,用 --perms 会让 47 的
bwrap 0.4.0 直接拒启动(沙箱全挂)。
6) 跨机隧道 src/worker/tunnel.ts:SSH ControlMaster + 动态 -R 转发;**自愈由 agent 本地
20s 定时器驱动**(不能只放 /healthz —— 心跳本身经隧道进来,断了就没人触发它)。
7) 文件面按归属路由(RemoteUserFs):实例与文件必须落在同一台机器,否则实例看不到自己的文件。
8) 观测面:dshs doctor / dshs cluster status。
验证(本次均已实跑)
- test/lease.test.mjs:SQLite 10/10 == PG 10/10
- 组件级端到端 5 个:verify-cluster-{agent,lease,fs,migrate,live}.mjs
- 真跨机(47 Manager / 106 Worker,跨云 + 反向隧道)verify-cluster-cross.mjs 九步全绿
- 域名形态访问 verify-cluster-domain.mjs(<user>.域名 → Manager → 远端实例;越权 403)
- 冒烟 scripts/smoke-*:6/8,失败项与改动前基线完全相同(无回归)
- 生产切换与回滚剧本见 dsh-server-docs/交接单/T08-集群化落地-兼容单例模式.md §16
This commit is contained in:
1 parent
68c0a320ed
commit
c70d5d860e
47 files changed
+5367
-14
No files matched your search
@@ -57,18 +57,32 @@ import {
|
||||
setUserRole as setUserRoleSync,
|
||||
setUserUid as setUserUidSync,
|
||||
toggleCredentialKey as toggleCredentialKeySync,
|
||||
// 集群化(v7;T08 S2)
|
||||
claimInstance as claimInstanceSync,
|
||||
findDshHost as findDshHostSync,
|
||||
listDshHosts as listDshHostsSync,
|
||||
listExpiredInstanceLeases as listExpiredInstanceLeasesSync,
|
||||
listInstancesByHost as listInstancesByHostSync,
|
||||
pinInstanceHost as pinInstanceHostSync,
|
||||
releaseInstanceLease as releaseInstanceLeaseSync,
|
||||
renewInstanceLease as renewInstanceLeaseSync,
|
||||
setDshHostStatus as setDshHostStatusSync,
|
||||
upsertDshHost as upsertDshHostSync,
|
||||
upsertBusinessPlugin as upsertBusinessPluginSync,
|
||||
upsertDomain as upsertDomainSync,
|
||||
upsertInstance as upsertInstanceSync,
|
||||
} from './repo.js'
|
||||
import type {
|
||||
BusinessPlugin,
|
||||
ClaimResult,
|
||||
CredentialKey,
|
||||
CredentialKeyMeta,
|
||||
CredentialLandingRow,
|
||||
CreateSessionInput,
|
||||
CreateUserInput,
|
||||
Domain,
|
||||
DshHost,
|
||||
DshHostStatus,
|
||||
DshInstance,
|
||||
DshInstanceRole,
|
||||
DshInstanceStatus,
|
||||
@@ -76,6 +90,7 @@ import type {
|
||||
SessionRow,
|
||||
SessionUser,
|
||||
UpsertBusinessPluginInput,
|
||||
UpsertDshHostInput,
|
||||
UpsertDshInstanceInput,
|
||||
User,
|
||||
UserRole,
|
||||
@@ -321,6 +336,64 @@ export class SqliteAdapter implements DbAdapter {
|
||||
deleteUserInstancesSync(this.db, userId)
|
||||
}
|
||||
|
||||
// ── 集群化:worker 注册表 + 归属/租约(v7;T08 S2)────────────────────────
|
||||
// local 模式不会走到这些方法(`LocalSpawner` 不写库),它们只是让
|
||||
// **SQLite 侧与 PG 侧行为一致** —— 测试与单机试跑都需要。
|
||||
|
||||
async upsertDshHost(input: UpsertDshHostInput): Promise<DshHost> {
|
||||
try {
|
||||
return upsertDshHostSync(this.db, input)
|
||||
} catch (e) {
|
||||
mapSqliteError(e)
|
||||
}
|
||||
}
|
||||
|
||||
async findDshHost(id: string): Promise<DshHost | undefined> {
|
||||
return findDshHostSync(this.db, id)
|
||||
}
|
||||
|
||||
async listDshHosts(): Promise<DshHost[]> {
|
||||
return listDshHostsSync(this.db)
|
||||
}
|
||||
|
||||
async setDshHostStatus(
|
||||
id: string,
|
||||
status: DshHostStatus,
|
||||
usedMb?: number,
|
||||
heartbeatAt?: number,
|
||||
): Promise<boolean> {
|
||||
return setDshHostStatusSync(this.db, id, status, usedMb, heartbeatAt)
|
||||
}
|
||||
|
||||
async claimInstance(
|
||||
userId: string,
|
||||
hostId: string,
|
||||
ttlMs: number,
|
||||
meta?: { folder?: string; patch?: string },
|
||||
): Promise<ClaimResult> {
|
||||
return claimInstanceSync(this.db, userId, hostId, ttlMs, meta)
|
||||
}
|
||||
|
||||
async renewInstanceLease(userId: string, hostId: string, epoch: number, ttlMs: number): Promise<boolean> {
|
||||
return renewInstanceLeaseSync(this.db, userId, hostId, epoch, ttlMs)
|
||||
}
|
||||
|
||||
async releaseInstanceLease(userId: string, hostId: string, epoch: number): Promise<boolean> {
|
||||
return releaseInstanceLeaseSync(this.db, userId, hostId, epoch)
|
||||
}
|
||||
|
||||
async pinInstanceHost(userId: string, hostId: string): Promise<void> {
|
||||
pinInstanceHostSync(this.db, userId, hostId)
|
||||
}
|
||||
|
||||
async listExpiredInstanceLeases(now: number): Promise<DshInstance[]> {
|
||||
return listExpiredInstanceLeasesSync(this.db, now)
|
||||
}
|
||||
|
||||
async listInstancesByHost(hostId: string): Promise<DshInstance[]> {
|
||||
return listInstancesByHostSync(this.db, hostId)
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.db.close()
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user