feat(cluster): 集群化落地 —— Manager/Worker 拆分 + 归属租约 + 跨机验证(T08)

背景:把平台从「单机单进程」改造成「1 组 Manager + N 台 Worker + 共享归属状态」,
硬约束 = 全程兼容单例模式(deployMode 默认 local;生产切换前 47 一行未动)。

主要改动
1) 数据模型 v7(SQLite 与 PG 两方言同步):新增 dsh_hosts 注册表 +
   dsh_instances.{host_id,epoch,heartbeat_at,lease_until};claimInstance 原子抢占
   (UPDATE … WHERE host_id IS NULL OR lease_until < now)+ pinInstanceHost 钉住归属。
2) 租约与 fencing:src/supervisor/lease.ts(acquire/renew/release + stillHolder 判据 +
   ttl > 2×renew 硬校验);心跳里续租,失权即向 worker 下发更高 epoch(self-fencing)。
   ⚠️ release 只清租约(lease_until),**保留 host_id** —— host_id 是「用户数据在哪台」的锚点。
3) Worker agent(src/worker/agent.ts,子命令 dshs worker):实例生命周期 + 文件面 /fs/*
   + 幂等键(operationId)+ 鉴权(timingSafeEqual);Worker 不写控制面数据
   (apiKey/uid 由 Manager 随 launch 投递,R5 收窄)。
4) 远端 Spawner + LeasedSpawner:按 host 路由(**粘性优先**:有历史归属且那台 up 就留在原地,
   否则按容量选最空的)+ 容量准入 + deployMode=cluster 装配(systemd drop-in,可回滚)。
5) bwrap 修正:**所有挂载点的中间目录统一前置 + 去重 + 由外到内**(「就近创建」会在嵌套前缀下
   遮掉已绑挂载点 ⇒ bwrap: Can't chdir);且**只能用 --tmpfs**,用 --perms 会让 47 的
   bwrap 0.4.0 直接拒启动(沙箱全挂)。
6) 跨机隧道 src/worker/tunnel.ts:SSH ControlMaster + 动态 -R 转发;**自愈由 agent 本地
   20s 定时器驱动**(不能只放 /healthz —— 心跳本身经隧道进来,断了就没人触发它)。
7) 文件面按归属路由(RemoteUserFs):实例与文件必须落在同一台机器,否则实例看不到自己的文件。
8) 观测面:dshs doctor / dshs cluster status。

验证(本次均已实跑)
- test/lease.test.mjs:SQLite 10/10 == PG 10/10
- 组件级端到端 5 个:verify-cluster-{agent,lease,fs,migrate,live}.mjs
- 真跨机(47 Manager / 106 Worker,跨云 + 反向隧道)verify-cluster-cross.mjs 九步全绿
- 域名形态访问 verify-cluster-domain.mjs(<user>.域名 → Manager → 远端实例;越权 403)
- 冒烟 scripts/smoke-*:6/8,失败项与改动前基线完全相同(无回归)
- 生产切换与回滚剧本见 dsh-server-docs/交接单/T08-集群化落地-兼容单例模式.md §16
This commit is contained in:
admin committed 2026-09-15 18:47:02 +08:00
1 parent 68c0a320ed
commit c70d5d860e
47 files changed
+5367 -14

No files matched your search

+168 -1
View File
@@ -12,20 +12,25 @@ import { randomUUID } from 'node:crypto'
import type { Database } from './connection.js'
import { prepare } from './prepared.js'
import {
clusterInstanceId,
toBusinessPlugin,
toDomain,
toDshHost,
toDshInstance,
toPublicUser,
toSession,
toUser,
toWorkspace,
type BusinessPlugin,
type ClaimResult,
type CredentialKey,
type CredentialKeyMeta,
type CredentialLandingRow,
type CreateSessionInput,
type CreateUserInput,
type Domain,
type DshHost,
type DshHostStatus,
type DshInstance,
type DshInstanceRole,
type DshInstanceStatus,
@@ -33,6 +38,7 @@ import {
type SessionRow,
type SessionUser,
type UpsertBusinessPluginInput,
type UpsertDshHostInput,
type UpsertDshInstanceInput,
type User,
type UserRole,
@@ -43,7 +49,8 @@ const USER_COLS = 'id, username, pass_hash, role, home_dir, api_key_ref, created
const DOMAIN_COLS = 'id, user_id, domain, verified, nginx_config, updated_at'
const BUSINESS_PLUGIN_COLS = 'id, name, description, version, tgz_path, file_size, uploaded_by, created_at, updated_at'
const INSTANCE_COLS =
'id, user_id, workspace_id, role, pid, port, status, started_at, last_exit, exit_code, last_error, folder, patch'
'id, user_id, workspace_id, role, pid, port, status, started_at, last_exit, exit_code, last_error, folder, patch, '
+ 'host_id, epoch, heartbeat_at, lease_until' // v7 集群化归属/租约(T08 S2)—— 漏了它们会让 hostId 恒为 null
export function createUser(db: Database, input: CreateUserInput, baseUid: number): User {
const createdAt = Date.now()
@@ -582,3 +589,163 @@ export function deleteBusinessPlugin(db: Database, id: string): boolean {
const info = prepare(db, 'DELETE FROM business_plugins WHERE id = ?').run(id)
return info.changes > 0
}
// ── 集群化:worker 注册表 + 实例归属/租约(v7;T08 S2 / 设计 §3.1–§3.2)────────
//
// ⚠️ local 模式**不调用**这些函数(`LocalSpawner` 靠进程内 Map + 单机互斥),
// 所以它们的存在不会改变现有单机行为。
const HOST_COLS = 'id, endpoint, agent_token, capacity_mb, used_mb, status, last_heartbeat'
/** 注册/更新一台 worker。join 幂等:同 id 重复执行 = 更新(并把它标回 `up`)。 */
export function upsertDshHost(db: Database, input: UpsertDshHostInput): DshHost {
prepare(db, `
INSERT INTO dsh_hosts (id, endpoint, agent_token, capacity_mb, used_mb, status, last_heartbeat)
VALUES (?, ?, ?, ?, 0, ?, NULL)
ON CONFLICT(id) DO UPDATE SET
endpoint = excluded.endpoint,
agent_token = excluded.agent_token,
capacity_mb = excluded.capacity_mb,
status = excluded.status
`).run(input.id, input.endpoint, input.agentToken, input.capacityMb, input.status ?? 'up')
const row = prepare(db, `SELECT ${HOST_COLS} FROM dsh_hosts WHERE id = ?`).get(input.id)
return toDshHost(row as Record<string, unknown>)
}
export function findDshHost(db: Database, id: string): DshHost | undefined {
const row = prepare(db, `SELECT ${HOST_COLS} FROM dsh_hosts WHERE id = ?`).get(id)
return row ? toDshHost(row as Record<string, unknown>) : undefined
}
export function listDshHosts(db: Database): DshHost[] {
const rows = prepare(db, `SELECT ${HOST_COLS} FROM dsh_hosts ORDER BY id ASC`).all() as Array<
Record<string, unknown>
>
return rows.map((row) => toDshHost(row))
}
/** 心跳/状态上报(只更新显式给出的字段,避免 heartbeat 覆盖 status)。 */
export function setDshHostStatus(
db: Database,
id: string,
status: DshHostStatus,
usedMb?: number,
heartbeatAt?: number,
): boolean {
const info = prepare(db, `
UPDATE dsh_hosts
SET status = ?,
used_mb = COALESCE(?, used_mb),
last_heartbeat = COALESCE(?, last_heartbeat)
WHERE id = ?
`).run(status, usedMb ?? null, heartbeatAt ?? null, id)
return info.changes > 0
}
/**
* **原子抢占**某用户 main 实例的归属(承重墙)。仅当"无人持有 **或** 租约已过期"才成功,
* 成功时 `epoch` +1(fencing token)。失败时返回当前持有者与租约到期时刻。
*/
export function claimInstance(
db: Database,
userId: string,
hostId: string,
ttlMs: number,
meta?: { folder?: string; patch?: string },
): ClaimResult {
const now = Date.now()
const id = clusterInstanceId(userId)
// 新用户没有 dsh_instances 行 ⇒ 先保证行存在(否则 UPDATE 影响 0 行被误判为"有人在管")。
prepare(db, `
INSERT INTO dsh_instances (id, user_id, role, status)
VALUES (?, ?, 'main', 'starting')
ON CONFLICT(id) DO NOTHING
`).run(id, userId)
// ⚠️ **必须把 folder/patch 一起落库**(2026-09-15 实测踩到):集群模式下实例行是这里建的,
// 而 local 模式不写库 ⇒ 若这里不记,`folder` 永远是 NULL,**迁移时复现不了启动参数**
// (表现为 `bwrap: Can't chdir to :` 空路径 ⇒ 崩溃循环)。用 COALESCE 保证不覆盖已有值。
const info = prepare(db, `
UPDATE dsh_instances
SET host_id = ?, epoch = epoch + 1, heartbeat_at = ?, lease_until = ?,
folder = COALESCE(?, folder), patch = COALESCE(?, patch)
WHERE id = ? AND (host_id IS NULL OR lease_until < ?)
`).run(hostId, now, now + ttlMs, meta?.folder ?? null, meta?.patch ?? null, id, now)
const row = prepare(db, 'SELECT host_id, epoch, lease_until FROM dsh_instances WHERE id = ?').get(id) as
| { host_id: string | null; epoch: number; lease_until: number }
| undefined
if (row === undefined) return { ok: false, holder: null, leaseUntil: 0 }
return info.changes > 0
? { ok: true, epoch: row.epoch, leaseUntil: row.lease_until }
: { ok: false, holder: row.host_id, leaseUntil: row.lease_until }
}
/** 续租。**必须带 epoch**:不匹配说明已被他人抢占 ⇒ 返回 false(fencing 生效)。 */
export function renewInstanceLease(
db: Database,
userId: string,
hostId: string,
epoch: number,
ttlMs: number,
): boolean {
const now = Date.now()
const info = prepare(db, `
UPDATE dsh_instances SET heartbeat_at = ?, lease_until = ?
WHERE id = ? AND host_id = ? AND epoch = ?
`).run(now, now + ttlMs, clusterInstanceId(userId), hostId, epoch)
return info.changes > 0
}
/**
* 主动释放**租约**(停实例时)。带 epoch 校验,避免误清他人的归属。
*
* ⚠️ **只清 `lease_until`,保留 `host_id`**(2026-09-15 生产切换暴露):
* `host_id` 的语义是「**这个用户的数据在哪台机器**」—— 用户的工作区在**本地盘**上,
* 把归属一起清掉就等于**丢掉粘性锚点**,下次启动可能被调度到没有他数据的机器上(工作区看起来是空的)。
* 「谁现在在托管」是**租约**(`lease_until`)的语义,所以释放只该清租约。
*/
export function releaseInstanceLease(db: Database, userId: string, hostId: string, epoch: number): boolean {
const info = prepare(db, `
UPDATE dsh_instances SET lease_until = 0
WHERE id = ? AND host_id = ? AND epoch = ?
`).run(clusterInstanceId(userId), hostId, epoch)
return info.changes > 0
}
/**
* **钉住**某用户的归属(首次触达其工作区时用):只写 `host_id`,不动 epoch/租约。
*
* 为什么需要:新用户还没有归属,`selectHost` 会在**写文件那一步**与**launch 那一步**各自选一次,
* 两次可能选到不同机器 ⇒ 「文件写到 A、实例起在 B」⇒ 实例看不到自己的文件(2026-09-15 实测)。
* 首次触达就把归属钉住,后续(含 launch)都走粘性,两面必然一致。
*/
export function pinInstanceHost(db: Database, userId: string, hostId: string): void {
const now = Date.now()
const id = clusterInstanceId(userId)
prepare(db, `
INSERT INTO dsh_instances (id, user_id, role, status, host_id, epoch, heartbeat_at, lease_until)
VALUES (?, ?, 'main', 'stopped', ?, 0, 0, 0)
ON CONFLICT(id) DO NOTHING
`).run(id, userId, hostId)
prepare(db, `
UPDATE dsh_instances SET host_id = ?
WHERE id = ? AND (host_id IS NULL OR lease_until < ?)
`).run(hostId, id, now)
}
/** 租约过期但仍标着归属的 main 实例(供巡检/自愈;**不等于可以立即接管**,见 R9)。 */
export function listExpiredInstanceLeases(db: Database, now: number): DshInstance[] {
const rows = prepare(db, `
SELECT ${INSTANCE_COLS} FROM dsh_instances
WHERE role = 'main' AND host_id IS NOT NULL AND lease_until < ? AND status <> 'stopped'
ORDER BY lease_until ASC
`).all(now) as Array<Record<string, unknown>>
return rows.map((row) => toDshInstance(row))
}
/** 某 worker 上的全部实例 —— 对账**一次拿回整机**(替代逐用户查询)。 */
export function listInstancesByHost(db: Database, hostId: string): DshInstance[] {
const rows = prepare(db, `SELECT ${INSTANCE_COLS} FROM dsh_instances WHERE host_id = ? ORDER BY user_id ASC`).all(
hostId,
) as Array<Record<string, unknown>>
return rows.map((row) => toDshInstance(row))
}