admin
c70d5d860e
背景：把平台从「单机单进程」改造成「1 组 Manager + N 台 Worker + 共享归属状态」，
硬约束 = 全程兼容单例模式（deployMode 默认 local；生产切换前 47 一行未动）。
主要改动
1) 数据模型 v7（SQLite 与 PG 两方言同步）：新增 dsh_hosts 注册表 +
dsh_instances.{host_id,epoch,heartbeat_at,lease_until}；claimInstance 原子抢占
（UPDATE … WHERE host_id IS NULL OR lease_until < now）+ pinInstanceHost 钉住归属。
2) 租约与 fencing：src/supervisor/lease.ts（acquire/renew/release + stillHolder 判据 +
ttl > 2×renew 硬校验）；心跳里续租，失权即向 worker 下发更高 epoch（self-fencing）。
⚠️ release 只清租约（lease_until），**保留 host_id** —— host_id 是「用户数据在哪台」的锚点。
3) Worker agent（src/worker/agent.ts，子命令 dshs worker）：实例生命周期 + 文件面 /fs/*
+ 幂等键（operationId）+ 鉴权（timingSafeEqual）；Worker 不写控制面数据
（apiKey/uid 由 Manager 随 launch 投递，R5 收窄）。
4) 远端 Spawner + LeasedSpawner：按 host 路由（**粘性优先**：有历史归属且那台 up 就留在原地，
否则按容量选最空的）+ 容量准入 + deployMode=cluster 装配（systemd drop-in，可回滚）。
5) bwrap 修正：**所有挂载点的中间目录统一前置 + 去重 + 由外到内**（「就近创建」会在嵌套前缀下
遮掉已绑挂载点 ⇒ bwrap: Can't chdir）；且**只能用 --tmpfs**，用 --perms 会让 47 的
bwrap 0.4.0 直接拒启动（沙箱全挂）。
6) 跨机隧道 src/worker/tunnel.ts：SSH ControlMaster + 动态 -R 转发；**自愈由 agent 本地
20s 定时器驱动**（不能只放 /healthz —— 心跳本身经隧道进来，断了就没人触发它）。
7) 文件面按归属路由（RemoteUserFs）：实例与文件必须落在同一台机器，否则实例看不到自己的文件。
8) 观测面：dshs doctor / dshs cluster status。
验证（本次均已实跑）
- test/lease.test.mjs：SQLite 10/10 == PG 10/10
- 组件级端到端 5 个：verify-cluster-{agent,lease,fs,migrate,live}.mjs
- 真跨机（47 Manager / 106 Worker，跨云 + 反向隧道）verify-cluster-cross.mjs 九步全绿
- 域名形态访问 verify-cluster-domain.mjs（<user>.域名 → Manager → 远端实例；越权 403）
- 冒烟 scripts/smoke-*：6/8，失败项与改动前基线完全相同（无回归）
- 生产切换与回滚剧本见 dsh-server-docs/交接单/T08-集群化落地-兼容单例模式.md §16
357 lines
15 KiB
TypeScript
357 lines
15 KiB
TypeScript
/**
|
||
* Deployment-varying configuration. Every tunable is a validated field here
|
||
* (or read from env), never a hardcoded constant inside the app.
|
||
* @module dshs/config
|
||
*/
|
||
|
||
import { randomBytes } from 'node:crypto'
|
||
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
|
||
import { homedir, hostname } from 'node:os'
|
||
import { join } from 'node:path'
|
||
|
||
/** Isolation tier. `soft` = per-user home/workspace + sandbox (same OS user);
|
||
* `account` = per-user OS account via a setuid wrapper (Linux, needs root). */
|
||
export type IsolationMode = 'soft' | 'account'
|
||
|
||
/** Deployment mode. `local` = single-host child_process (setuid/iptables);
|
||
* `k8s` = multi-replica control plane spawning per-user DSH Pods via the K8s API. */
|
||
export type DeployMode = 'local' | 'k8s' | 'cluster'
|
||
|
||
/** Resolved, immutable runtime configuration. */
|
||
export interface ServerConfig {
|
||
/** Bind host for the orchestrator HTTP server. */
|
||
host: string
|
||
/** Bind port; `0` requests an ephemeral port. */
|
||
port: number
|
||
/** SQLite database path. */
|
||
dbPath: string
|
||
/** Postgres connection string; when set, the DB backend is Postgres (k8s/HA). */
|
||
dbUrl?: string
|
||
/** Root under which per-user homes (`users/<id>/home`) and workspaces live. */
|
||
dataRoot: string
|
||
/** Shared read-only skill directory for all users (injected as
|
||
* `DSH_BUNDLED_SKILL_DIR` into every spawned DSH); empty = feature off. */
|
||
bundledSkillDir: string
|
||
/** Argv used to launch a child DSH; first element is the executable. */
|
||
dshCommand: string[]
|
||
/** Pino log level. */
|
||
logLevel: string
|
||
/** Set the `Secure` flag on session cookies (enable behind HTTPS). */
|
||
secureCookies: boolean
|
||
/** Session lifetime in seconds. */
|
||
sessionTtlSeconds: number
|
||
/** Max upload request body in bytes (base64 JSON; ~0.75× the file size). */
|
||
maxUploadBytes: number
|
||
/** Delay before auto-restarting a crashed child DSH, in milliseconds. */
|
||
restartBackoffMs: number
|
||
/** Upper bound for the exponential crash-restart backoff (档案 20). */
|
||
restartBackoffMaxMs: number
|
||
/** Auto-restarts allowed inside `crashWindowMs` before the circuit opens. */
|
||
crashMaxRestarts: number
|
||
/** Rolling window used to count auto-restarts (档案 20). */
|
||
crashWindowMs: number
|
||
/** Uptime after which a main counts as recovered and the backoff resets. */
|
||
crashStableMs: number
|
||
/** 档案 78：熔断首次冷却时长（冷却期内拒绝隐式启动）。 */
|
||
crashBreakerCooldownMs: number
|
||
/** 档案 78：熔断冷却上限（多次熔断后指数加长到此为止）。 */
|
||
crashBreakerMaxCooldownMs: number
|
||
/** Isolation tier (see {@link IsolationMode}); local mode only. */
|
||
isolationMode: IsolationMode
|
||
/** Argv prefix that drops privileges; `{UID}`/`{GID}` are substituted. Local mode only. */
|
||
spawnAsUserCommand: string[]
|
||
/** Base uid for the deterministic per-user uid. */
|
||
baseUid: number
|
||
/** Parent domain for per-user subdomains (`<username>.<baseDomain>`); empty = disabled. */
|
||
baseDomain: string
|
||
/** Cookie `Domain` value (e.g. `.example.com`) so the session reaches subdomains; empty = host-only. */
|
||
cookieDomain: string
|
||
/** Whether to pass `--patch` to child DSHs (needs a dsh CLI that supports it). */
|
||
enablePatch: boolean
|
||
/** Enable the loopback OUTPUT owner-match port guard (Linux + root). Local mode only. */
|
||
portGuard: boolean
|
||
/** Cap on resident main instances per host (0 = no cap). Local mode idle reap. */
|
||
maxIdleInstances: number
|
||
/** A main instance with no proxied/entered activity for this long is stopped
|
||
* (0 = disabled). Local mode idle reap. */
|
||
instanceIdleTtlSeconds: number
|
||
/** Period between idle-reap scans (seconds). Local mode idle reap. */
|
||
idleReapIntervalSeconds: number
|
||
/** Secret used to encrypt per-user secrets at rest (from env or dataRoot/secret.key). */
|
||
encryptionSecret: string
|
||
/** Deployment mode (see {@link DeployMode}). */
|
||
deployMode: DeployMode
|
||
/** K8s namespace for per-user DSH resources (k8s mode only). */
|
||
k8sNamespace: string
|
||
/** Image for per-user DSH Pods (k8s mode only). */
|
||
dshImage: string
|
||
/** Image for the per-user file sidecar (k8s mode only; shares the control-plane image). */
|
||
controlPlaneImage: string
|
||
/** imagePullSecret every generated per-user Pod/Job uses (k8s mode only). */
|
||
imagePullSecret: string
|
||
/** 443 egress whitelist CIDRs for per-user DSH Pods (Phase 4 egress 收敛);
|
||
* empty = keep the current `0.0.0.0/0` (except private ranges). */
|
||
egressCidrs: string[]
|
||
/** ServiceAccount the orchestrator runs as (k8s mode only). */
|
||
k8sServiceAccount: string
|
||
/** This replica's identity for leader election (POD_NAME, else hostname). */
|
||
podName: string
|
||
// ── cluster 模式（T08 S3/S4；设计 §1.1）────────────────────────────────
|
||
/** 本机在 `dsh_hosts.id` 里的标识（`deployMode=cluster` 时必填语义）。 */
|
||
clusterHostId: string
|
||
/** 本机 worker agent 的**基址**（Manager 侧用它投递实例操作），如 `http://127.0.0.1:9000`。 */
|
||
clusterAgentUrl: string
|
||
/** 与 agent 约定的共享密钥（仅内网 + nft 白名单）。 */
|
||
clusterAgentToken: string
|
||
/** agent 返回给 Manager 做代理的实例地址（同机 1a = `127.0.0.1`）。 */
|
||
clusterInstanceHost: string
|
||
/**
|
||
* **worker 上**的 dataRoot（T08 S5）。
|
||
* 空 = 与本地 `dataRoot` 相同（1a 形态）。多机部署必须显式配置 —— 而且是**基线约定**：
|
||
* 所有 worker 的 dataRoot 必须是同一个绝对路径（同镜像即可满足，设计 §14.3）。
|
||
*/
|
||
clusterWorkerDataRoot: string
|
||
}
|
||
|
||
/** Untyped overrides collected from argv / env. */
|
||
export interface ConfigOverrides {
|
||
host?: string
|
||
port?: string | number
|
||
dbPath?: string
|
||
dbUrl?: string
|
||
dataRoot?: string
|
||
bundledSkillDir?: string
|
||
dshCommand?: string[]
|
||
logLevel?: string
|
||
secureCookies?: boolean
|
||
sessionTtlSeconds?: number | string
|
||
maxUploadBytes?: number | string
|
||
restartBackoffMs?: number | string
|
||
restartBackoffMaxMs?: number | string
|
||
crashMaxRestarts?: number | string
|
||
crashWindowMs?: number | string
|
||
crashStableMs?: number | string
|
||
crashBreakerCooldownMs?: number | string
|
||
crashBreakerMaxCooldownMs?: number | string
|
||
isolationMode?: IsolationMode | string
|
||
spawnAsUserCommand?: string[]
|
||
baseUid?: number | string
|
||
baseDomain?: string
|
||
cookieDomain?: string
|
||
enablePatch?: boolean
|
||
portGuard?: boolean
|
||
maxIdleInstances?: number | string
|
||
instanceIdleTtlSeconds?: number | string
|
||
idleReapIntervalSeconds?: number | string
|
||
encryptionSecret?: string
|
||
deployMode?: DeployMode | string
|
||
k8sNamespace?: string
|
||
dshImage?: string
|
||
controlPlaneImage?: string
|
||
imagePullSecret?: string
|
||
egressCidrs?: string[]
|
||
k8sServiceAccount?: string
|
||
podName?: string
|
||
clusterHostId?: string
|
||
clusterAgentUrl?: string
|
||
clusterAgentToken?: string
|
||
clusterInstanceHost?: string
|
||
clusterWorkerDataRoot?: string
|
||
}
|
||
|
||
const DEFAULT_HOST = '127.0.0.1'
|
||
const DEFAULT_PORT = 3080
|
||
const DEFAULT_DSH_COMMAND = ['dsh']
|
||
const DEFAULT_LOG_LEVEL = 'info'
|
||
const DEFAULT_SESSION_TTL_SECONDS = 60 * 60 * 24 * 7
|
||
const DEFAULT_MAX_UPLOAD_BYTES = 25 * 1024 * 1024
|
||
const DEFAULT_RESTART_BACKOFF_MS = 1000
|
||
/** 崩溃自愈退避上限（档案 20）。 */
|
||
const DEFAULT_RESTART_BACKOFF_MAX_MS = 30000
|
||
/** 熔断窗口内允许的自动重启次数（档案 20）。 */
|
||
const DEFAULT_CRASH_MAX_RESTARTS = 5
|
||
/** 熔断窗口长度（档案 20）。 */
|
||
const DEFAULT_CRASH_WINDOW_MS = 600000
|
||
/** 连续运行多久视为已恢复、重置退避步数（档案 20）。 */
|
||
const DEFAULT_CRASH_STABLE_MS = 60000
|
||
// 档案 78：熔断冷却 —— 首次 10 分钟，指数加长，封顶 6 小时
|
||
const DEFAULT_CRASH_BREAKER_COOLDOWN_MS = 600000
|
||
const DEFAULT_CRASH_BREAKER_MAX_COOLDOWN_MS = 21600000
|
||
const DEFAULT_ISOLATION_MODE: IsolationMode = 'soft'
|
||
const DEFAULT_SPAWN_AS_USER_COMMAND = [
|
||
'setpriv',
|
||
'--reuid',
|
||
'{UID}',
|
||
'--regid',
|
||
'{GID}',
|
||
'--inh-caps=-all',
|
||
'--clear-groups',
|
||
'--',
|
||
]
|
||
const DEFAULT_BASE_UID = 100000
|
||
const DEFAULT_BASE_DOMAIN = ''
|
||
const DEFAULT_COOKIE_DOMAIN = ''
|
||
const DEFAULT_ENABLE_PATCH = false
|
||
const DEFAULT_MAX_IDLE_INSTANCES = 4
|
||
const DEFAULT_INSTANCE_IDLE_TTL_SECONDS = 60 * 60 * 24 * 7
|
||
const DEFAULT_IDLE_REAP_INTERVAL_SECONDS = 60
|
||
const DEFAULT_DEPLOY_MODE: DeployMode = 'local'
|
||
const DEFAULT_K8S_NAMESPACE = 'dsh'
|
||
const DEFAULT_K8S_SERVICE_ACCOUNT = 'dsh-orchestrator'
|
||
const DEFAULT_IMAGE_PULL_SECRET = 'dsh-acr-pull'
|
||
|
||
/** Load the encryption secret from env, or persist a generated one at
|
||
* `<dataRoot>/secret.key` (0600) so it survives restarts without setup. */
|
||
function resolveEncryptionSecret(dataRoot: string): string {
|
||
const fromEnv = process.env.DSHS_SECRET
|
||
if (fromEnv !== undefined && fromEnv !== '') return fromEnv
|
||
const path = join(dataRoot, 'secret.key')
|
||
try {
|
||
const existing = readFileSync(path, 'utf8').trim()
|
||
if (existing !== '') return existing
|
||
} catch {
|
||
// fall through to generate
|
||
}
|
||
const secret = randomBytes(32).toString('hex')
|
||
mkdirSync(dataRoot, { recursive: true })
|
||
writeFileSync(path, secret, { mode: 0o600 })
|
||
return secret
|
||
}
|
||
|
||
function toBool(value: string | undefined, fallback: boolean): boolean {
|
||
if (value === undefined) return fallback
|
||
return value === 'true' || value === '1'
|
||
}
|
||
|
||
/** Split a comma-separated CIDR list into a trimmed, de-duplicated array. */
|
||
function parseCidrs(value: string | undefined): string[] {
|
||
if (value === undefined || value === '') return []
|
||
return [...new Set(value.split(',').map((c) => c.trim()).filter((c) => c !== ''))]
|
||
}
|
||
|
||
/** Parse an isolation-mode value, rejecting anything outside `soft`/`account`
|
||
* so a typo in the env var fails loudly at startup instead of silently
|
||
* falling back to `soft` isolation. */
|
||
function toIsolationMode(value: string | undefined): IsolationMode | undefined {
|
||
if (value === undefined) return undefined
|
||
const normalized = value.trim().toLowerCase()
|
||
if (normalized === 'soft' || normalized === 'account') return normalized
|
||
throw new Error(`invalid isolation mode "${value}" (expected "soft" or "account")`)
|
||
}
|
||
|
||
/** Parse a deploy-mode value, rejecting anything outside `local`/`k8s`. */
|
||
function toDeployMode(value: string | undefined): DeployMode | undefined {
|
||
if (value === undefined) return undefined
|
||
const normalized = value.trim().toLowerCase()
|
||
if (normalized === 'local' || normalized === 'k8s' || normalized === 'cluster') return normalized
|
||
throw new Error(`invalid deploy mode "${value}" (expected "local", "k8s" or "cluster")`)
|
||
}
|
||
|
||
/**
|
||
* Fold argv/env overrides over defaults. `dataRoot` defaults to
|
||
* `~/.dshs` (always writable for dev); production sets
|
||
* `DSHS_DATA_ROOT=/var/lib/dshs`.
|
||
*/
|
||
export function resolveConfig(overrides: ConfigOverrides = {}): ServerConfig {
|
||
const dataRoot =
|
||
overrides.dataRoot ?? process.env.DSHS_DATA_ROOT ?? join(homedir(), '.dshs')
|
||
const port = overrides.port ?? process.env.DSHS_PORT ?? DEFAULT_PORT
|
||
const dshBin = process.env.DSHS_DSH_BIN
|
||
const isolationMode =
|
||
toIsolationMode(overrides.isolationMode) ??
|
||
toIsolationMode(process.env.DSHS_ISOLATION_MODE) ??
|
||
DEFAULT_ISOLATION_MODE
|
||
const deployMode =
|
||
toDeployMode(overrides.deployMode) ??
|
||
toDeployMode(process.env.DSHS_DEPLOY_MODE) ??
|
||
DEFAULT_DEPLOY_MODE
|
||
return {
|
||
host: overrides.host ?? DEFAULT_HOST,
|
||
port: typeof port === 'number' ? port : Number(port),
|
||
dbPath: overrides.dbPath ?? join(dataRoot, 'dshs.db'),
|
||
dbUrl: overrides.dbUrl ?? process.env.DSHS_DB_URL,
|
||
dataRoot,
|
||
bundledSkillDir:
|
||
overrides.bundledSkillDir ??
|
||
process.env.DSHS_BUNDLED_SKILL_DIR ??
|
||
join(dataRoot, 'bundled-skills'),
|
||
dshCommand: overrides.dshCommand ?? (dshBin !== undefined ? [dshBin] : DEFAULT_DSH_COMMAND),
|
||
logLevel: overrides.logLevel ?? DEFAULT_LOG_LEVEL,
|
||
secureCookies:
|
||
overrides.secureCookies ?? toBool(process.env.DSHS_SECURE_COOKIES, false),
|
||
sessionTtlSeconds: Number(
|
||
overrides.sessionTtlSeconds ?? process.env.DSHS_SESSION_TTL ?? DEFAULT_SESSION_TTL_SECONDS,
|
||
),
|
||
maxUploadBytes: Number(
|
||
overrides.maxUploadBytes ?? process.env.DSHS_MAX_UPLOAD ?? DEFAULT_MAX_UPLOAD_BYTES,
|
||
),
|
||
restartBackoffMs: Number(
|
||
overrides.restartBackoffMs ?? process.env.DSHS_RESTART_BACKOFF ?? DEFAULT_RESTART_BACKOFF_MS,
|
||
),
|
||
restartBackoffMaxMs: Number(
|
||
overrides.restartBackoffMaxMs ??
|
||
process.env.DSHS_RESTART_BACKOFF_MAX ??
|
||
DEFAULT_RESTART_BACKOFF_MAX_MS,
|
||
),
|
||
crashMaxRestarts: Number(
|
||
overrides.crashMaxRestarts ??
|
||
process.env.DSHS_CRASH_MAX_RESTARTS ??
|
||
DEFAULT_CRASH_MAX_RESTARTS,
|
||
),
|
||
crashWindowMs: Number(
|
||
overrides.crashWindowMs ?? process.env.DSHS_CRASH_WINDOW ?? DEFAULT_CRASH_WINDOW_MS,
|
||
),
|
||
crashStableMs: Number(
|
||
overrides.crashStableMs ?? process.env.DSHS_CRASH_STABLE ?? DEFAULT_CRASH_STABLE_MS,
|
||
),
|
||
crashBreakerCooldownMs: Number(
|
||
overrides.crashBreakerCooldownMs ??
|
||
process.env.DSHS_CRASH_BREAKER_COOLDOWN ??
|
||
DEFAULT_CRASH_BREAKER_COOLDOWN_MS,
|
||
),
|
||
crashBreakerMaxCooldownMs: Number(
|
||
overrides.crashBreakerMaxCooldownMs ??
|
||
process.env.DSHS_CRASH_BREAKER_MAX_COOLDOWN ??
|
||
DEFAULT_CRASH_BREAKER_MAX_COOLDOWN_MS,
|
||
),
|
||
isolationMode,
|
||
spawnAsUserCommand: overrides.spawnAsUserCommand ?? DEFAULT_SPAWN_AS_USER_COMMAND,
|
||
baseUid: Number(overrides.baseUid ?? process.env.DSHS_BASE_UID ?? DEFAULT_BASE_UID),
|
||
baseDomain: overrides.baseDomain ?? process.env.DSHS_BASE_DOMAIN ?? DEFAULT_BASE_DOMAIN,
|
||
cookieDomain: overrides.cookieDomain ?? process.env.DSHS_COOKIE_DOMAIN ?? DEFAULT_COOKIE_DOMAIN,
|
||
enablePatch: overrides.enablePatch ?? toBool(process.env.DSHS_ENABLE_PATCH, DEFAULT_ENABLE_PATCH),
|
||
portGuard: overrides.portGuard ?? toBool(process.env.DSHS_PORT_GUARD, false),
|
||
maxIdleInstances: Number(
|
||
overrides.maxIdleInstances ?? process.env.DSHS_MAX_IDLE_INSTANCES ?? DEFAULT_MAX_IDLE_INSTANCES,
|
||
),
|
||
instanceIdleTtlSeconds: Number(
|
||
overrides.instanceIdleTtlSeconds ??
|
||
process.env.DSHS_INSTANCE_IDLE_TTL ??
|
||
DEFAULT_INSTANCE_IDLE_TTL_SECONDS,
|
||
),
|
||
idleReapIntervalSeconds: Number(
|
||
overrides.idleReapIntervalSeconds ??
|
||
process.env.DSHS_IDLE_REAP_INTERVAL ??
|
||
DEFAULT_IDLE_REAP_INTERVAL_SECONDS,
|
||
),
|
||
encryptionSecret:
|
||
overrides.encryptionSecret ?? resolveEncryptionSecret(dataRoot),
|
||
deployMode,
|
||
k8sNamespace: overrides.k8sNamespace ?? process.env.DSHS_NAMESPACE ?? DEFAULT_K8S_NAMESPACE,
|
||
dshImage: overrides.dshImage ?? process.env.DSHS_DSH_IMAGE ?? '',
|
||
controlPlaneImage:
|
||
overrides.controlPlaneImage ?? process.env.DSHS_CONTROL_PLANE_IMAGE ?? '',
|
||
imagePullSecret:
|
||
overrides.imagePullSecret ?? process.env.DSHS_IMAGE_PULL_SECRET ?? DEFAULT_IMAGE_PULL_SECRET,
|
||
egressCidrs: overrides.egressCidrs ?? parseCidrs(process.env.DSHS_EGRESS_CIDRS),
|
||
k8sServiceAccount:
|
||
overrides.k8sServiceAccount ?? process.env.DSHS_K8S_SERVICE_ACCOUNT ?? DEFAULT_K8S_SERVICE_ACCOUNT,
|
||
podName: overrides.podName ?? process.env.POD_NAME ?? hostname(),
|
||
clusterHostId: overrides.clusterHostId ?? process.env.DSHS_CLUSTER_HOST_ID ?? hostname(),
|
||
clusterAgentUrl: overrides.clusterAgentUrl ?? process.env.DSHS_CLUSTER_AGENT_URL ?? '',
|
||
clusterAgentToken: overrides.clusterAgentToken ?? process.env.DSHS_CLUSTER_AGENT_TOKEN ?? '',
|
||
clusterInstanceHost: overrides.clusterInstanceHost ?? process.env.DSHS_CLUSTER_INSTANCE_HOST ?? '127.0.0.1',
|
||
clusterWorkerDataRoot: overrides.clusterWorkerDataRoot ?? process.env.DSHS_CLUSTER_WORKER_DATA_ROOT ?? '',
|
||
}
|
||
}
|