mirror of
https://github.com/stack-auth/stack.git
synced 2026-06-13 21:01:21 +08:00
## Summary
`stack emulator start` now resumes a fully-warm VM snapshot instead of
cold-booting, bringing startup from 30–120s down to ~5–8s with
per-install secret rotation, or ~2.5s with rotation opt-out. The
snapshot is captured **locally on first `stack emulator pull`**, not
shipped from CI — QEMU migration state isn't portable across
accelerators (KVM/HVF/TCG) or `-cpu max` feature sets, so a CI-captured
snapshot couldn't resume reliably on arbitrary user hardware.
Also bundles a pile of CLI QoL fixes (progress bars, PR/run artifact
pulls, PR-build download, native-TS ISO writer replacing
`hdiutil`/`mkisofs`/`genisoimage` host dep, unit tests).
| Scenario | Before | After |
|---|---|---|
| Cold boot (no snapshot) | 30–120s | same, works as fallback |
| `stack emulator pull` (one-time, includes local snapshot capture) |
~30s download | ~30s download + ~1–3 min cold-boot capture |
| Snapshot resume, normal start | — | **~5–8s** |
| Snapshot resume, `EMULATOR_NO_ROTATION=1` | — | **~2.5s** |
Backend (`/health?db=1`) and dashboard (`/handler/sign-in`) return 200
on all paths. Two successive snapshot resumes produce different rotated
PCK/SSK/SAK/CRON_SECRET values per install.
## How it works
**Build (CI)** — `docker/local-emulator/qemu/build-image.sh`:
1. Cloud-init provisioning runs to completion (migrations, seed,
slim-image) producing `stack-emulator-<arch>.qcow2`.
2. Image is built with a topology compatible with later snapshot capture
(pinned SMP=4, phantom seed/bundle ISOs, STACKCFG runtime ISO mounted at
build time, qemu-guest-agent running, placeholder hex secrets baked in
under `STACK_EMULATOR_BUILD_SNAPSHOT=1`).
3. CI publishes **only the qcow2** — no `.savevm.zst` ships.
**Pull (user's machine)** —
`packages/stack-cli/src/commands/emulator.ts` + `run-emulator.sh
capture`:
1. `stack emulator pull` downloads the qcow2 with a progress bar (or
from a PR / workflow run via `--pr` / `--run`).
2. CLI invokes `run-emulator.sh capture`: cold-boots the qcow2 with a
matching device layout (phantom ISOs, fsdev, pcie-root-port, virtfs
detached — migration-incompatible), waits for backend+dashboard health,
then drives QMP: `stop` → set `mapped-ram` + `multifd` caps → `migrate
file:state.raw` → poll `query-migrate` → `quit`. Raw mapped-ram file is
zstd-compressed to `stack-emulator-<arch>.savevm.zst` in the images dir.
3. `--skip-snapshot` opts out (first `start` will then cold-boot).
**Runtime** — `run-emulator.sh start`:
1. Launch QEMU with `-incoming defer` when a `.savevm.zst` is present;
decompress on first use, keep the `.raw` cached for subsequent starts.
2. QMP: same `mapped-ram` + `multifd` caps → `migrate-incoming
file:<.raw>` → poll for `paused` → `cont`.
3. Generate fresh per-install secrets on the host; pipe them
base64-encoded through QGA `guest-exec input-data` →
`trigger-fast-rotate` in the guest → `docker exec -e … rotate-secrets`.
4. `rotate-secrets` in the container: validate keys (hex-only), targeted
`sed` on the placeholder PCK across built JS, `UPDATE ApiKeySet`,
`supervisorctl restart stack-app cron-jobs` (with
`stopasgroup`/`killasgroup` so the Node children actually die and
release their ports).
5. Poll backend+dashboard health; if anything fails, clean up and fall
back to cold boot transparently.
**Security model**: placeholder hex values are baked into the snapshot
(`00…ff` PCK, `00…ee` SSK, `00…dd` SAK, `00…cc` CRON_SECRET). They are
non-secret by construction. Real per-install secrets are generated at
each `emulator start` and never leave the host.
## CLI changes (`packages/stack-cli`)
- **`src/lib/iso.ts`** (new): native TypeScript ISO 9660 + Joliet
writer, replacing the host-side `hdiutil`/`mkisofs`/`genisoimage`
dependency for generating the STACKCFG runtime config disk. Unit tests
in `src/lib/iso.test.ts`.
- **`src/commands/emulator.ts`**:
- `pull`: streamed downloads with progress bar + ETA; `--pr <number>`
and `--run <id>` to pull from a PR build's CI artifacts (uses
`extract-zip` for the nested zip); `--skip-snapshot` to opt out of the
one-time local capture.
- `start` (existing, extended): auto-pulls AND auto-captures when no
image exists, so first-ever `start` is self-bootstrapping; emits
`STACK_EMULATOR_CLI_WROTE_ISO=1` so the shell helper skips its own ISO
regen (avoids the genisoimage host dep).
- `capture` (new, invoked by `pull` and the auto-pull path of `start`):
drives the local snapshot capture via `run-emulator.sh`.
- `status`, `stop`, `reset`, `list-releases`: preflight +
path-resolution tightening (`STACK_EMULATOR_HOME` → images/run dirs).
- Unit tests in `src/commands/emulator.test.ts`.
- **`EMULATOR_NO_ROTATION=1`** env var skips the post-resume rotation
(intended for tests/CI where the placeholder secrets are fine — comes
with a loud warning).
## CI (`.github/workflows/qemu-emulator-build.yaml`)
- Builds **QEMU 10.2.2 from source** (cached), because
`mapped-ram`/`multifd` migration capabilities aren't available in the
distro's QEMU. Enables KVM on ubicloud runners so amd64 boots at
hardware speed.
- amd64 + arm64 both build on the same amd64 matrix
(`ubicloud-standard-8`); arm64 runs under cross-arch TCG (provisioning
only — boot/verify smoke test is amd64-only).
- Verification now runs through the CLI: `emulator start` → `emulator
status` → `emulator stop` against the freshly-built qcow2 (via
`STACK_EMULATOR_HOME` pointing at the workspace, so the CLI doesn't
silently auto-pull a prior release).
- Packages **only** the qcow2. No `.savevm.zst` upload / publish.
- Release notes updated.
## Key files
**Shell / guest:**
- `docker/local-emulator/qemu/build-image.sh` — snapshot-compatible
device topology + STACKCFG runtime ISO at build time
- `docker/local-emulator/qemu/run-emulator.sh` — `start`, `capture`,
`stop`, `reset`, `status`; `-incoming defer`, `.raw` cache, QGA-driven
rotation, cold-boot fallback
- `docker/local-emulator/qemu/common.sh` (new) — shared `qmp_session` +
`capture_vm_state` (factored out so build-image.sh and run-emulator.sh
share the capture path)
- `docker/local-emulator/qemu/cloud-init/emulator/user-data` —
placeholder secrets in snapshot mode, `wait-for-stack-ready`,
`trigger-fast-rotate`, qemu-guest-agent enabled
- `docker/local-emulator/rotate-secrets.sh` (new) — in-container
rotation (sed + UPDATE + supervisorctl)
- `docker/local-emulator/supervisord.conf` — `stopasgroup`/`killasgroup`
on `stack-app` and `cron-jobs`
- `docker/local-emulator/entrypoint.sh` — only mint CRON_SECRET if unset
(placeholder supplied in snapshot mode via --env-file)
- `docker/local-emulator/Dockerfile` — ships `rotate-secrets` to
`/usr/local/bin`
- `docker/server/entrypoint.sh` — source
`/run/stack-auth/rotated-secrets.env`; skip full-tree sentinel scan on
warm restarts via marker
**CLI:**
- `packages/stack-cli/src/lib/iso.ts` (new) + `iso.test.ts` (new)
- `packages/stack-cli/src/commands/emulator.ts` + `emulator.test.ts`
(new)
- `packages/stack-cli/vitest.config.ts` (new)
**CI:**
- `.github/workflows/qemu-emulator-build.yaml`
## Test plan
- [x] `docker/local-emulator/qemu/build-image.sh {amd64,arm64}` produces
`stack-emulator-<arch>.qcow2` with snapshot-compatible topology
- [x] `stack emulator pull` downloads qcow2 with progress, then captures
locally (~1–3 min) and writes `stack-emulator-<arch>.savevm.zst` in the
images dir
- [x] `stack emulator pull --skip-snapshot` stops after download
- [x] `stack emulator pull --pr <n>` / `--run <id>` pull from PR /
workflow run artifacts
- [x] `stack emulator start` on a fresh dir auto-pulls **and**
auto-captures, then starts; subsequent starts fast-resume in ~5–8s;
backend + dashboard return 200
- [x] `EMULATOR_NO_ROTATION=1 stack emulator start` completes in ~2.5s;
backend + dashboard return 200 with warning printed
- [x] Two consecutive `emulator start` invocations produce different PCK
values in the internal `ApiKeySet` row
- [x] `stack emulator status` / `stop` / `reset` resolve paths from
`STACK_EMULATOR_HOME`
- [x] Verified end-to-end on arm64 macOS under HVF (capture ~50s,
fast-resume ~6.5s)
- [x] `pnpm lint` and `pnpm typecheck` pass; stack-cli unit tests (iso +
emulator) pass
- [ ] CI green on this PR (qemu-emulator-build matrix, smoke test)
- [ ] `gh release download emulator-<branch>-latest` contains only
`stack-emulator-<arch>.qcow2` once this PR merges and publish runs
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Snapshot fast-start/resume with optional warm-snapshot assets, runtime
ISO generation, and a cached QEMU build to speed emulator setup.
* CLI: streamed artifact downloads with progress, improved release/asset
handling, stronger preflight checks, and start/status/stop emulator
commands.
* Automated secret rotation and ability to apply rotated secrets at
container startup; supervisor control socket enabled.
* **Bug Fixes**
* More robust start/stop/resume flows with automatic fallback to cold
boot and improved process-group shutdown behavior.
* **Tests**
* New tests for CLI utilities and ISO image generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
260 lines
11 KiB
TypeScript
260 lines
11 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { buildIso, type IsoFile } from "./iso.js";
|
|
|
|
const SECTOR = 2048;
|
|
|
|
// --- Test helpers: a minimal ISO 9660 parser, just enough to walk the
|
|
// directory records we produce so tests can assert the bytes we emitted really
|
|
// are addressable at the offsets claimed in the directory records.
|
|
|
|
function readSector(iso: Buffer, sector: number): Buffer {
|
|
return iso.subarray(sector * SECTOR, (sector + 1) * SECTOR);
|
|
}
|
|
|
|
function readVolumeDescriptor(iso: Buffer, sector: number): { type: number, id: string } {
|
|
const buf = readSector(iso, sector);
|
|
return { type: buf[0], id: buf.toString("ascii", 1, 6) };
|
|
}
|
|
|
|
type DirRecord = {
|
|
lenDr: number,
|
|
extentSector: number,
|
|
dataLength: number,
|
|
isDir: boolean,
|
|
fileId: Buffer,
|
|
};
|
|
|
|
function parseDirRecords(sector: Buffer): DirRecord[] {
|
|
const records: DirRecord[] = [];
|
|
let offset = 0;
|
|
while (offset < sector.length) {
|
|
const lenDr = sector[offset];
|
|
if (lenDr === 0) break;
|
|
const extentSector = sector.readUInt32LE(offset + 2);
|
|
const dataLength = sector.readUInt32LE(offset + 10);
|
|
const flags = sector[offset + 25];
|
|
const lenFi = sector[offset + 32];
|
|
const fileId = sector.subarray(offset + 33, offset + 33 + lenFi);
|
|
records.push({
|
|
lenDr,
|
|
extentSector,
|
|
dataLength,
|
|
isDir: (flags & 0x02) !== 0,
|
|
fileId: Buffer.from(fileId),
|
|
});
|
|
offset += lenDr;
|
|
}
|
|
return records;
|
|
}
|
|
|
|
// Follow PVD → root dir → pull file bytes by ISO-9660 name ("NAME.EXT;1").
|
|
function readIsoFile(iso: Buffer, isoName: string): Buffer | null {
|
|
const pvd = readSector(iso, 16);
|
|
const rootSector = pvd.readUInt32LE(156 + 2);
|
|
const rootRecords = parseDirRecords(readSector(iso, rootSector));
|
|
const match = rootRecords.find((r) => r.fileId.toString("ascii") === isoName);
|
|
if (!match) return null;
|
|
const start = match.extentSector * SECTOR;
|
|
return iso.subarray(start, start + match.dataLength);
|
|
}
|
|
|
|
// Same, but follow the Joliet SVD (so names are UCS-2 BE).
|
|
function readJolietFile(iso: Buffer, name: string): Buffer | null {
|
|
const svd = readSector(iso, 17);
|
|
if (svd[0] !== 2) return null;
|
|
const rootSector = svd.readUInt32LE(156 + 2);
|
|
const rootRecords = parseDirRecords(readSector(iso, rootSector));
|
|
const expected = Buffer.alloc(name.length * 2);
|
|
for (let i = 0; i < name.length; i++) expected.writeUInt16BE(name.charCodeAt(i), i * 2);
|
|
const match = rootRecords.find((r) => r.fileId.equals(expected));
|
|
if (!match) return null;
|
|
const start = match.extentSector * SECTOR;
|
|
return iso.subarray(start, start + match.dataLength);
|
|
}
|
|
|
|
function sampleFile(name: string, size: number, byte = 0x41): IsoFile {
|
|
return { name, data: Buffer.alloc(size, byte) };
|
|
}
|
|
|
|
describe("buildIso — structural invariants", () => {
|
|
it("emits the ISO 9660 standard identifiers at sectors 16, 17, 18", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("hi") }]);
|
|
expect(readVolumeDescriptor(iso, 16)).toEqual({ type: 1, id: "CD001" });
|
|
expect(readVolumeDescriptor(iso, 17)).toEqual({ type: 2, id: "CD001" });
|
|
expect(readVolumeDescriptor(iso, 18)).toEqual({ type: 0xff, id: "CD001" });
|
|
});
|
|
|
|
it("stores the volume identifier verbatim in the PVD for blkid discovery", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
|
|
const pvd = readSector(iso, 16);
|
|
expect(pvd.toString("ascii", 40, 40 + 8)).toBe("STACKCFG");
|
|
});
|
|
|
|
it("stores the volume identifier in the Joliet SVD as UCS-2 BE", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
|
|
const svd = readSector(iso, 17);
|
|
const ucs = svd.subarray(40, 40 + 16);
|
|
let decoded = "";
|
|
for (let i = 0; i < ucs.length; i += 2) decoded += String.fromCharCode(ucs.readUInt16BE(i));
|
|
expect(decoded).toBe("STACKCFG");
|
|
});
|
|
|
|
it("sets the Joliet escape sequence %/E", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
|
|
const svd = readSector(iso, 17);
|
|
expect(svd[88]).toBe(0x25);
|
|
expect(svd[89]).toBe(0x2f);
|
|
expect(svd[90]).toBe(0x45);
|
|
});
|
|
|
|
it("declares a volume space size equal to the emitted sector count", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("hello world") }]);
|
|
const pvd = readSector(iso, 16);
|
|
const declared = pvd.readUInt32LE(80);
|
|
expect(iso.length).toBe(declared * SECTOR);
|
|
});
|
|
});
|
|
|
|
describe("buildIso — file round-trip", () => {
|
|
it("makes files readable by ISO 9660 name", () => {
|
|
const iso = buildIso("STACKCFG", [
|
|
{ name: "runtime.env", data: Buffer.from("KEY=value\n") },
|
|
{ name: "base.env", data: Buffer.from("FOO=bar\n") },
|
|
]);
|
|
expect(readIsoFile(iso, "RUNTIME.ENV;1")?.toString()).toBe("KEY=value\n");
|
|
expect(readIsoFile(iso, "BASE.ENV;1")?.toString()).toBe("FOO=bar\n");
|
|
});
|
|
|
|
it("makes files readable by Joliet (lowercase) name", () => {
|
|
const iso = buildIso("STACKCFG", [
|
|
{ name: "runtime.env", data: Buffer.from("KEY=value\n") },
|
|
{ name: "base.env", data: Buffer.from("FOO=bar\n") },
|
|
]);
|
|
expect(readJolietFile(iso, "runtime.env")?.toString()).toBe("KEY=value\n");
|
|
expect(readJolietFile(iso, "base.env")?.toString()).toBe("FOO=bar\n");
|
|
});
|
|
|
|
it("preserves exact file contents byte-for-byte", () => {
|
|
const content = Buffer.from([0x00, 0xff, 0x7f, 0x80, 0x41, 0x42, 0x43]);
|
|
const iso = buildIso("STACKCFG", [{ name: "bin.dat", data: content }]);
|
|
expect(readJolietFile(iso, "bin.dat")?.equals(content)).toBe(true);
|
|
});
|
|
|
|
it("handles files whose length is exactly one sector", () => {
|
|
const content = Buffer.alloc(SECTOR, 0x37);
|
|
const iso = buildIso("STACKCFG", [{ name: "one.bin", data: content }]);
|
|
expect(readJolietFile(iso, "one.bin")?.equals(content)).toBe(true);
|
|
});
|
|
|
|
it("handles files that span multiple sectors", () => {
|
|
const content = Buffer.alloc(SECTOR * 3 + 17, 0x55);
|
|
const iso = buildIso("STACKCFG", [{ name: "big.bin", data: content }]);
|
|
expect(readJolietFile(iso, "big.bin")?.equals(content)).toBe(true);
|
|
});
|
|
|
|
it("keeps files byte-exact at the claimed extent sector across multi-file layouts", () => {
|
|
// Fingerprint each file so we can tell them apart even if extents shift.
|
|
const files: IsoFile[] = [
|
|
{ name: "alpha.bin", data: Buffer.alloc(SECTOR + 5, 0xaa) },
|
|
{ name: "beta.bin", data: Buffer.alloc(SECTOR * 2, 0xbb) },
|
|
{ name: "gamma.bin", data: Buffer.alloc(42, 0xcc) },
|
|
];
|
|
const iso = buildIso("STACKCFG", files);
|
|
for (const f of files) {
|
|
expect(readJolietFile(iso, f.name)?.equals(f.data)).toBe(true);
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("buildIso — edge cases", () => {
|
|
it("handles empty files without misaligning subsequent file extents", () => {
|
|
// Regression: `padToSector(Buffer.alloc(0))` used to return a 0-byte
|
|
// buffer, but the layout reserved 1 sector for the empty file — the next
|
|
// file was then read from the empty file's reserved slot.
|
|
const files: IsoFile[] = [
|
|
{ name: "empty.txt", data: Buffer.alloc(0) },
|
|
{ name: "after.txt", data: Buffer.from("marker\n") },
|
|
];
|
|
const iso = buildIso("STACKCFG", files);
|
|
expect(readJolietFile(iso, "empty.txt")?.length).toBe(0);
|
|
expect(readJolietFile(iso, "after.txt")?.toString()).toBe("marker\n");
|
|
// And: the declared volume space size must cover every emitted byte.
|
|
const pvd = readSector(iso, 16);
|
|
expect(iso.length).toBe(pvd.readUInt32LE(80) * SECTOR);
|
|
});
|
|
|
|
it("writes the exact file length in the directory record (not padded to sector)", () => {
|
|
const content = Buffer.from("abc");
|
|
const iso = buildIso("STACKCFG", [{ name: "tiny.txt", data: content }]);
|
|
const svd = readSector(iso, 17);
|
|
const rootSector = svd.readUInt32LE(156 + 2);
|
|
const records = parseDirRecords(readSector(iso, rootSector));
|
|
const file = records.find((r) => !r.isDir);
|
|
expect(file?.dataLength).toBe(3);
|
|
});
|
|
|
|
it("places the root directory records for . and .. pointing at the root extent", () => {
|
|
const iso = buildIso("STACKCFG", [{ name: "x.txt", data: Buffer.from("1") }]);
|
|
const svd = readSector(iso, 17);
|
|
const rootSector = svd.readUInt32LE(156 + 2);
|
|
const records = parseDirRecords(readSector(iso, rootSector));
|
|
expect(records.length).toBeGreaterThanOrEqual(2);
|
|
expect(records[0].fileId.equals(Buffer.from([0x00]))).toBe(true);
|
|
expect(records[1].fileId.equals(Buffer.from([0x01]))).toBe(true);
|
|
expect(records[0].isDir).toBe(true);
|
|
expect(records[0].extentSector).toBe(rootSector);
|
|
expect(records[1].extentSector).toBe(rootSector);
|
|
});
|
|
|
|
it("truncates volume identifiers longer than 32 bytes rather than corrupting the PVD", () => {
|
|
const longId = "A".repeat(64);
|
|
const iso = buildIso(longId, [{ name: "x.txt", data: Buffer.from("1") }]);
|
|
const pvd = readSector(iso, 16);
|
|
expect(pvd.toString("ascii", 40, 40 + 32)).toBe("A".repeat(32));
|
|
// Sector 17 should still be the Joliet SVD, not clobbered.
|
|
expect(pvd[881]).toBe(1);
|
|
expect(readVolumeDescriptor(iso, 17).type).toBe(2);
|
|
});
|
|
|
|
it("rejects an input set whose root directory record overflows one sector", () => {
|
|
// Each Joliet dir record for an N-char name is 33 + 2N + (2N even ? 1 : 0)
|
|
// ≈ 2N + 34 bytes. A sector is 2048. Thirty 30-char names → ~1860 bytes
|
|
// plus "." + ".." (68) → fits. Eighty of them → well over a sector.
|
|
const many: IsoFile[] = Array.from({ length: 80 }, (_, i) => ({
|
|
name: `file-${String(i).padStart(3, "0")}-padding-padding.bin`,
|
|
data: Buffer.from("x"),
|
|
}));
|
|
expect(() => buildIso("STACKCFG", many)).toThrow(/Root directory exceeds/);
|
|
});
|
|
|
|
it("produces a sector-aligned buffer regardless of file sizes", () => {
|
|
for (const size of [0, 1, SECTOR - 1, SECTOR, SECTOR + 1, SECTOR * 5 - 1]) {
|
|
const iso = buildIso("STACKCFG", [sampleFile("a.bin", size)]);
|
|
expect(iso.length % SECTOR).toBe(0);
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("buildIso — multiple file sector layout", () => {
|
|
it("assigns non-overlapping extents to all files", () => {
|
|
const files: IsoFile[] = [
|
|
sampleFile("a.bin", 10, 0x01),
|
|
sampleFile("b.bin", SECTOR, 0x02),
|
|
sampleFile("c.bin", SECTOR * 2 + 500, 0x03),
|
|
sampleFile("d.bin", 1, 0x04),
|
|
];
|
|
const iso = buildIso("STACKCFG", files);
|
|
const svd = readSector(iso, 17);
|
|
const rootSector = svd.readUInt32LE(156 + 2);
|
|
const records = parseDirRecords(readSector(iso, rootSector)).filter((r) => !r.isDir);
|
|
|
|
// Extents must be strictly ordered and non-overlapping.
|
|
const sorted = [...records].sort((a, b) => a.extentSector - b.extentSector);
|
|
for (let i = 1; i < sorted.length; i++) {
|
|
const prev = sorted[i - 1];
|
|
const prevEndSector = prev.extentSector + Math.max(1, Math.ceil(prev.dataLength / SECTOR));
|
|
expect(sorted[i].extentSector).toBeGreaterThanOrEqual(prevEndSector);
|
|
}
|
|
});
|
|
});
|