stack/packages/stack-cli/src/lib/iso.test.ts
BilalG1 37ee5ec320
Fast-start local emulator via RAM snapshot + live secret rotation (#1340)
## Summary

`stack emulator start` now resumes a fully-warm VM snapshot instead of
cold-booting, bringing startup from 30–120s down to ~5–8s with
per-install secret rotation, or ~2.5s with rotation opt-out. The
snapshot is captured **locally on first `stack emulator pull`**, not
shipped from CI — QEMU migration state isn't portable across
accelerators (KVM/HVF/TCG) or `-cpu max` feature sets, so a CI-captured
snapshot couldn't resume reliably on arbitrary user hardware.

Also bundles a pile of CLI QoL fixes (progress bars, PR/run artifact
pulls, PR-build download, native-TS ISO writer replacing
`hdiutil`/`mkisofs`/`genisoimage` host dep, unit tests).

| Scenario | Before | After |
|---|---|---|
| Cold boot (no snapshot) | 30–120s | same, works as fallback |
| `stack emulator pull` (one-time, includes local snapshot capture) |
~30s download | ~30s download + ~1–3 min cold-boot capture |
| Snapshot resume, normal start | — | **~5–8s** |
| Snapshot resume, `EMULATOR_NO_ROTATION=1` | — | **~2.5s** |

Backend (`/health?db=1`) and dashboard (`/handler/sign-in`) return 200
on all paths. Two successive snapshot resumes produce different rotated
PCK/SSK/SAK/CRON_SECRET values per install.

## How it works

**Build (CI)** — `docker/local-emulator/qemu/build-image.sh`:

1. Cloud-init provisioning runs to completion (migrations, seed,
slim-image) producing `stack-emulator-<arch>.qcow2`.
2. Image is built with a topology compatible with later snapshot capture
(pinned SMP=4, phantom seed/bundle ISOs, STACKCFG runtime ISO mounted at
build time, qemu-guest-agent running, placeholder hex secrets baked in
under `STACK_EMULATOR_BUILD_SNAPSHOT=1`).
3. CI publishes **only the qcow2** — no `.savevm.zst` ships.

**Pull (user's machine)** —
`packages/stack-cli/src/commands/emulator.ts` + `run-emulator.sh
capture`:

1. `stack emulator pull` downloads the qcow2 with a progress bar (or
from a PR / workflow run via `--pr` / `--run`).
2. CLI invokes `run-emulator.sh capture`: cold-boots the qcow2 with a
matching device layout (phantom ISOs, fsdev, pcie-root-port, virtfs
detached — migration-incompatible), waits for backend+dashboard health,
then drives QMP: `stop` → set `mapped-ram` + `multifd` caps → `migrate
file:state.raw` → poll `query-migrate` → `quit`. Raw mapped-ram file is
zstd-compressed to `stack-emulator-<arch>.savevm.zst` in the images dir.
3. `--skip-snapshot` opts out (first `start` will then cold-boot).

**Runtime** — `run-emulator.sh start`:

1. Launch QEMU with `-incoming defer` when a `.savevm.zst` is present;
decompress on first use, keep the `.raw` cached for subsequent starts.
2. QMP: same `mapped-ram` + `multifd` caps → `migrate-incoming
file:<.raw>` → poll for `paused` → `cont`.
3. Generate fresh per-install secrets on the host; pipe them
base64-encoded through QGA `guest-exec input-data` →
`trigger-fast-rotate` in the guest → `docker exec -e … rotate-secrets`.
4. `rotate-secrets` in the container: validate keys (hex-only), targeted
`sed` on the placeholder PCK across built JS, `UPDATE ApiKeySet`,
`supervisorctl restart stack-app cron-jobs` (with
`stopasgroup`/`killasgroup` so the Node children actually die and
release their ports).
5. Poll backend+dashboard health; if anything fails, clean up and fall
back to cold boot transparently.

**Security model**: placeholder hex values are baked into the snapshot
(`00…ff` PCK, `00…ee` SSK, `00…dd` SAK, `00…cc` CRON_SECRET). They are
non-secret by construction. Real per-install secrets are generated at
each `emulator start` and never leave the host.

## CLI changes (`packages/stack-cli`)

- **`src/lib/iso.ts`** (new): native TypeScript ISO 9660 + Joliet
writer, replacing the host-side `hdiutil`/`mkisofs`/`genisoimage`
dependency for generating the STACKCFG runtime config disk. Unit tests
in `src/lib/iso.test.ts`.
- **`src/commands/emulator.ts`**:
- `pull`: streamed downloads with progress bar + ETA; `--pr <number>`
and `--run <id>` to pull from a PR build's CI artifacts (uses
`extract-zip` for the nested zip); `--skip-snapshot` to opt out of the
one-time local capture.
- `start` (existing, extended): auto-pulls AND auto-captures when no
image exists, so first-ever `start` is self-bootstrapping; emits
`STACK_EMULATOR_CLI_WROTE_ISO=1` so the shell helper skips its own ISO
regen (avoids the genisoimage host dep).
- `capture` (new, invoked by `pull` and the auto-pull path of `start`):
drives the local snapshot capture via `run-emulator.sh`.
- `status`, `stop`, `reset`, `list-releases`: preflight +
path-resolution tightening (`STACK_EMULATOR_HOME` → images/run dirs).
  - Unit tests in `src/commands/emulator.test.ts`.
- **`EMULATOR_NO_ROTATION=1`** env var skips the post-resume rotation
(intended for tests/CI where the placeholder secrets are fine — comes
with a loud warning).

## CI (`.github/workflows/qemu-emulator-build.yaml`)

- Builds **QEMU 10.2.2 from source** (cached), because
`mapped-ram`/`multifd` migration capabilities aren't available in the
distro's QEMU. Enables KVM on ubicloud runners so amd64 boots at
hardware speed.
- amd64 + arm64 both build on the same amd64 matrix
(`ubicloud-standard-8`); arm64 runs under cross-arch TCG (provisioning
only — boot/verify smoke test is amd64-only).
- Verification now runs through the CLI: `emulator start` → `emulator
status` → `emulator stop` against the freshly-built qcow2 (via
`STACK_EMULATOR_HOME` pointing at the workspace, so the CLI doesn't
silently auto-pull a prior release).
- Packages **only** the qcow2. No `.savevm.zst` upload / publish.
- Release notes updated.

## Key files

**Shell / guest:**
- `docker/local-emulator/qemu/build-image.sh` — snapshot-compatible
device topology + STACKCFG runtime ISO at build time
- `docker/local-emulator/qemu/run-emulator.sh` — `start`, `capture`,
`stop`, `reset`, `status`; `-incoming defer`, `.raw` cache, QGA-driven
rotation, cold-boot fallback
- `docker/local-emulator/qemu/common.sh` (new) — shared `qmp_session` +
`capture_vm_state` (factored out so build-image.sh and run-emulator.sh
share the capture path)
- `docker/local-emulator/qemu/cloud-init/emulator/user-data` —
placeholder secrets in snapshot mode, `wait-for-stack-ready`,
`trigger-fast-rotate`, qemu-guest-agent enabled
- `docker/local-emulator/rotate-secrets.sh` (new) — in-container
rotation (sed + UPDATE + supervisorctl)
- `docker/local-emulator/supervisord.conf` — `stopasgroup`/`killasgroup`
on `stack-app` and `cron-jobs`
- `docker/local-emulator/entrypoint.sh` — only mint CRON_SECRET if unset
(placeholder supplied in snapshot mode via --env-file)
- `docker/local-emulator/Dockerfile` — ships `rotate-secrets` to
`/usr/local/bin`
- `docker/server/entrypoint.sh` — source
`/run/stack-auth/rotated-secrets.env`; skip full-tree sentinel scan on
warm restarts via marker

**CLI:**
- `packages/stack-cli/src/lib/iso.ts` (new) + `iso.test.ts` (new)
- `packages/stack-cli/src/commands/emulator.ts` + `emulator.test.ts`
(new)
- `packages/stack-cli/vitest.config.ts` (new)

**CI:**
- `.github/workflows/qemu-emulator-build.yaml`

## Test plan

- [x] `docker/local-emulator/qemu/build-image.sh {amd64,arm64}` produces
`stack-emulator-<arch>.qcow2` with snapshot-compatible topology
- [x] `stack emulator pull` downloads qcow2 with progress, then captures
locally (~1–3 min) and writes `stack-emulator-<arch>.savevm.zst` in the
images dir
- [x] `stack emulator pull --skip-snapshot` stops after download
- [x] `stack emulator pull --pr <n>` / `--run <id>` pull from PR /
workflow run artifacts
- [x] `stack emulator start` on a fresh dir auto-pulls **and**
auto-captures, then starts; subsequent starts fast-resume in ~5–8s;
backend + dashboard return 200
- [x] `EMULATOR_NO_ROTATION=1 stack emulator start` completes in ~2.5s;
backend + dashboard return 200 with warning printed
- [x] Two consecutive `emulator start` invocations produce different PCK
values in the internal `ApiKeySet` row
- [x] `stack emulator status` / `stop` / `reset` resolve paths from
`STACK_EMULATOR_HOME`
- [x] Verified end-to-end on arm64 macOS under HVF (capture ~50s,
fast-resume ~6.5s)
- [x] `pnpm lint` and `pnpm typecheck` pass; stack-cli unit tests (iso +
emulator) pass
- [ ] CI green on this PR (qemu-emulator-build matrix, smoke test)
- [ ] `gh release download emulator-<branch>-latest` contains only
`stack-emulator-<arch>.qcow2` once this PR merges and publish runs

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Snapshot fast-start/resume with optional warm-snapshot assets, runtime
ISO generation, and a cached QEMU build to speed emulator setup.
* CLI: streamed artifact downloads with progress, improved release/asset
handling, stronger preflight checks, and start/status/stop emulator
commands.
* Automated secret rotation and ability to apply rotated secrets at
container startup; supervisor control socket enabled.

* **Bug Fixes**
* More robust start/stop/resume flows with automatic fallback to cold
boot and improved process-group shutdown behavior.

* **Tests**
  * New tests for CLI utilities and ISO image generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-20 14:24:49 -07:00

260 lines
11 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { buildIso, type IsoFile } from "./iso.js";
const SECTOR = 2048;
// --- Test helpers: a minimal ISO 9660 parser, just enough to walk the
// directory records we produce so tests can assert the bytes we emitted really
// are addressable at the offsets claimed in the directory records.
function readSector(iso: Buffer, sector: number): Buffer {
return iso.subarray(sector * SECTOR, (sector + 1) * SECTOR);
}
function readVolumeDescriptor(iso: Buffer, sector: number): { type: number, id: string } {
const buf = readSector(iso, sector);
return { type: buf[0], id: buf.toString("ascii", 1, 6) };
}
type DirRecord = {
lenDr: number,
extentSector: number,
dataLength: number,
isDir: boolean,
fileId: Buffer,
};
function parseDirRecords(sector: Buffer): DirRecord[] {
const records: DirRecord[] = [];
let offset = 0;
while (offset < sector.length) {
const lenDr = sector[offset];
if (lenDr === 0) break;
const extentSector = sector.readUInt32LE(offset + 2);
const dataLength = sector.readUInt32LE(offset + 10);
const flags = sector[offset + 25];
const lenFi = sector[offset + 32];
const fileId = sector.subarray(offset + 33, offset + 33 + lenFi);
records.push({
lenDr,
extentSector,
dataLength,
isDir: (flags & 0x02) !== 0,
fileId: Buffer.from(fileId),
});
offset += lenDr;
}
return records;
}
// Follow PVD → root dir → pull file bytes by ISO-9660 name ("NAME.EXT;1").
function readIsoFile(iso: Buffer, isoName: string): Buffer | null {
const pvd = readSector(iso, 16);
const rootSector = pvd.readUInt32LE(156 + 2);
const rootRecords = parseDirRecords(readSector(iso, rootSector));
const match = rootRecords.find((r) => r.fileId.toString("ascii") === isoName);
if (!match) return null;
const start = match.extentSector * SECTOR;
return iso.subarray(start, start + match.dataLength);
}
// Same, but follow the Joliet SVD (so names are UCS-2 BE).
function readJolietFile(iso: Buffer, name: string): Buffer | null {
const svd = readSector(iso, 17);
if (svd[0] !== 2) return null;
const rootSector = svd.readUInt32LE(156 + 2);
const rootRecords = parseDirRecords(readSector(iso, rootSector));
const expected = Buffer.alloc(name.length * 2);
for (let i = 0; i < name.length; i++) expected.writeUInt16BE(name.charCodeAt(i), i * 2);
const match = rootRecords.find((r) => r.fileId.equals(expected));
if (!match) return null;
const start = match.extentSector * SECTOR;
return iso.subarray(start, start + match.dataLength);
}
function sampleFile(name: string, size: number, byte = 0x41): IsoFile {
return { name, data: Buffer.alloc(size, byte) };
}
describe("buildIso — structural invariants", () => {
it("emits the ISO 9660 standard identifiers at sectors 16, 17, 18", () => {
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("hi") }]);
expect(readVolumeDescriptor(iso, 16)).toEqual({ type: 1, id: "CD001" });
expect(readVolumeDescriptor(iso, 17)).toEqual({ type: 2, id: "CD001" });
expect(readVolumeDescriptor(iso, 18)).toEqual({ type: 0xff, id: "CD001" });
});
it("stores the volume identifier verbatim in the PVD for blkid discovery", () => {
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
const pvd = readSector(iso, 16);
expect(pvd.toString("ascii", 40, 40 + 8)).toBe("STACKCFG");
});
it("stores the volume identifier in the Joliet SVD as UCS-2 BE", () => {
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
const svd = readSector(iso, 17);
const ucs = svd.subarray(40, 40 + 16);
let decoded = "";
for (let i = 0; i < ucs.length; i += 2) decoded += String.fromCharCode(ucs.readUInt16BE(i));
expect(decoded).toBe("STACKCFG");
});
it("sets the Joliet escape sequence %/E", () => {
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("x") }]);
const svd = readSector(iso, 17);
expect(svd[88]).toBe(0x25);
expect(svd[89]).toBe(0x2f);
expect(svd[90]).toBe(0x45);
});
it("declares a volume space size equal to the emitted sector count", () => {
const iso = buildIso("STACKCFG", [{ name: "a.txt", data: Buffer.from("hello world") }]);
const pvd = readSector(iso, 16);
const declared = pvd.readUInt32LE(80);
expect(iso.length).toBe(declared * SECTOR);
});
});
describe("buildIso — file round-trip", () => {
it("makes files readable by ISO 9660 name", () => {
const iso = buildIso("STACKCFG", [
{ name: "runtime.env", data: Buffer.from("KEY=value\n") },
{ name: "base.env", data: Buffer.from("FOO=bar\n") },
]);
expect(readIsoFile(iso, "RUNTIME.ENV;1")?.toString()).toBe("KEY=value\n");
expect(readIsoFile(iso, "BASE.ENV;1")?.toString()).toBe("FOO=bar\n");
});
it("makes files readable by Joliet (lowercase) name", () => {
const iso = buildIso("STACKCFG", [
{ name: "runtime.env", data: Buffer.from("KEY=value\n") },
{ name: "base.env", data: Buffer.from("FOO=bar\n") },
]);
expect(readJolietFile(iso, "runtime.env")?.toString()).toBe("KEY=value\n");
expect(readJolietFile(iso, "base.env")?.toString()).toBe("FOO=bar\n");
});
it("preserves exact file contents byte-for-byte", () => {
const content = Buffer.from([0x00, 0xff, 0x7f, 0x80, 0x41, 0x42, 0x43]);
const iso = buildIso("STACKCFG", [{ name: "bin.dat", data: content }]);
expect(readJolietFile(iso, "bin.dat")?.equals(content)).toBe(true);
});
it("handles files whose length is exactly one sector", () => {
const content = Buffer.alloc(SECTOR, 0x37);
const iso = buildIso("STACKCFG", [{ name: "one.bin", data: content }]);
expect(readJolietFile(iso, "one.bin")?.equals(content)).toBe(true);
});
it("handles files that span multiple sectors", () => {
const content = Buffer.alloc(SECTOR * 3 + 17, 0x55);
const iso = buildIso("STACKCFG", [{ name: "big.bin", data: content }]);
expect(readJolietFile(iso, "big.bin")?.equals(content)).toBe(true);
});
it("keeps files byte-exact at the claimed extent sector across multi-file layouts", () => {
// Fingerprint each file so we can tell them apart even if extents shift.
const files: IsoFile[] = [
{ name: "alpha.bin", data: Buffer.alloc(SECTOR + 5, 0xaa) },
{ name: "beta.bin", data: Buffer.alloc(SECTOR * 2, 0xbb) },
{ name: "gamma.bin", data: Buffer.alloc(42, 0xcc) },
];
const iso = buildIso("STACKCFG", files);
for (const f of files) {
expect(readJolietFile(iso, f.name)?.equals(f.data)).toBe(true);
}
});
});
describe("buildIso — edge cases", () => {
it("handles empty files without misaligning subsequent file extents", () => {
// Regression: `padToSector(Buffer.alloc(0))` used to return a 0-byte
// buffer, but the layout reserved 1 sector for the empty file — the next
// file was then read from the empty file's reserved slot.
const files: IsoFile[] = [
{ name: "empty.txt", data: Buffer.alloc(0) },
{ name: "after.txt", data: Buffer.from("marker\n") },
];
const iso = buildIso("STACKCFG", files);
expect(readJolietFile(iso, "empty.txt")?.length).toBe(0);
expect(readJolietFile(iso, "after.txt")?.toString()).toBe("marker\n");
// And: the declared volume space size must cover every emitted byte.
const pvd = readSector(iso, 16);
expect(iso.length).toBe(pvd.readUInt32LE(80) * SECTOR);
});
it("writes the exact file length in the directory record (not padded to sector)", () => {
const content = Buffer.from("abc");
const iso = buildIso("STACKCFG", [{ name: "tiny.txt", data: content }]);
const svd = readSector(iso, 17);
const rootSector = svd.readUInt32LE(156 + 2);
const records = parseDirRecords(readSector(iso, rootSector));
const file = records.find((r) => !r.isDir);
expect(file?.dataLength).toBe(3);
});
it("places the root directory records for . and .. pointing at the root extent", () => {
const iso = buildIso("STACKCFG", [{ name: "x.txt", data: Buffer.from("1") }]);
const svd = readSector(iso, 17);
const rootSector = svd.readUInt32LE(156 + 2);
const records = parseDirRecords(readSector(iso, rootSector));
expect(records.length).toBeGreaterThanOrEqual(2);
expect(records[0].fileId.equals(Buffer.from([0x00]))).toBe(true);
expect(records[1].fileId.equals(Buffer.from([0x01]))).toBe(true);
expect(records[0].isDir).toBe(true);
expect(records[0].extentSector).toBe(rootSector);
expect(records[1].extentSector).toBe(rootSector);
});
it("truncates volume identifiers longer than 32 bytes rather than corrupting the PVD", () => {
const longId = "A".repeat(64);
const iso = buildIso(longId, [{ name: "x.txt", data: Buffer.from("1") }]);
const pvd = readSector(iso, 16);
expect(pvd.toString("ascii", 40, 40 + 32)).toBe("A".repeat(32));
// Sector 17 should still be the Joliet SVD, not clobbered.
expect(pvd[881]).toBe(1);
expect(readVolumeDescriptor(iso, 17).type).toBe(2);
});
it("rejects an input set whose root directory record overflows one sector", () => {
// Each Joliet dir record for an N-char name is 33 + 2N + (2N even ? 1 : 0)
// ≈ 2N + 34 bytes. A sector is 2048. Thirty 30-char names → ~1860 bytes
// plus "." + ".." (68) → fits. Eighty of them → well over a sector.
const many: IsoFile[] = Array.from({ length: 80 }, (_, i) => ({
name: `file-${String(i).padStart(3, "0")}-padding-padding.bin`,
data: Buffer.from("x"),
}));
expect(() => buildIso("STACKCFG", many)).toThrow(/Root directory exceeds/);
});
it("produces a sector-aligned buffer regardless of file sizes", () => {
for (const size of [0, 1, SECTOR - 1, SECTOR, SECTOR + 1, SECTOR * 5 - 1]) {
const iso = buildIso("STACKCFG", [sampleFile("a.bin", size)]);
expect(iso.length % SECTOR).toBe(0);
}
});
});
describe("buildIso — multiple file sector layout", () => {
it("assigns non-overlapping extents to all files", () => {
const files: IsoFile[] = [
sampleFile("a.bin", 10, 0x01),
sampleFile("b.bin", SECTOR, 0x02),
sampleFile("c.bin", SECTOR * 2 + 500, 0x03),
sampleFile("d.bin", 1, 0x04),
];
const iso = buildIso("STACKCFG", files);
const svd = readSector(iso, 17);
const rootSector = svd.readUInt32LE(156 + 2);
const records = parseDirRecords(readSector(iso, rootSector)).filter((r) => !r.isDir);
// Extents must be strictly ordered and non-overlapping.
const sorted = [...records].sort((a, b) => a.extentSector - b.extentSector);
for (let i = 1; i < sorted.length; i++) {
const prev = sorted[i - 1];
const prevEndSector = prev.extentSector + Math.max(1, Math.ceil(prev.dataLength / SECTOR));
expect(sorted[i].extentSector).toBeGreaterThanOrEqual(prevEndSector);
}
});
});