CodeCow/stack
1
0
Fork 0
mirror of https://github.com/stack-auth/stack.git synced 2026-07-20 21:29:36 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Fast-start local emulator via RAM snapshot + live secret rotation (#1340)

Browse Source 
## Summary

`stack emulator start` now resumes a fully-warm VM snapshot instead of
cold-booting, bringing startup from 30–120s down to ~5–8s with
per-install secret rotation, or ~2.5s with rotation opt-out. The
snapshot is captured **locally on first `stack emulator pull`**, not
shipped from CI — QEMU migration state isn't portable across
accelerators (KVM/HVF/TCG) or `-cpu max` feature sets, so a CI-captured
snapshot couldn't resume reliably on arbitrary user hardware.

Also bundles a pile of CLI QoL fixes (progress bars, PR/run artifact
pulls, PR-build download, native-TS ISO writer replacing
`hdiutil`/`mkisofs`/`genisoimage` host dep, unit tests).

| Scenario | Before | After |
|---|---|---|
| Cold boot (no snapshot) | 30–120s | same, works as fallback |
| `stack emulator pull` (one-time, includes local snapshot capture) |
~30s download | ~30s download + ~1–3 min cold-boot capture |
| Snapshot resume, normal start | — | **~5–8s** |
| Snapshot resume, `EMULATOR_NO_ROTATION=1` | — | **~2.5s** |

Backend (`/health?db=1`) and dashboard (`/handler/sign-in`) return 200
on all paths. Two successive snapshot resumes produce different rotated
PCK/SSK/SAK/CRON_SECRET values per install.

## How it works

**Build (CI)** — `docker/local-emulator/qemu/build-image.sh`:

1. Cloud-init provisioning runs to completion (migrations, seed,
slim-image) producing `stack-emulator-<arch>.qcow2`.
2. Image is built with a topology compatible with later snapshot capture
(pinned SMP=4, phantom seed/bundle ISOs, STACKCFG runtime ISO mounted at
build time, qemu-guest-agent running, placeholder hex secrets baked in
under `STACK_EMULATOR_BUILD_SNAPSHOT=1`).
3. CI publishes **only the qcow2** — no `.savevm.zst` ships.

**Pull (user's machine)** —
`packages/stack-cli/src/commands/emulator.ts` + `run-emulator.sh
capture`:

1. `stack emulator pull` downloads the qcow2 with a progress bar (or
from a PR / workflow run via `--pr` / `--run`).
2. CLI invokes `run-emulator.sh capture`: cold-boots the qcow2 with a
matching device layout (phantom ISOs, fsdev, pcie-root-port, virtfs
detached — migration-incompatible), waits for backend+dashboard health,
then drives QMP: `stop` → set `mapped-ram` + `multifd` caps → `migrate
file:state.raw` → poll `query-migrate` → `quit`. Raw mapped-ram file is
zstd-compressed to `stack-emulator-<arch>.savevm.zst` in the images dir.
3. `--skip-snapshot` opts out (first `start` will then cold-boot).

**Runtime** — `run-emulator.sh start`:

1. Launch QEMU with `-incoming defer` when a `.savevm.zst` is present;
decompress on first use, keep the `.raw` cached for subsequent starts.
2. QMP: same `mapped-ram` + `multifd` caps → `migrate-incoming
file:<.raw>` → poll for `paused` → `cont`.
3. Generate fresh per-install secrets on the host; pipe them
base64-encoded through QGA `guest-exec input-data` →
`trigger-fast-rotate` in the guest → `docker exec -e … rotate-secrets`.
4. `rotate-secrets` in the container: validate keys (hex-only), targeted
`sed` on the placeholder PCK across built JS, `UPDATE ApiKeySet`,
`supervisorctl restart stack-app cron-jobs` (with
`stopasgroup`/`killasgroup` so the Node children actually die and
release their ports).
5. Poll backend+dashboard health; if anything fails, clean up and fall
back to cold boot transparently.

**Security model**: placeholder hex values are baked into the snapshot
(`00…ff` PCK, `00…ee` SSK, `00…dd` SAK, `00…cc` CRON_SECRET). They are
non-secret by construction. Real per-install secrets are generated at
each `emulator start` and never leave the host.

## CLI changes (`packages/stack-cli`)

- **`src/lib/iso.ts`** (new): native TypeScript ISO 9660 + Joliet
writer, replacing the host-side `hdiutil`/`mkisofs`/`genisoimage`
dependency for generating the STACKCFG runtime config disk. Unit tests
in `src/lib/iso.test.ts`.
- **`src/commands/emulator.ts`**:
- `pull`: streamed downloads with progress bar + ETA; `--pr <number>`
and `--run <id>` to pull from a PR build's CI artifacts (uses
`extract-zip` for the nested zip); `--skip-snapshot` to opt out of the
one-time local capture.
- `start` (existing, extended): auto-pulls AND auto-captures when no
image exists, so first-ever `start` is self-bootstrapping; emits
`STACK_EMULATOR_CLI_WROTE_ISO=1` so the shell helper skips its own ISO
regen (avoids the genisoimage host dep).
- `capture` (new, invoked by `pull` and the auto-pull path of `start`):
drives the local snapshot capture via `run-emulator.sh`.
- `status`, `stop`, `reset`, `list-releases`: preflight +
path-resolution tightening (`STACK_EMULATOR_HOME` → images/run dirs).
  - Unit tests in `src/commands/emulator.test.ts`.
- **`EMULATOR_NO_ROTATION=1`** env var skips the post-resume rotation
(intended for tests/CI where the placeholder secrets are fine — comes
with a loud warning).

## CI (`.github/workflows/qemu-emulator-build.yaml`)

- Builds **QEMU 10.2.2 from source** (cached), because
`mapped-ram`/`multifd` migration capabilities aren't available in the
distro's QEMU. Enables KVM on ubicloud runners so amd64 boots at
hardware speed.
- amd64 + arm64 both build on the same amd64 matrix
(`ubicloud-standard-8`); arm64 runs under cross-arch TCG (provisioning
only — boot/verify smoke test is amd64-only).
- Verification now runs through the CLI: `emulator start` → `emulator
status` → `emulator stop` against the freshly-built qcow2 (via
`STACK_EMULATOR_HOME` pointing at the workspace, so the CLI doesn't
silently auto-pull a prior release).
- Packages **only** the qcow2. No `.savevm.zst` upload / publish.
- Release notes updated.

## Key files

**Shell / guest:**
- `docker/local-emulator/qemu/build-image.sh` — snapshot-compatible
device topology + STACKCFG runtime ISO at build time
- `docker/local-emulator/qemu/run-emulator.sh` — `start`, `capture`,
`stop`, `reset`, `status`; `-incoming defer`, `.raw` cache, QGA-driven
rotation, cold-boot fallback
- `docker/local-emulator/qemu/common.sh` (new) — shared `qmp_session` +
`capture_vm_state` (factored out so build-image.sh and run-emulator.sh
share the capture path)
- `docker/local-emulator/qemu/cloud-init/emulator/user-data` —
placeholder secrets in snapshot mode, `wait-for-stack-ready`,
`trigger-fast-rotate`, qemu-guest-agent enabled
- `docker/local-emulator/rotate-secrets.sh` (new) — in-container
rotation (sed + UPDATE + supervisorctl)
- `docker/local-emulator/supervisord.conf` — `stopasgroup`/`killasgroup`
on `stack-app` and `cron-jobs`
- `docker/local-emulator/entrypoint.sh` — only mint CRON_SECRET if unset
(placeholder supplied in snapshot mode via --env-file)
- `docker/local-emulator/Dockerfile` — ships `rotate-secrets` to
`/usr/local/bin`
- `docker/server/entrypoint.sh` — source
`/run/stack-auth/rotated-secrets.env`; skip full-tree sentinel scan on
warm restarts via marker

**CLI:**
- `packages/stack-cli/src/lib/iso.ts` (new) + `iso.test.ts` (new)
- `packages/stack-cli/src/commands/emulator.ts` + `emulator.test.ts`
(new)
- `packages/stack-cli/vitest.config.ts` (new)

**CI:**
- `.github/workflows/qemu-emulator-build.yaml`

## Test plan

- [x] `docker/local-emulator/qemu/build-image.sh {amd64,arm64}` produces
`stack-emulator-<arch>.qcow2` with snapshot-compatible topology
- [x] `stack emulator pull` downloads qcow2 with progress, then captures
locally (~1–3 min) and writes `stack-emulator-<arch>.savevm.zst` in the
images dir
- [x] `stack emulator pull --skip-snapshot` stops after download
- [x] `stack emulator pull --pr <n>` / `--run <id>` pull from PR /
workflow run artifacts
- [x] `stack emulator start` on a fresh dir auto-pulls **and**
auto-captures, then starts; subsequent starts fast-resume in ~5–8s;
backend + dashboard return 200
- [x] `EMULATOR_NO_ROTATION=1 stack emulator start` completes in ~2.5s;
backend + dashboard return 200 with warning printed
- [x] Two consecutive `emulator start` invocations produce different PCK
values in the internal `ApiKeySet` row
- [x] `stack emulator status` / `stop` / `reset` resolve paths from
`STACK_EMULATOR_HOME`
- [x] Verified end-to-end on arm64 macOS under HVF (capture ~50s,
fast-resume ~6.5s)
- [x] `pnpm lint` and `pnpm typecheck` pass; stack-cli unit tests (iso +
emulator) pass
- [ ] CI green on this PR (qemu-emulator-build matrix, smoke test)
- [ ] `gh release download emulator-<branch>-latest` contains only
`stack-emulator-<arch>.qcow2` once this PR merges and publish runs

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Snapshot fast-start/resume with optional warm-snapshot assets, runtime
ISO generation, and a cached QEMU build to speed emulator setup.
* CLI: streamed artifact downloads with progress, improved release/asset
handling, stronger preflight checks, and start/status/stop emulator
commands.
* Automated secret rotation and ability to apply rotated secrets at
container startup; supervisor control socket enabled.

* **Bug Fixes**
* More robust start/stop/resume flows with automatic fallback to cold
boot and improved process-group shutdown behavior.

* **Tests**
  * New tests for CLI utilities and ISO image generation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
BilalG1
2026-04-20 14:24:49 -07:00
committed by GitHub
parent 9f3ea9a766
commit 37ee5ec320
20 changed files with 2885 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/qemu-emulator-build.yaml
+169 -42
View File
@@ -22,8 +22,16 @@ concurrency:
env:
EMULATOR_IMAGE_NAME: stack-local-emulator
# Shell scripts (build-image.sh, run-emulator.sh) read these directly.
EMULATOR_IMAGE_DIR: ${{ github.workspace }}/docker/local-emulator/qemu/images
EMULATOR_RUN_DIR: ${{ github.workspace }}/docker/local-emulator/qemu/run
# The stack-cli ignores EMULATOR_IMAGE_DIR/RUN_DIR and derives its own paths
# from STACK_EMULATOR_HOME. Point it at the same workspace so `emulator
# start` finds the freshly-built qcow2 from build-image.sh and cold-boots
# it, instead of auto-pulling from a prior release. CI doesn't capture a
# savevm (EMULATOR_CAPTURE_SAVEVM defaults to 0); users capture locally
# on first `stack emulator pull`.
STACK_EMULATOR_HOME: ${{ github.workspace }}/docker/local-emulator/qemu
jobs:
build:
@@ -34,15 +42,16 @@ jobs:
fail-fast: false
matrix:
include:
# amd64 runs natively under KVM on ubicloud's amd64 runner.
# Both arches build on ubicloud's amd64 runner. amd64 uses KVM;
# arm64 runs under cross-arch TCG (slow, but only cloud-init
# provisioning has to complete — the boot/verify smoke test below
# is gated to amd64 because TCG can't boot Next.js in any
# reasonable time). Snapshots are NOT published — `stack emulator
# pull` captures one locally on first run, which is the only way
# to guarantee KVM/HVF/TCG + `-cpu max` compatibility on the
# user's machine.
- arch: amd64
runner: ubicloud-standard-8
# arm64 runs under cross-arch TCG on ubicloud's amd64 runner.
# No KVM for arm64 guests on an amd64 host; cortex-a72 + V8
# --jitless together sidestep the SIGTRAPs that cross-arch TCG
# hits on aggressive arm64 JIT code. Smoke test is still skipped
# because the backend can't come up reliably under cross-arch
# TCG within any sane window.
- arch: arm64
runner: ubicloud-standard-8
@@ -55,10 +64,60 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Install QEMU dependencies
# Node/pnpm are needed on both arches: arm64 also runs
# generate-env-development.mjs inside build-image.sh. amd64 additionally
# builds and runs the CLI for the verification steps below.
- uses: pnpm/action-setup@v4
with:
version: 10.23.0
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y qemu-system-x86 qemu-system-arm qemu-kvm qemu-utils genisoimage socat qemu-efi-aarch64
# qemu-utils gives us qemu-img; qemu-efi-aarch64 provides the arm64
# UEFI firmware. The actual qemu-system-* binaries come from the
# source build below — Ubuntu 24.04 ships QEMU 8.2 which predates
# the mapped-ram migration capability we rely on.
sudo apt-get install -y qemu-utils qemu-efi-aarch64 socat genisoimage zstd \
ninja-build pkg-config python3-venv \
libglib2.0-dev libpixman-1-dev libslirp-dev libepoxy-dev libgbm-dev
# QEMU 10.2.2 is required for the mapped-ram + multifd migration path
# used by the fast-resume snapshot. Cache the compiled prefix so CI
# only pays the ~5-8 min build cost once per runner image.
- name: Restore QEMU 10.2.2 cache
id: qemu-cache
uses: actions/cache@v4
with:
path: /opt/qemu
key: qemu-10.2.2-${{ runner.os }}-${{ runner.arch }}-v1
- name: Build QEMU 10.2.2 from source
if: steps.qemu-cache.outputs.cache-hit != 'true'
run: |
set -euxo pipefail
curl -fsSL https://download.qemu.org/qemu-10.2.2.tar.xz -o /tmp/qemu.tar.xz
mkdir -p /tmp/qemu-src
tar -xf /tmp/qemu.tar.xz -C /tmp/qemu-src --strip-components=1
cd /tmp/qemu-src
./configure --prefix=/opt/qemu \
--target-list=x86_64-softmmu,aarch64-softmmu \
--enable-kvm --enable-slirp --enable-tcg \
--disable-docs --disable-gtk --disable-sdl --disable-vnc \
--disable-guest-agent --disable-tools
make -j"$(nproc)"
sudo make install
- name: Put QEMU 10.2.2 on PATH
run: |
echo "/opt/qemu/bin" >> "$GITHUB_PATH"
/opt/qemu/bin/qemu-system-x86_64 --version
/opt/qemu/bin/qemu-system-aarch64 --version
- name: Enable KVM access
run: |
@@ -82,41 +141,56 @@ jobs:
- name: Generate emulator env
run: node docker/local-emulator/generate-env-development.mjs
# arm64 runs under cross-arch TCG on an amd64 runner; the backend's
# V8 TurboFan JIT re-triggers the SIGTRAPs we dodge in migrations
# with --no-opt, and even if it didn't, boot is too slow under TCG
# to verify in any sane window. amd64 KVM already exercises the
# service stack; real arm64 hosts have KVM for end-users.
- name: Start emulator and verify
# amd64 runs under KVM on the runner so we can boot the newly-built
# image to verify it works end-to-end before publishing. arm64 runs
# under cross-arch TCG on an amd64 host, which can't reliably boot
# Next.js within any sane window — skipped.
- name: Build stack-cli (for emulator CLI)
if: matrix.arch == 'amd64'
run: |
chmod +x docker/local-emulator/qemu/run-emulator.sh
EMULATOR_ARCH=${{ matrix.arch }} \
EMULATOR_READY_TIMEOUT=3200 \
docker/local-emulator/qemu/run-emulator.sh start
pnpm install --frozen-lockfile --filter '@stackframe/stack-cli...'
# Turbo's trailing `...` filter builds stack-cli AND its workspace
# deps (@stackframe/js, @stackframe/stack-shared, etc.) — stack-cli
# imports them at runtime from their dist/ outputs.
pnpm exec turbo run build --filter='@stackframe/stack-cli...'
- name: Start emulator and verify
if: matrix.arch == 'amd64'
env:
EMULATOR_ARCH: ${{ matrix.arch }}
EMULATOR_READY_TIMEOUT: 3200
EMULATOR_IMAGE_DIR: ${{ env.EMULATOR_IMAGE_DIR }}
EMULATOR_RUN_DIR: ${{ env.EMULATOR_RUN_DIR }}
run: node packages/stack-cli/dist/index.js emulator start
- name: Verify services are healthy
if: matrix.arch == 'amd64'
run: |
EMULATOR_ARCH=${{ matrix.arch }} \
docker/local-emulator/qemu/run-emulator.sh status
env:
EMULATOR_ARCH: ${{ matrix.arch }}
EMULATOR_IMAGE_DIR: ${{ env.EMULATOR_IMAGE_DIR }}
EMULATOR_RUN_DIR: ${{ env.EMULATOR_RUN_DIR }}
run: node packages/stack-cli/dist/index.js emulator status
- name: Stop emulator
if: always() && matrix.arch == 'amd64'
run: |
EMULATOR_ARCH=${{ matrix.arch }} \
docker/local-emulator/qemu/run-emulator.sh stop
env:
EMULATOR_ARCH: ${{ matrix.arch }}
EMULATOR_IMAGE_DIR: ${{ env.EMULATOR_IMAGE_DIR }}
EMULATOR_RUN_DIR: ${{ env.EMULATOR_RUN_DIR }}
run: node packages/stack-cli/dist/index.js emulator stop
- name: Package image
run: |
BASE_IMG="docker/local-emulator/qemu/images/stack-emulator-${{ matrix.arch }}.qcow2"
cp "$BASE_IMG" "stack-emulator-${{ matrix.arch }}.qcow2"
ls -lh "stack-emulator-${{ matrix.arch }}.qcow2"
- name: Upload image artifact
uses: actions/upload-artifact@v4
with:
name: qemu-emulator-${{ matrix.arch }}
path: stack-emulator-${{ matrix.arch }}.qcow2
if-no-files-found: error
retention-days: 30
compression-level: 0
@@ -134,31 +208,80 @@ jobs:
steps:
- uses: actions/checkout@v6
- name: Install QEMU dependencies
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y qemu-system-x86 qemu-utils genisoimage socat
sudo apt-get install -y qemu-utils socat zstd \
ninja-build pkg-config python3-venv \
libglib2.0-dev libpixman-1-dev libslirp-dev libepoxy-dev libgbm-dev
- name: Restore QEMU 10.2.2 cache
id: qemu-cache
uses: actions/cache@v4
with:
path: /opt/qemu
key: qemu-10.2.2-${{ runner.os }}-${{ runner.arch }}-v1
- name: Build QEMU 10.2.2 from source
if: steps.qemu-cache.outputs.cache-hit != 'true'
run: |
set -euxo pipefail
curl -fsSL https://download.qemu.org/qemu-10.2.2.tar.xz -o /tmp/qemu.tar.xz
mkdir -p /tmp/qemu-src
tar -xf /tmp/qemu.tar.xz -C /tmp/qemu-src --strip-components=1
cd /tmp/qemu-src
./configure --prefix=/opt/qemu \
--target-list=x86_64-softmmu,aarch64-softmmu \
--enable-kvm --enable-slirp --enable-tcg \
--disable-docs --disable-gtk --disable-sdl --disable-vnc \
--disable-guest-agent --disable-tools
make -j"$(nproc)"
sudo make install
- name: Put QEMU 10.2.2 on PATH
run: |
echo "/opt/qemu/bin" >> "$GITHUB_PATH"
/opt/qemu/bin/qemu-system-x86_64 --version
- uses: pnpm/action-setup@v4
with:
version: 10.23.0
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install stack-cli deps + build
run: |
pnpm install --frozen-lockfile --filter '@stackframe/stack-cli...'
# Turbo's trailing `...` filter builds stack-cli AND its workspace
# deps (@stackframe/js, @stackframe/stack-shared, etc.) — stack-cli
# imports them at runtime from their dist/ outputs.
pnpm exec turbo run build --filter='@stackframe/stack-cli...'
- name: Download built image
uses: actions/download-artifact@v4
with:
name: qemu-emulator-${{ matrix.arch }}
path: docker/local-emulator/qemu/images/
path: ${{ github.workspace }}/.stack-emulator-images/
- name: Generate emulator env
run: node docker/local-emulator/generate-env-development.mjs
- name: Place qcow2 into STACK_EMULATOR_HOME layout
run: |
mkdir -p "$STACK_EMULATOR_HOME/images"
cp "${{ github.workspace }}/.stack-emulator-images/stack-emulator-${{ matrix.arch }}.qcow2" "$STACK_EMULATOR_HOME/images/"
ls -lh "$STACK_EMULATOR_HOME/images/"
- name: Start emulator from artifact
# No savevm.zst artifact (users capture locally via `emulator pull`),
# so `emulator start` cold-boots the qcow2. Budget accordingly.
- name: Start emulator via CLI
run: |
chmod +x docker/local-emulator/qemu/run-emulator.sh docker/local-emulator/qemu/common.sh
EMULATOR_ARCH=${{ matrix.arch }} \
EMULATOR_READY_TIMEOUT=600 \
docker/local-emulator/qemu/run-emulator.sh start
node packages/stack-cli/dist/index.js emulator start
- name: Verify services are healthy
run: |
EMULATOR_ARCH=${{ matrix.arch }} \
docker/local-emulator/qemu/run-emulator.sh status
run: node packages/stack-cli/dist/index.js emulator status
- name: Smoke test — backend health
run: curl -sf http://localhost:26701/health?db=1
@@ -174,13 +297,11 @@ jobs:
- name: Stop emulator
if: always()
run: |
EMULATOR_ARCH=${{ matrix.arch }} \
docker/local-emulator/qemu/run-emulator.sh stop
run: node packages/stack-cli/dist/index.js emulator stop
- name: Print serial log on failure
if: failure()
run: tail -100 docker/local-emulator/qemu/run/vm/serial.log 2>/dev/null || true
run: tail -100 "$STACK_EMULATOR_HOME/run/vm/serial.log" 2>/dev/null || true
publish:
name: Publish to GitHub Releases
@@ -220,8 +341,14 @@ jobs:
### Images
| File | Description |
|------|-------------|
| \`stack-emulator-arm64.qcow2\` | ARM64 emulator image |
| \`stack-emulator-amd64.qcow2\` | AMD64 emulator image |
| \`stack-emulator-arm64.qcow2\` | ARM64 disk image |
| \`stack-emulator-amd64.qcow2\` | AMD64 disk image |
\`emulator pull\` downloads the qcow2 and captures a local fast-start
snapshot (~1-3 min). Subsequent \`emulator start\`s resume in ~3-8 s.
Snapshots are captured locally because QEMU migration state isn't
portable across accelerators (KVM / HVF / TCG) or \`-cpu max\`
feature sets.
### Usage
\`\`\`bash