mirror of
https://github.com/stack-auth/stack.git
synced 2026-07-20 21:29:36 +08:00
2f0d9bc6cb
## Problem
Session replay batches are uploaded as **raw, uncompressed JSON**. A
single large rrweb full snapshot (>900 KB) is dropped client-side in
`SessionRecorder._flush`, and any batch near the server's 1 MB body
limit is rejected. Meanwhile the **analytics events** path right next
door already gzips on the wire (`encodeAnalyticsBody` + a server-side
gunzip with zip-bomb guards) — replays just never got the same
treatment.
Notably, `dev` was already **red** on the SDK test `sends a single
oversized event alone without dropping it` (the code dropped at 900 KB
while the test expected the event to be sent). This change makes that
test pass.
## What this does
Ports the proven analytics compression pattern to session replays:
- **`packages/shared` (client transport):** generalize
`encodeAnalyticsBody` → `encodeGzipJsonBody` and wire it into
`sendSessionReplayBatch`. Replays now gzip via native
`CompressionStream` (`application/octet-stream`), with the same
fallbacks: keepalive flushes send plain JSON (so the request survives
page teardown), and a missing/throwing `CompressionStream` (Safari <
16.4) degrades to plain JSON.
- **`packages/template` (SDK recorder):** replace the 900 KB hard-drop
with `MAX_SINGLE_EVENT_BYTES = 8 MB`, kept in sync with the server's
decompressed cap. The wire is gzipped downstream, so an oversized single
event is now sent compressed instead of discarded; only an event that
could never fit *decompressed* is dropped.
- **`apps/backend` (route):** add `maybeDecodeBinaryBody` (gunzip with a
1 MB compressed cap + 8 MB `maxOutputLength` zip-bomb guard) via a
`.transform` on the batch route body schema. Plain-JSON bodies pass
through untouched.
## Why gzip / CompressionStream
Benchmarked on a real rrweb capture (Puppeteer, rrweb 1.1.3): a **1.09
MiB full snapshot → 134 KiB gzip (~8×) in ~6.6 ms**. Native
`CompressionStream` adds **0 bundle bytes** and was faster than fflate.
This mirrors what PostHog (gzip, native `CompressionStream` + fflate
fallback) and Sentry (fflate deflate in a worker) both ship — both rely
on compression, not splitting, as the primary mechanism for large
events.
## Tests
- **Client SDK** (`client-interface.test.ts`): gzip+octet-stream on
non-keepalive (round-trip + size assert), plain JSON on keepalive, and
both `CompressionStream`-unavailable fallbacks.
- **SDK recorder** (`session-replay.test.ts`): existing oversized-event
test now passes; new test that a >8 MB event is dropped while a
following normal event still sends.
- **Backend e2e** (`session-replays.test.ts`): gzipped happy path
(`compressed < 1MB < raw`), invalid gzip → 400, oversized compressed →
413, zip-bomb → 400. Verified passing against a live local backend.
## Verification notes
- `@hexclave/shared` + `@hexclave/template` typecheck pass; touched
files lint clean; SDK unit tests pass.
- Two **pre-existing, unrelated** local failures (each confirmed by
stashing this change and re-running on clean `dev`): a backend typecheck
error in `oauth/ssrf-protection.test.ts` (`NODE_ENV` read-only), and 3
session-replay **quota** e2e tests failing in the payments
`setSessionReplayItemQuantity` setup helper.
## Follow-up (out of scope)
The keepalive/unload path still sends uncompressed (64 KB browser
keepalive cap), so a very large final batch on page exit can still be
lost — not addressed here.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Gzip session replay batches so large rrweb snapshots are sent instead of
dropped. Keepalive flushes cap single-event size to avoid 413s on page
unload, and limits now include a 1 KiB envelope margin to match server
caps.
- **Bug Fixes**
- Client (`@hexclave/shared`): generalized `encodeAnalyticsBody` →
`encodeGzipJsonBody`; gzip via native `CompressionStream` for
`/session-replays/batch`; keepalive and missing/throwing
`CompressionStream` fall back to JSON.
- SDK recorder (`@hexclave/template`): normal flushes allow a single
event up to `MAX_SINGLE_EVENT_BYTES = 8 MiB - 1 KiB`; keepalive flushes
cap single-event size at `MAX_BATCH_UNCOMPRESSED_BYTES = 900 KB`; send
oversized events alone and rely on gzip; drop only above those
thresholds; log a distinct 413 warning with the count of buffered events
dropped and stop the loop.
- Backend: gunzip `application/octet-stream` bodies with caps (`1 MB`
compressed, `8 MiB` decompressed); invalid gzip → 400; oversized
compressed → 413; plain JSON passes through.
- Tests: added client encoding tests for session replays; SDK tests for
>8 MiB drop, keepalive drop, and 413 warning; backend e2e for gzip,
invalid gzip, oversized compressed, and zip-bomb. The oversized-event
SDK test now passes.
- **Refactors**
- Hoisted shared test helpers and trimmed comments.
<sup>Written for commit
|
||
|---|---|---|
| .. | ||
| tests | ||
| .env | ||
| .env.development | ||
| .eslintrc.cjs | ||
| LICENSE | ||
| package.json | ||
| tsconfig.json | ||
| vitest.config.ts | ||
