<img width="1920" height="969" alt="Screenshot 2026-02-04 at 9 47 16 AM"
src="https://github.com/user-attachments/assets/d7d0cd04-0051-4fc4-b857-e6f87ee97a59"
/>
**This PR revolves around the following components**
1. Sequencer - sequences the updates in the internal db
2. Poller - polls for the latest updates to sync with the external db
3. Outgoing Request Handler - essentially a trigger that can make http
requests based on a change in the internal db
4. Sync Engine - syncs with the latest changes from the internal db to
the external db
**What has been done**
- Added a global sequence id for ProjectUser, ContactChannel and
DeletedRow.
- Added the deletedRow table to keep track of the rows that were deleted
across ProjectUser and ContactChannel.
- Added the OutgoingRequest table to keep track of the outgoing requests
- Added function for the sequencer to call to sequence updates
- Added a sequencer that sequences all the changes in the internal db
every 50 ms
- Added a poller that polls for the latest changes in the internal db
every 50 ms, and adds to a queue
- Added a Vercel cron that calls sequencer and poller every minute
- Added a queue that fulfills the outgoing requests by making http calls
(for external db sync, it calls the sync engine endpoint)
- Added a sync engine that uses the defined sql mapping query in the
user's schema to pull in the changes for the user, and sync them with
the external db
- Added tests to test out each functionality
**How to review this PR:**
1. Review the migrations (sequence id, deletedRow, triggers, backlog
sync) (all files created under the migrations folder)
2. Review sequencer
3. Review poller
4. Review the changes in schema
5. Review sync-engine (the function, and it's helper file)
6. Review the schema changes, and query mappings
7. Review the tests (basic, advanced and race, along with the helper
file)
8. Review the changes made in Dockerfile to support local testing using
the postgres docker
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces a cron-driven external DB sync pipeline with global
sequencing, internal poller and webhook sync engine, new DB
tables/functions, config schema/mappings, and comprehensive e2e tests.
>
> - **Database (Prisma/Migrations)**:
> - Add global sequence (`global_seq_id`) and
`sequenceId`/`shouldUpdateSequenceId` to `ProjectUser`,
`ContactChannel`, `DeletedRow` with partial indexes.
> - Create `DeletedRow` (capture deletes) and `OutgoingRequest` (queue)
tables; add unique/indexes.
> - Add triggers/functions: `log_deleted_row`,
`reset_sequence_id_on_update`, `backfill_null_sequence_ids`,
`enqueue_tenant_sync`.
> - **Backend/API**:
> - New internal routes: `GET
/api/latest/internal/external-db-sync/sequencer`, `GET /poller`, `POST
/sync-engine` (Upstash-verified) for sync orchestration.
> - Add cron wiring: `vercel.json` schedules and local
`scripts/run-cron-jobs.ts`; start in dev via `dev` script.
> - Tweak route handler (remove noisy logging) without behavior change.
> - **Sync Engine**:
> - Implement `src/lib/external-db-sync.ts` to read tenant mappings and
upsert to external Postgres (schema bootstrap, param checks,
sequencing).
> - Add default mappings `DEFAULT_DB_SYNC_MAPPINGS` and config schema
`dbSync.externalDatabases` in shared config.
> - **Testing/Infra**:
> - Add extensive e2e tests (basics, advanced, race conditions) for
sequencing, idempotency, deletes, pagination, multi-mapping, and
permissions.
> - Docker compose: add `external-db-test` Postgres for tests; e2e deps
for `pg` types.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3f2a8efcfb. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* External PostgreSQL sync: automatic, batched replication with
mappings, resume/idempotency, and on-demand enqueueing.
* **Admin UI**
* Real-time External DB Sync dashboard and status API showing
per-mapping backlog, sequencer/poller/sync-engine telemetry, and fusebox
controls.
* **Tests**
* Large e2e suite: basic, advanced, race, high-volume tests and test
utilities for external DB sync.
* **Chores**
* DB migrations, CI/workflow updates, background cron runner and
local/dev test support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
Co-authored-by: Bilal Godil <bg2002@gmail.com>
<!--
ONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md
-->
<!-- RECURSEML_SUMMARY:START -->
## High-level PR Summary
This PR changes the default development ports for several background
services to avoid conflicts. PostgreSQL moves from port `5432` to
`8128`, Inbucket SMTP from `2500` to `8129`, Inbucket POP3 from `1100`
to `8130`, and the OpenTelemetry collector from `4318` to `8131`. All
references across configuration files, Docker Compose setups,
environment files, CI/CD workflows, test files, and documentation have
been updated to reflect these new port assignments. A knowledge base
document has been added to document the new port mappings.
⏱️ Estimated Review Time: 15-30 minutes
<details>
<summary>💡 Review Order Suggestion</summary>
| Order | File Path |
| --- | --- |
| 1 | `claude/CLAUDE-KNOWLEDGE.md` |
| 2 | `apps/dev-launchpad/public/index.html` |
| 3 | `docker/dependencies/docker.compose.yaml` |
| 4 | `docker/emulator/docker.compose.yaml` |
| 5 | `apps/backend/.env` |
| 6 | `apps/backend/.env.development` |
| 7 | `docker/server/.env.example` |
| 8 | `package.json` |
| 9 | `.devcontainer/devcontainer.json` |
| 10 | `apps/e2e/.env.development` |
| 11 | `.github/workflows/check-prisma-migrations.yaml` |
| 12 | `.github/workflows/docker-server-test.yaml` |
| 13 | `.github/workflows/e2e-api-tests.yaml` |
| 14 | `.github/workflows/e2e-source-of-truth-api-tests.yaml` |
| 15 | `.github/workflows/restart-dev-and-test.yaml` |
| 16 |
`apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts`
|
| 17 | `apps/e2e/tests/backend/endpoints/api/v1/internal/email.test.ts`
|
| 18 | `apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts` |
| 19 |
`apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.ts` |
| 20 | `apps/e2e/tests/backend/workflows.test.ts` |
| 21 | `docs/templates/others/self-host.mdx` |
</details>
[](https://discord.gg/n3SsVDAW6U)
[
<!-- RECURSEML_SUMMARY:END -->
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> This PR introduces customizable development ports using
`NEXT_PUBLIC_STACK_PORT_PREFIX`, updating configurations, documentation,
and tests accordingly.
>
> - **Behavior**:
> - Default development ports for services are now customizable via
`NEXT_PUBLIC_STACK_PORT_PREFIX`.
> - PostgreSQL port changed from `5432` to
`${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}28`.
> - Inbucket SMTP port changed from `2500` to
`${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}29`.
> - Inbucket POP3 port changed from `1100` to
`${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}30`.
> - OpenTelemetry collector port changed from `4318` to
`${NEXT_PUBLIC_STACK_PORT_PREFIX:-81}31`.
> - **Configuration**:
> - Updated `docker.compose.yaml` to use new port variables for services
like PostgreSQL, Inbucket, and OpenTelemetry.
> - Environment files in `apps/backend`, `apps/dashboard`, and
`apps/e2e` updated to use `NEXT_PUBLIC_STACK_PORT_PREFIX`.
> - `package.json` scripts updated to reflect new port configurations.
> - **Documentation**:
> - Added `CLAUDE-KNOWLEDGE.md` to document new port mappings.
> - Updated `self-host.mdx` to reflect new port configurations.
> - **Testing**:
> - Updated test files in `apps/e2e/tests` to use new port
configurations.
> - Added `helpers/ports.ts` for port-related utilities in tests.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=stack-auth%2Fstack-auth&utm_source=github&utm_medium=referral)<sup>
for 76ef55f58f. You can
[customize](https://app.ellipsis.dev/stack-auth/settings/summaries) this
summary. It will automatically update as commits are pushed.</sup>
----
<!-- ELLIPSIS_HIDDEN -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enable configurable development ports via a
NEXT_PUBLIC_STACK_PORT_PREFIX, allowing parallel local environments with
custom port prefixes.
- **Bug Fixes**
- Updated local service port mappings and CI/workflow settings so
tooling and tests use the new prefixed ports consistently.
- **Documentation**
- Added docs and contributor guidance for running multiple parallel
workspaces with custom port prefixes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: N2D4 <N2D4@users.noreply.github.com>
<!--
Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md
-->
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> Add Convex integration with new auth helpers, update access token
handling, and include documentation, examples, and tests for the new
features.
>
> - **Features**:
> - Add Convex integration with new auth helpers for Convex clients and
HTTP in `client-app-impl.ts` and `server-app-impl.ts`.
> - Support for Convex context in user APIs and partial user retrieval.
> - Access tokens now include `is_anonymous` for better anonymous
handling in `tokens.tsx`.
> - **Documentation**:
> - Add Convex integration guide in `docs/templates/others/convex.mdx`.
> - Update docs navigation in `docs/docs-platform.yml` and
`docs/templates/meta.json`.
> - **Examples**:
> - Add Convex + Next.js example app in `examples/convex` with auth
wiring, functions, schema, and UI.
> - **Tests**:
> - Add E2E tests for Convex auth flows in `convex.test.ts`.
> - Update JWT payload checks in `backend-helpers.ts` and
`anonymous-comprehensive.test.ts`.
> - **Chores**:
> - Add Convex dependencies in `package.json` files.
> - Update CI steps for example environments in `e2e-api-tests.yaml` and
`e2e-source-of-truth-api-tests.yaml`.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=stack-auth%2Fstack-auth&utm_source=github&utm_medium=referral)<sup>
for aa0983a8b7. You can
[customize](https://app.ellipsis.dev/stack-auth/settings/summaries) this
summary. It will automatically update as commits are pushed.</sup>
----
<!-- ELLIPSIS_HIDDEN -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Convex integration: auth helpers for Convex clients/HTTP, Convex-aware
user APIs, and partial-user retrieval (token/convex).
- Access tokens now surface is_anonymous for clearer anonymous handling.
- **Documentation**
- Added Convex integration guide and nav entries.
- **Examples**
- New Convex + Next.js example app with auth wiring, backend functions,
schema, and UI.
- **Tests**
- Added E2E tests covering Convex auth flows and JWT payload checks.
- **Chores**
- Added Convex deps, CI env steps, and workspace/test config updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Konstantin Wohlwend <n2d4xc@gmail.com>
<!--
Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md
-->
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> Introduces an automated database migration system, replacing manual
Prisma commands with new scripts and updating workflows, configurations,
and tests accordingly.
>
> - **Auto-Migration System**:
> - Introduces `db-migrations.ts` script for handling database
migrations automatically.
> - Adds utility functions in `utils.tsx` for managing migration files.
> - Implements `applyMigrations` and `runMigrationNeeded` in `index.tsx`
for executing migrations.
> - **Workflow and Scripts**:
> - Updates GitHub workflows (`check-prisma-migrations.yaml`,
`e2e-api-tests.yaml`) to use new migration commands.
> - Replaces `prisma migrate` commands with `db:init`, `db:migrate`,
etc., in `package.json` and `README.md`.
> - **Testing**:
> - Adds `auto-migration.tests.ts` for testing migration logic and
concurrency handling.
> - **Configuration**:
> - Updates `.env.development` and `vitest.config.ts` for new
environment variables and paths.
> - Modifies `turbo.json` and `package.json` to include new migration
tasks and scripts.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=stack-auth%2Fstack-auth&utm_source=github&utm_medium=referral)<sup>
for 2c24183879. You can
[customize](https://app.ellipsis.dev/stack-auth/settings/summaries) this
summary. It will automatically update as commits are pushed.</sup>
<!-- ELLIPSIS_HIDDEN -->
---------
Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
