CodeCow/stack
1
0
Fork 0
mirror of https://github.com/stack-auth/stack.git synced 2026-07-20 21:29:36 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

External db sync (#1036)

Browse Source 
<img width="1920" height="969" alt="Screenshot 2026-02-04 at 9 47 16 AM"
src="https://github.com/user-attachments/assets/d7d0cd04-0051-4fc4-b857-e6f87ee97a59"
/>

**This PR revolves around the following components**
1. Sequencer - sequences the updates in the internal db
2. Poller - polls for the latest updates to sync with the external db
3. Outgoing Request Handler - essentially a trigger that can make http
requests based on a change in the internal db
4. Sync Engine - syncs with the latest changes from the internal db to
the external db

**What has been done**
- Added a global sequence id for ProjectUser, ContactChannel and
DeletedRow.
- Added the deletedRow table to keep track of the rows that were deleted
across ProjectUser and ContactChannel.
- Added the OutgoingRequest table to keep track of the outgoing requests
- Added function for the sequencer to call to sequence updates
- Added a sequencer that sequences all the changes in the internal db
every 50 ms
- Added a poller that polls for the latest changes in the internal db
every 50 ms, and adds to a queue
- Added a Vercel cron that calls sequencer and poller every minute
- Added a queue that fulfills the outgoing requests by making http calls
(for external db sync, it calls the sync engine endpoint)
- Added a sync engine that uses the defined sql mapping query in the
user's schema to pull in the changes for the user, and sync them with
the external db
- Added tests to test out each functionality


**How to review this PR:**
1. Review the migrations (sequence id, deletedRow, triggers, backlog
sync) (all files created under the migrations folder)
2. Review sequencer
3. Review poller
4. Review the changes in schema
5. Review sync-engine (the function, and it's helper file)
6. Review the schema changes, and query mappings
7. Review the tests (basic, advanced and race, along with the helper
file)
8. Review the changes made in Dockerfile to support local testing using
the postgres docker

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Introduces a cron-driven external DB sync pipeline with global
sequencing, internal poller and webhook sync engine, new DB
tables/functions, config schema/mappings, and comprehensive e2e tests.
> 
> - **Database (Prisma/Migrations)**:
> - Add global sequence (`global_seq_id`) and
`sequenceId`/`shouldUpdateSequenceId` to `ProjectUser`,
`ContactChannel`, `DeletedRow` with partial indexes.
> - Create `DeletedRow` (capture deletes) and `OutgoingRequest` (queue)
tables; add unique/indexes.
> - Add triggers/functions: `log_deleted_row`,
`reset_sequence_id_on_update`, `backfill_null_sequence_ids`,
`enqueue_tenant_sync`.
> - **Backend/API**:
> - New internal routes: `GET
/api/latest/internal/external-db-sync/sequencer`, `GET /poller`, `POST
/sync-engine` (Upstash-verified) for sync orchestration.
> - Add cron wiring: `vercel.json` schedules and local
`scripts/run-cron-jobs.ts`; start in dev via `dev` script.
> - Tweak route handler (remove noisy logging) without behavior change.
> - **Sync Engine**:
> - Implement `src/lib/external-db-sync.ts` to read tenant mappings and
upsert to external Postgres (schema bootstrap, param checks,
sequencing).
> - Add default mappings `DEFAULT_DB_SYNC_MAPPINGS` and config schema
`dbSync.externalDatabases` in shared config.
> - **Testing/Infra**:
> - Add extensive e2e tests (basics, advanced, race conditions) for
sequencing, idempotency, deletes, pagination, multi-mapping, and
permissions.
> - Docker compose: add `external-db-test` Postgres for tests; e2e deps
for `pg` types.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3f2a8efcfb. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* External PostgreSQL sync: automatic, batched replication with
mappings, resume/idempotency, and on-demand enqueueing.

* **Admin UI**
* Real-time External DB Sync dashboard and status API showing
per-mapping backlog, sequencer/poller/sync-engine telemetry, and fusebox
controls.

* **Tests**
* Large e2e suite: basic, advanced, race, high-volume tests and test
utilities for external DB sync.

* **Chores**
* DB migrations, CI/workflow updates, background cron runner and
local/dev test support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Konsti Wohlwend <[email protected]>
Co-authored-by: Bilal Godil <[email protected]>
This commit is contained in:
aadesh18
2026-02-05 12:04:31 -08:00
committed by GitHub
co-authored by Konsti Wohlwend Bilal Godil
parent cf86ea5952
commit 2055d98dea
51 changed files with 6537 additions and 572 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/db-migration-backwards-compatibility.yaml
+11 -1
View File
@@ -193,6 +193,17 @@ jobs:
wait-for: 30s
log-output-if: true
- name: Start run-cron-jobs in background
uses: JarvusInnovations/[email protected]
if: ${{ hashFiles('apps/backend/scripts/run-cron-jobs.ts') != '' }}
with:
run: pnpm -C apps/backend run with-env:dev tsx scripts/run-cron-jobs.ts --log-order=stream &
wait-on: |
http://localhost:8102
tail: true
wait-for: 30s
log-output-if: true
- name: Wait 10 seconds
run: sleep 10
@@ -230,4 +241,3 @@ jobs:
steps:
- name: No migration changes detected
run: echo "No changes to migrations folder detected. Skipping backwards compatibility test."
.github/workflows/e2e-api-tests.yaml
+20 -5
View File
@@ -19,6 +19,9 @@ jobs:
NODE_ENV: test
STACK_ENABLE_HARDCODED_PASSKEY_CHALLENGE_FOR_TESTING: yes
STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@localhost:8128/stackframe"
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
strategy:
matrix:
@@ -100,6 +103,9 @@ jobs:
- name: Wait on Svix
run: pnpx wait-on tcp:localhost:8113
- name: Wait on QStash
run: pnpx wait-on tcp:localhost:8125
- name: Initialize database
run: pnpm run db:init
@@ -140,20 +146,29 @@ jobs:
tail: true
wait-for: 30s
log-output-if: true
- name: Start run-cron-jobs in background
uses: JarvusInnovations/[email protected]
with:
run: pnpm -C apps/backend run run-cron-jobs:test --log-order=stream &
wait-on: |
http://localhost:8102
tail: true
wait-for: 30s
log-output-if: true
- name: Wait 10 seconds
run: sleep 10
- name: Run tests
run: pnpm test ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
run: pnpm test run ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
- name: Run tests again, to make sure they are stable (attempt 1)
- name: Run tests again (attempt 1)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
run: pnpm test run ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
- name: Run tests again, to make sure they are stable (attempt 2)
- name: Run tests again (attempt 2)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
run: pnpm test run ${{ matrix.freestyle-mode == 'prod' && '--min-workers=1 --max-workers=1' || '' }}
- name: Verify data integrity
run: pnpm run verify-data-integrity --no-bail
.github/workflows/e2e-custom-base-port-api-tests.yaml
+20 -5
View File
@@ -19,6 +19,9 @@ jobs:
STACK_ENABLE_HARDCODED_PASSKEY_CHALLENGE_FOR_TESTING: yes
STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@localhost:6728/stackframe"
NEXT_PUBLIC_STACK_PORT_PREFIX: "67"
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
strategy:
matrix:
@@ -94,6 +97,9 @@ jobs:
- name: Wait on Svix
run: pnpx wait-on tcp:localhost:6713
- name: Wait on QStash
run: pnpx wait-on tcp:localhost:6725
- name: Initialize database
run: pnpm run db:init
@@ -134,20 +140,29 @@ jobs:
tail: true
wait-for: 30s
log-output-if: true
- name: Start run-cron-jobs in background
uses: JarvusInnovations/[email protected]
with:
run: pnpm -C apps/backend run run-cron-jobs --log-order=stream &
wait-on: |
http://localhost:6702
tail: true
wait-for: 30s
log-output-if: true
- name: Wait 10 seconds
run: sleep 10
- name: Run tests
run: pnpm test
run: pnpm test run
- name: Run tests again, to make sure they are stable (attempt 1)
- name: Run tests again (attempt 1)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test
run: pnpm test run
- name: Run tests again, to make sure they are stable (attempt 2)
- name: Run tests again (attempt 2)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test
run: pnpm test run
- name: Verify data integrity
run: pnpm run verify-data-integrity --no-bail
.github/workflows/e2e-source-of-truth-api-tests.yaml
+21 -5
View File
@@ -17,9 +17,13 @@ jobs:
env:
NODE_ENV: test
STACK_ENABLE_HARDCODED_PASSKEY_CHALLENGE_FOR_TESTING: yes
STACK_ACCESS_TOKEN_EXPIRATION_TIME: 30m
STACK_OVERRIDE_SOURCE_OF_TRUTH: '{"type": "postgres", "connectionString": "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@localhost:8128/source-of-truth-db?schema=sot-schema"}'
STACK_TEST_SOURCE_OF_TRUTH: true
STACK_DATABASE_CONNECTION_STRING: "postgres://postgres:PASSWORD-PLACEHOLDER--uqfEC1hmmv@localhost:8128/stackframe"
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
strategy:
matrix:
@@ -95,6 +99,9 @@ jobs:
- name: Wait on Svix
run: pnpx wait-on tcp:localhost:8113
- name: Wait on QStash
run: pnpx wait-on tcp:localhost:8125
- name: Create source-of-truth database and schema
run: |
@@ -140,20 +147,29 @@ jobs:
tail: true
wait-for: 30s
log-output-if: true
- name: Start run-cron-jobs in background
uses: JarvusInnovations/[email protected]
with:
run: pnpm -C apps/backend run run-cron-jobs --log-order=stream &
wait-on: |
http://localhost:8102
tail: true
wait-for: 30s
log-output-if: true
- name: Wait 10 seconds
run: sleep 10
- name: Run tests
run: pnpm test
run: pnpm test run --exclude "**/external-db-sync*.test.ts" # external-db-sync does not support external sot
- name: Run tests again, to make sure they are stable (attempt 1)
- name: Run tests again (attempt 1)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test
run: pnpm test run --exclude "**/external-db-sync*.test.ts"
- name: Run tests again, to make sure they are stable (attempt 2)
- name: Run tests again (attempt 2)
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
run: pnpm test
run: pnpm test run --exclude "**/external-db-sync*.test.ts"
- name: Verify data integrity
run: pnpm run verify-data-integrity --no-bail
.github/workflows/restart-dev-and-test-with-custom-base-port.yaml
+4 -1
View File
@@ -19,6 +19,9 @@ jobs:
runs-on: ubicloud-standard-16
env:
NEXT_PUBLIC_STACK_PORT_PREFIX: "69"
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
steps:
- uses: actions/checkout@v6
@@ -38,7 +41,7 @@ jobs:
run: pnpm run restart-dev-environment
- name: Run tests
run: pnpm run test --reporter=verbose
run: pnpm run test run --reporter=verbose
- name: Print dev server logs
run: cat dev-server.log.untracked.txt
.github/workflows/restart-dev-and-test.yaml
+5 -1
View File
@@ -17,6 +17,10 @@ env:
jobs:
restart-dev-and-test:
runs-on: ubicloud-standard-16
env:
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
steps:
- uses: actions/checkout@v6
@@ -36,7 +40,7 @@ jobs:
run: pnpm run restart-dev-environment
- name: Run tests
run: pnpm run test --reporter=verbose
run: pnpm run test run --reporter=verbose
- name: Print dev server logs
run: cat dev-server.log.untracked.txt
.github/workflows/setup-tests-with-custom-base-port.yaml
+5 -1
View File
@@ -19,6 +19,9 @@ jobs:
runs-on: ubicloud-standard-16
env:
NEXT_PUBLIC_STACK_PORT_PREFIX: "69"
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
steps:
- uses: actions/checkout@v6
@@ -46,4 +49,5 @@ jobs:
tail: true
wait-for: 120s
log-output-if: true
- run: pnpm run test --reporter=verbose
- name: Run tests
run: pnpm run test run --reporter=verbose
.github/workflows/setup-tests.yaml
+6 -1
View File
@@ -17,6 +17,10 @@ env:
jobs:
setup-tests:
runs-on: ubicloud-standard-16
env:
STACK_FORCE_EXTERNAL_DB_SYNC: "true"
STACK_EXTERNAL_DB_SYNC_MAX_DURATION_MS: "20000"
STACK_EXTERNAL_DB_SYNC_DIRECT: "false"
steps:
- uses: actions/checkout@v6
@@ -43,4 +47,5 @@ jobs:
tail: true
wait-for: 120s
log-output-if: true
- run: pnpm run test --reporter=verbose
- name: Run tests
run: pnpm run test run --reporter=verbose