Commit Graph

180 Commits

Author SHA1 Message Date
Konstantin Wohlwend
c46767f4e8 Retry setup tests on non-staging branches to prevent flakeyness 2026-04-13 14:32:49 -07:00
Mantra
bb277d33c9
Backend fallback (cloud run) (#1306)
- Added support for `@opentelemetry/sdk-node` in the backend.
- Updated various dependencies including AWS SDK and OpenTelemetry
packages.
- Implemented graceful shutdown handling for non-Vercel runtimes in
`prisma-client.tsx`.
- Enhanced AWS credentials retrieval to support GCP Workload Identity
Federation.
- Introduced a Dockerfile for Cloud Run deployment, optimizing the
backend build process.
- Updated `.gitignore` to include Terraform runtime files and secrets.

This commit improves the backend's observability and deployment
flexibility, particularly for Cloud Run environments.

<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* OpenTelemetry observability with dynamic provider selection per
deployment.
  * Cloud Run trusted-proxy support for accurate client IP handling.
  * Graceful shutdown that waits for in-flight background work.
* New background-task handling to improve async webhook/email delivery
reliability.
* AWS credential providers added (Vercel OIDC & GCP Workload Identity
Federation).
  * Dockerized backend image for Cloud Run / self-host deployments.

* **Chores**
  * Updated dependencies for OpenTelemetry and AWS SDK support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Konstantin Wohlwend <n2d4xc@gmail.com>
2026-04-11 00:57:37 +00:00
Konstantin Wohlwend
71f96ee221 Remove restart-dev-and-test workflows 2026-04-10 15:16:50 -07:00
Konstantin Wohlwend
452690f6d7 Revert "Make restart-dev-and-test save backend logs"
This reverts commit 0ebca10091.
2026-04-10 15:15:54 -07:00
Konstantin Wohlwend
0ebca10091 Make restart-dev-and-test save backend logs 2026-04-10 10:57:11 -07:00
BilalG1
9e342da0f2
Fix cron jobs using dev env instead of test env in CI workflows (#1319)
The custom-base-port and db-migration-backwards-compatibility workflows
were running cron jobs with `with-env:dev` instead of `with-env:test`,
causing ClickHouse sync mismatches in verify-data-integrity.

<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Streamlined CI test workflows to standardize background cron job
startup for more consistent test runs.
* **Tests**
* Improved end-to-end test reliability by aligning background process
behavior across suites.
* **Bug Fixes**
* Enhanced data verification reliability by ensuring external database
sync before integrity checks and tightening comparison ordering for
certain records, reducing false mismatch detections.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-09 21:27:18 -07:00
Konsti Wohlwend
d6c067972d
Replace npx with pnpm exec (#1300) 2026-04-08 17:08:55 -07:00
Bilal Godil
e8428f57c0 pin node version to 24 to fix mintlify lint issue on lint-and-build gh action 2026-04-07 10:11:57 -07:00
Mantra
ce49eae155
emu with a q stuff (#1266)
Some checks failed
all-good: Did all the other checks pass? / all-good (push) Has been cancelled
Ensure Prisma migrations are in sync with the schema / check_prisma_migrations (22.x) (push) Has been cancelled
DB migration compat / Check if migrations changed (push) Has been cancelled
Docker Server Build and Push / Docker Build and Push Server (push) Has been cancelled
Docker Server Build and Run / docker (push) Has been cancelled
Runs E2E API Tests (Local Emulator) / E2E Tests (Local Emulator, Node ${{ matrix.node-version }}) (22.x) (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (mock, 22.x) (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (prod, 22.x) (push) Has been cancelled
Runs E2E API Tests with custom port prefix / build (22.x) (push) Has been cancelled
Lint & build / lint_and_build (latest) (push) Has been cancelled
Dev Environment Test With Custom Base Port / restart-dev-and-test-with-custom-base-port (push) Has been cancelled
Dev Environment Test / restart-dev-and-test (push) Has been cancelled
Run setup tests with custom base port / setup-tests-with-custom-base-port (push) Has been cancelled
Run setup tests / setup-tests (push) Has been cancelled
TOC Generator / TOC Generator (push) Has been cancelled
DB migration compat / Back-compat — Current branch migrations with ${{ needs.check-migrations-changed.outputs.base_branch }} branch code (push) Has been cancelled
DB migration compat / Forward-compat — Current branch code with ${{ needs.check-migrations-changed.outputs.base_branch }} branch migrations (push) Has been cancelled
DB migration compat / No migration changes (skipped) (push) Has been cancelled
commit 5d43722575b826a8ed8dbb6b828f48eae4bca02c
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Wed Mar 18 12:27:01 2026 -0700

    Add QEMU emulator snapshot functionality and reset command

- Introduced a new `emulator-qemu:reset` command in package.json to
clear snapshots and force a fresh boot of the emulator.
- Enhanced the `run-emulator.sh` script to support saving and restoring
snapshots, significantly reducing restart time from ~62s to ~4s.
- Implemented logic to check for existing snapshots and restore them
during startup, improving the emulator's efficiency.
- Updated documentation in CLAUDE-KNOWLEDGE.md to explain the new
snapshot restore process and its benefits.

These changes enhance the QEMU emulator's performance and usability for
developers, providing a more efficient workflow during development.

commit 3877445bdd83cb8690da18c8520bf260d2795172
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Wed Mar 18 11:55:18 2026 -0700

    Enhance QEMU emulator performance and configuration management

- Added optimizations to the QEMU emulator's app container startup
process, reducing startup time from ~92s to ~62s by using qcow2 backing
files and setting the working directory to /app.
- Updated the build-image.sh script to conditionally wait for background
processes, improving robustness.
- Modified the run-emulator.sh script to create the disk image using
qcow2 format instead of copying, enhancing efficiency.
- Adjusted the cloud-init user-data to set STACK_RUNTIME_WORK_DIR to
/app, streamlining file operations during container initialization.
- Improved the entrypoint script to avoid unnecessary file copying when
the working directory is set to /app.

These changes significantly enhance the performance and usability of the
QEMU emulator for developers.

commit e0b86d3f1d5c08e46d0d343bc632e2a8c5777845
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Wed Mar 18 11:07:55 2026 -0700

Refactor local emulator configuration management and enhance Docker
setup

- Removed redundant comments and improved code clarity in the local
emulator's route handling.
- Streamlined the Dockerfile and docker-compose.yaml for better
readability and maintenance.
- Updated entrypoint and initialization scripts to enhance service
startup processes.
- Introduced a new common script for QEMU emulator to centralize
architecture detection and firmware handling.
- Enhanced error handling in the host file bridge for improved
robustness.
    - Removed obsolete country code utilities to clean up the codebase.

These changes significantly improve the local emulator's configuration
management and overall setup experience for developers.

commit 4fb0f93c6cc4f749a14acf0228c261e180875609
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Wed Mar 18 10:24:53 2026 -0700

Implement local emulator file bridge for enhanced configuration
management

- Introduced a new host file bridge to facilitate reading and writing
configuration files between the local emulator and the host system.
- Refactored the local-emulator module to utilize the file bridge for
file operations, improving error handling and response validation.
- Added tests to ensure the file bridge functionality works as expected,
including handling of non-existent files and writing configurations.
- Updated the run-emulator script to start the file bridge
automatically, ensuring seamless integration during emulator startup.
- Enhanced documentation to reflect the new file bridge capabilities and
usage instructions.

These changes significantly improve the local emulator's ability to
manage configuration files, enhancing the development experience.

commit 3d18a7ce5bbf00a62a40a3f48f27856e79ecc62f
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Tue Mar 17 22:36:46 2026 -0700

    Refactor QEMU local emulator setup and enhance app bundle handling

- Introduced a new script for packaging Docker images into a compressed
app bundle, improving the emulator's deployment process.
- Updated build-image.sh to create a runtime configuration ISO, ensuring
better management of environment settings.
- Enhanced cloud-init user-data scripts for both dev-server and deps
guests, streamlining service setup and configuration.
- Improved the run-emulator.sh script to facilitate better handling of
runtime configurations and dependencies.
- Adjusted the .gitignore to include .DS_Store and removed obsolete
entries, cleaning up the repository.

These changes significantly enhance the local emulator's functionality
and reliability for developers.

commit 8a35fb1ce79898d73e2259e256c11b6fd9b0a584
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Tue Mar 17 21:52:24 2026 -0700

    Enhance local emulator functionality and configuration

- Updated package.json to improve the start-emulator command, providing
clearer dashboard and backend URLs.
- Added a new wait-until-emulator-is-ready command to ensure the
emulator is fully operational before proceeding.
- Refactored the local-emulator project route to streamline file
existence checks and default config creation.
- Enhanced user guidance in the dashboard for local Stack config file
handling.
- Updated tests to reflect changes in config file handling, ensuring
non-existent files are created with default settings.
- Improved Docker configurations for the local emulator, including new
environment variables and service dependencies.

These changes significantly enhance the local development experience and
emulator reliability.

commit 3910ed4bc40bbb37340c1c316c24c2826ba372bd
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Tue Mar 17 19:59:36 2026 -0700

    Remove unused stash-0.patch file to clean up the repository.

commit 74146d974458037a7a9590120a524629a1a6a162
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Tue Mar 17 19:58:46 2026 -0700

Enhance QEMU local emulator with app bundle support and runtime
configuration

- Introduced a new script to package the backend and dashboard assets
into a standalone app bundle for the QEMU emulator.
- Updated the build-image.sh script to create an ISO containing the app
bundle, ensuring the guest image includes the full runtime.
- Modified cloud-init user-data to handle the new app bundle and runtime
configuration, improving the setup process for local development.
- Enhanced the run-emulator.sh script to prepare and mount the runtime
configuration ISO, facilitating better environment management for the
emulator.
- Updated the user-data to include necessary environment variables for
the stack application, ensuring seamless integration during startup.

These changes significantly improve the local emulator's functionality
and ease of use for developers.

commit 9e865a1cf524398bc58f00e0836278775c4ae936
Author: mantrakp04 <mantrakp@gmail.com>
Date:   Tue Mar 17 16:50:45 2026 -0700

    Enhance local emulator setup with new services and configurations

- Added Docker support for a local emulator, integrating PostgreSQL,
Redis, Inbucket, Svix, ClickHouse, MinIO, and QStash.
- Introduced new scripts for managing the emulator lifecycle, including
build and run commands.
- Implemented cloud-init provisioning for automatic service setup on
first boot.
- Updated package.json with new commands for emulator management and
added dotenv-cli for environment variable management.
    - Added tests for OAuth authorization flow to return JSON responses.
    - Included configuration files for ClickHouse and user management.

This commit significantly improves the local development experience by
providing a comprehensive emulator environment.

<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

* **New Features**
* Introduced a local QEMU-based emulator for development with bundled
services (PostgreSQL, Redis, ClickHouse, MinIO, Inbucket, Svix, QStash).
* Added CLI commands to manage the emulator (start, stop, reset, status,
pull images).
  * Added emulator status dashboard to monitor service health.
  * Introduced new configuration system via `stack.config.ts`.

* **Tests**
  * Added configuration read/write tests for the emulator.
  * Added emulator CLI validation tests.

* **Documentation**
  * Added emulator setup and usage guide.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-04 00:33:52 +00:00
Konstantin Wohlwend
8c4951fc0a Only run setup wizard in E2E tests 2026-03-31 09:03:35 -07:00
Bilal Godil
a354590321 stack-cli npm publish 2026-03-13 09:26:12 -07:00
BilalG1
66adb4e50f
Local emulator base (#1233)
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Provision local-emulator projects from a local config file and return
emulator credentials via a new internal endpoint.
* Dashboard: "Open config file" flow to open local projects and refresh
owned projects.

* **Changes**
* Branch config can prefer/read/write local files for emulator projects.
* Environment config updates/resets are blocked for local-emulator
projects.
* Dashboard UI shows read-only notices and disables project creation in
emulator mode.
* Added DB mapping and a standard env flag to identify local-emulator
projects.

* **Tests**
  * New E2E tests covering provisioning and config restrictions.

* **Chores**
* Removed legacy emulator docs and compose; added CI workflow for
local-emulator E2E runs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Konstantin Wohlwend <n2d4xc@gmail.com>
2026-03-10 15:15:06 -07:00
Konsti Wohlwend
8052a2be62
"Require publishable client key" toggle (#1158)
Some checks failed
all-good: Did all the other checks pass? / all-good (push) Has been cancelled
Ensure Prisma migrations are in sync with the schema / check_prisma_migrations (22.x) (push) Has been cancelled
DB migration compat / Check if migrations changed (push) Has been cancelled
Docker Server Build and Push / Docker Build and Push Server (push) Has been cancelled
Docker Server Build and Run / docker (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (mock, 22.x) (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (prod, 22.x) (push) Has been cancelled
Runs E2E API Tests with custom port prefix / build (22.x) (push) Has been cancelled
Lint & build / lint_and_build (latest) (push) Has been cancelled
Dev Environment Test With Custom Base Port / restart-dev-and-test-with-custom-base-port (push) Has been cancelled
Dev Environment Test / restart-dev-and-test (push) Has been cancelled
Run setup tests with custom base port / setup-tests-with-custom-base-port (push) Has been cancelled
Run setup tests / setup-tests (push) Has been cancelled
TOC Generator / TOC Generator (push) Has been cancelled
DB migration compat / Back-compat — Current branch migrations with ${{ needs.check-migrations-changed.outputs.base_branch }} branch code (push) Has been cancelled
DB migration compat / Forward-compat — Current branch code with ${{ needs.check-migrations-changed.outputs.base_branch }} branch migrations (push) Has been cancelled
DB migration compat / No migration changes (skipped) (push) Has been cancelled
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **High Risk**
> Touches authentication and OAuth token/authorize flows and changes how
client requests are validated, so regressions could cause widespread
login/client-access failures. Also includes a data migration that alters
effective security posture for existing projects.
> 
> **Overview**
> Adds a **project-level toggle**
(`project.requirePublishableClientKey`) to control whether client
requests/OAuth flows must include a publishable client key, including a
DB migration that backfills existing projects to require it.
> 
> Backend auth now treats the publishable client key as *optional when
allowed*, introducing a public sentinel (`__stack_public_client__`) and
returning a new specific error
(`PUBLISHABLE_CLIENT_KEY_REQUIRED_FOR_PROJECT`) across smart request
auth + OAuth `authorize`/`callback`/`token` endpoints.
> 
> Dashboard and SDKs update key generation/display and request
construction to handle missing publishable keys, expose an advanced
toggle on the Project Keys page, and extend internal config overrides to
support a new `project` level; E2E/tests and schema fuzzing are expanded
accordingly, and CI adds a forward-compat migration check job when
back-compat fails.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5d06c08613. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Project-level config to require publishable client keys; migration
applied to existing projects.

* **Improvements**
* Auth flows now support optional publishable client keys with explicit
validation and a sentinel for keyless OAuth.
* Dashboard/UI and SDKs handle publishable keys as optional and
conditionally show/generate them.
  * Admin/client APIs extended to manage project-level overrides.

* **Bug Fixes**
  * Key validation behavior aligned with project config.

* **Tests**
* Expanded E2E and unit tests covering optional/required publishable-key
scenarios.

* **Documentation**
* Spec and knowledge docs updated to describe the sentinel and config
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-19 10:23:16 -08:00
BilalG1
907a98320a
Clickhouse sync fixing (#1198)
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->
2026-02-16 11:30:38 -08:00
BilalG1
d09a180dfe
clickhouse user sync (#1159)
Some checks failed
all-good: Did all the other checks pass? / all-good (push) Has been cancelled
Ensure Prisma migrations are in sync with the schema / check_prisma_migrations (22.x) (push) Has been cancelled
DB migrations are backwards-compatible / Check if migrations changed (push) Has been cancelled
Docker Server Build and Push / Docker Build and Push Server (push) Has been cancelled
Docker Server Build and Run / docker (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (mock, 22.x) (push) Has been cancelled
Runs E2E API Tests / E2E Tests (Node ${{ matrix.node-version }}, Freestyle ${{ matrix.freestyle-mode }}) (prod, 22.x) (push) Has been cancelled
Runs E2E API Tests with custom port prefix / build (22.x) (push) Has been cancelled
Lint & build / lint_and_build (latest) (push) Has been cancelled
Dev Environment Test With Custom Base Port / restart-dev-and-test-with-custom-base-port (push) Has been cancelled
Dev Environment Test / restart-dev-and-test (push) Has been cancelled
Run setup tests with custom base port / setup-tests-with-custom-base-port (push) Has been cancelled
Run setup tests / setup-tests (push) Has been cancelled
TOC Generator / TOC Generator (push) Has been cancelled
DB migrations are backwards-compatible / Test migrations with ${{ needs.check-migrations-changed.outputs.base_branch }} branch code (push) Has been cancelled
DB migrations are backwards-compatible / No migration changes (skipped) (push) Has been cancelled
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Real-time AI search with project-scoped analytics and dynamic query
execution; streaming AI responses replace the placeholder flow.
* External DB sync adds ClickHouse support: users sync, sync metadata
tracking, tenancy-aware status, and per-mapping throttling.
* AI assistant UI shows expandable tool-invocation results and streams
via the real AI pipeline.

* **Chores**
* Dashboard dependencies and workspace exclusions updated; development
OpenAI env var added; editor config flag toggled.

* **Tests**
* E2E coverage extended to validate ClickHouse user sync and analytics
queries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: aadesh18 <110230993+aadesh18@users.noreply.github.com>
Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
2026-02-12 16:52:20 -08:00
Konsti Wohlwend
adb14f1635
[Refactor] Improve CI Run Times by Reducing Test Flakiness and Speeding up Test Suite (#1166) 2026-02-10 19:43:10 -08:00
aadesh18
2055d98dea
External db sync (#1036)
<img width="1920" height="969" alt="Screenshot 2026-02-04 at 9 47 16 AM"
src="https://github.com/user-attachments/assets/d7d0cd04-0051-4fc4-b857-e6f87ee97a59"
/>

**This PR revolves around the following components**
1. Sequencer - sequences the updates in the internal db
2. Poller - polls for the latest updates to sync with the external db
3. Outgoing Request Handler - essentially a trigger that can make http
requests based on a change in the internal db
4. Sync Engine - syncs with the latest changes from the internal db to
the external db

**What has been done**
- Added a global sequence id for ProjectUser, ContactChannel and
DeletedRow.
- Added the deletedRow table to keep track of the rows that were deleted
across ProjectUser and ContactChannel.
- Added the OutgoingRequest table to keep track of the outgoing requests
- Added function for the sequencer to call to sequence updates
- Added a sequencer that sequences all the changes in the internal db
every 50 ms
- Added a poller that polls for the latest changes in the internal db
every 50 ms, and adds to a queue
- Added a Vercel cron that calls sequencer and poller every minute
- Added a queue that fulfills the outgoing requests by making http calls
(for external db sync, it calls the sync engine endpoint)
- Added a sync engine that uses the defined sql mapping query in the
user's schema to pull in the changes for the user, and sync them with
the external db
- Added tests to test out each functionality


**How to review this PR:**
1. Review the migrations (sequence id, deletedRow, triggers, backlog
sync) (all files created under the migrations folder)
2. Review sequencer
3. Review poller
4. Review the changes in schema
5. Review sync-engine (the function, and it's helper file)
6. Review the schema changes, and query mappings
7. Review the tests (basic, advanced and race, along with the helper
file)
8. Review the changes made in Dockerfile to support local testing using
the postgres docker

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Introduces a cron-driven external DB sync pipeline with global
sequencing, internal poller and webhook sync engine, new DB
tables/functions, config schema/mappings, and comprehensive e2e tests.
> 
> - **Database (Prisma/Migrations)**:
> - Add global sequence (`global_seq_id`) and
`sequenceId`/`shouldUpdateSequenceId` to `ProjectUser`,
`ContactChannel`, `DeletedRow` with partial indexes.
> - Create `DeletedRow` (capture deletes) and `OutgoingRequest` (queue)
tables; add unique/indexes.
> - Add triggers/functions: `log_deleted_row`,
`reset_sequence_id_on_update`, `backfill_null_sequence_ids`,
`enqueue_tenant_sync`.
> - **Backend/API**:
> - New internal routes: `GET
/api/latest/internal/external-db-sync/sequencer`, `GET /poller`, `POST
/sync-engine` (Upstash-verified) for sync orchestration.
> - Add cron wiring: `vercel.json` schedules and local
`scripts/run-cron-jobs.ts`; start in dev via `dev` script.
> - Tweak route handler (remove noisy logging) without behavior change.
> - **Sync Engine**:
> - Implement `src/lib/external-db-sync.ts` to read tenant mappings and
upsert to external Postgres (schema bootstrap, param checks,
sequencing).
> - Add default mappings `DEFAULT_DB_SYNC_MAPPINGS` and config schema
`dbSync.externalDatabases` in shared config.
> - **Testing/Infra**:
> - Add extensive e2e tests (basics, advanced, race conditions) for
sequencing, idempotency, deletes, pagination, multi-mapping, and
permissions.
> - Docker compose: add `external-db-test` Postgres for tests; e2e deps
for `pg` types.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3f2a8efcfb. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* External PostgreSQL sync: automatic, batched replication with
mappings, resume/idempotency, and on-demand enqueueing.

* **Admin UI**
* Real-time External DB Sync dashboard and status API showing
per-mapping backlog, sequencer/poller/sync-engine telemetry, and fusebox
controls.

* **Tests**
* Large e2e suite: basic, advanced, race, high-volume tests and test
utilities for external DB sync.

* **Chores**
* DB migrations, CI/workflow updates, background cron runner and
local/dev test support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
Co-authored-by: Bilal Godil <bg2002@gmail.com>
2026-02-05 12:04:31 -08:00
Konsti Wohlwend
7a35751f8e
Sign up rules (#1138)
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> **High Risk**
> Touches core sign-up/auth flows and user restriction semantics
(including new DB constraints) and introduces dynamic rule
evaluation/logging; misconfiguration or CEL/parser bugs could block
sign-ups or incorrectly restrict users.
> 
> **Overview**
> Introduces **CEL-based sign-up rules** (config-driven) that are
evaluated during password/OTP/OAuth sign-ups and anonymous upgrades;
matching rules can reject sign-ups or mark users as admin-restricted,
and triggers are logged for analytics.
> 
> Extends `ProjectUser` with `restrictedByAdmin` plus public/private
restriction details, updates restriction computation/filtering, and
exposes these fields via user CRUD (including validation + DB constraint
enforcing consistency when unrestricted).
> 
> Adds a new dashboard **Sign-up Rules** page with a visual condition
builder (CEL <-> visual tree), drag-reorder by priority, per-rule 48h
sparkline analytics via a new hidden internal endpoint, and adds
user-page UI to view/edit manual restrictions. Also refactors ClickHouse
client initialization to require env vars (removing
`isClickhouseConfigured` checks) and adjusts CI container startup wait
time.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
2141e689e8c1b72303b805e9234f996010d0880. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Sign-up Rules: visual rule builder, in-project CRUD with drag-reorder,
per-rule analytics, backend evaluation, and admin UI.
* Admin user restrictions: dashboard controls, banners/status,
public/private admin details surfaced in user views.

* **APIs & Schema**
* Config and user schemas extended; new SignUpRejected error and sign-up
rule types added.

* **Tests**
* Extensive unit and E2E coverage for rules, parser, evaluator,
analytics, and restricted-user flows.

* **Docs**
  * Editorial guidance added to AGENTS.md.

* **Chores**
* DB statement timeout, updated clean script, minor dependency
additions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-03 11:08:24 -08:00
BilalG1
7b5cf4f042
fix analytics queries (#1141)
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Added automatic retry with stricter result validation and clearer
error handling for query profiling to improve reliability.

* **Chores**
* CI workflows updated to use larger runner instances (upgraded runner
size across relevant jobs).

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-29 10:21:58 -08:00
Konstantin Wohlwend
4c22b37fdf --no-bail for verify-data-integrity script 2026-01-28 13:53:28 -08:00
BilalG1
484c3a6332
clickhouse setup (#1032) 2026-01-28 09:12:33 -08:00
Aman Ganapathy
c8694c7ff5
[Fix] [Feat] Update OAuth Sign-In and Get Token Functions to Work (#1130)
### Summary of Changes

Previously, on the Swift SDK, the `signInWithOAuth` function wasn't
working. In this PR, we fix it by having the `getOAuthUrl` function to
actually redirect correctly. Note that to do so, we updated the
`validRedirectUrl` check on the backend to accept app native redirects
(from our new trusted url scheme). Another thing to note is that we
added functionality to the `TokenStore` abstraction to conditionally
refresh the access token that the user is trying to fetch if it is
expired/close to expiring if possible. `getOAuthUrl` will attempt to get
a valid access token, and thus will rely on our algorithm documented in
`utilities.md`.

The specs serve as the source of truth.

We go further and implement Apple Native sign in. To do so, we have it
hit a new route on the backend and verify the `jwtToken` retrieved by
the sdk against an Apple-provided set of `jwks`. We use jose to do so,
in line with the rest of the codebase.

We take this opportunity to refactor the oauth provider route owing to
the amount of duplicated logic. Additionally, to enable the apple sign
in, users will have to update the Apple authentication method modal on
the dashboard and add accepted bundle ids. These are identifiers for
projects, and we will check the `JWT` on the backend to make sure the
audience is set to an accepted bundleId.

We also update the Apple modal to be more informative.

### Using the new Features

To use the Apple native sign in, users will have to 1) sign up with an
apple developer account, 2) set up their bundleids for their projects by
connecting them to the apple developer account, 3) update the Stack-Auth
Authentication Methods dashboard apple modal with the relevant fields.
Then, trying to sign in with apple with our Swift SDK will use the apple
native sign in.

### UI Changes

Renamed the fields in the apple modal. Added a new field for bundle ids.
See below.


https://github.com/user-attachments/assets/0e760c0e-3198-4818-ac7f-4900d7a125bb



Co-authored-by: Konstantin Wohlwend <n2d4xc@gmail.com>
2026-01-28 02:17:27 +00:00
Konstantin Wohlwend
3bb31badcd Fix env_to_branch migration 2026-01-23 17:17:43 -08:00
Konstantin Wohlwend
70ead41121 Clean up branch config overrides after completion 2026-01-23 16:47:19 -08:00
Konstantin Wohlwend
15b5336258 More stringent db migration backwards compatibility check 2026-01-23 12:41:59 -08:00
Konstantin Wohlwend
f97ad9ac3f Remove Hello World workflow 2026-01-22 10:25:39 -08:00
Konstantin Wohlwend
20ac0bff8a Hello World workflow 2026-01-22 10:22:44 -08:00
Konstantin Wohlwend
2b98fd2617 Don't run Freestyle prod tests in parallel 2026-01-21 11:59:15 -08:00
Konstantin Wohlwend
7c85d36c56 npm publish action 2026-01-12 15:41:41 -08:00
Konstantin Wohlwend
20f435fb6d Delete Claude Code review 2026-01-11 17:39:59 -08:00
Konstantin Wohlwend
a809ac16d6 GitHub action for checking migration backwards-compatibility 2026-01-10 15:09:30 -08:00
Konsti Wohlwend
8f74949a7f
Speed up tests (#1063) 2025-12-28 11:25:04 -08:00
Konstantin Wohlwend
671f6b2d1f Fix Prisma migrate 2025-12-27 20:43:54 +01:00
Konsti Wohlwend
b4ae80874e
Upgrade Prisma to v7 (#1064) 2025-12-26 08:13:34 -08:00
Konstantin Wohlwend
a17c841d77 Fix tests 2025-12-17 10:59:23 -08:00
Konstantin Wohlwend
5caf148b7d Less flakey tests 2025-12-15 13:30:35 -08:00
Konstantin Wohlwend
c9d0c1c3e1 Fix tests 2025-12-15 12:04:37 -08:00
Konstantin Wohlwend
de9cfb33a7 Environment variables for disabling email queue 2025-12-15 10:48:00 -08:00
Konstantin Wohlwend
58a4aa91df Fix syntax 2025-12-12 18:48:47 -08:00
Konstantin Wohlwend
77d7117333 More fixes 2025-12-12 18:47:48 -08:00
Konstantin Wohlwend
27b42ceb01 Fix custom base port tests 2025-12-12 18:44:14 -08:00
Konstantin Wohlwend
64c02cbd98 Run email queue in E2E tests 2025-12-12 18:38:29 -08:00
Konstantin Wohlwend
1c74c497c6 Freestyle E2E tests on GH Actions 2025-12-12 16:10:47 -08:00
Konsti Wohlwend
e7e792d462
Email outbox backend (#1030) 2025-12-12 10:26:38 -08:00
Konstantin Wohlwend
98b573877c Replace npx with pnpx across the board
Some checks failed
all-good: Did all the other checks pass? / all-good (push) Has been cancelled
Ensure Prisma migrations are in sync with the schema / check_prisma_migrations (22.x) (push) Has been cancelled
Docker Server Build and Push / Docker Build and Push Server (push) Has been cancelled
Docker Server Test / docker (push) Has been cancelled
Runs E2E API Tests / build (22.x) (push) Has been cancelled
Runs E2E API Tests with custom port prefix / build (22.x) (push) Has been cancelled
Runs E2E API Tests with external source of truth / build (22.x) (push) Has been cancelled
Lint & build / lint_and_build (latest) (push) Has been cancelled
Dev Environment Test With Custom Base Port / restart-dev-and-test-with-custom-base-port (push) Has been cancelled
Dev Environment Test / restart-dev-and-test (push) Has been cancelled
Run setup tests / setup-tests (push) Has been cancelled
TOC Generator / TOC Generator (push) Has been cancelled
2025-11-27 12:28:00 -08:00
Konstantin Wohlwend
5421e6dd38 Upgrade pnpm in GH workflows 2025-11-24 12:02:09 -08:00
Konstantin Wohlwend
8580a1ea14 Upgrade GitHub workflow scripts 2025-11-24 11:21:43 -08:00
BilalG1
b5b311554b
Metrics Endpoint Speed (#966)
<img width="567" height="249" alt="Screenshot 2025-10-20 at 11 23 10 AM"
src="https://github.com/user-attachments/assets/340df844-f619-489f-8d41-cc26bc165018"
/>
<img width="595" height="255" alt="Screenshot 2025-10-20 at 11 24 00 AM"
src="https://github.com/user-attachments/assets/9321bda1-e6f0-4f53-8c6b-e29d0fc16038"
/>

<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->

<!-- RECURSEML_SUMMARY:START -->
## High-level PR Summary
This PR optimizes the performance of user list and metrics endpoints by
refactoring SQL queries to use more efficient patterns. The changes
include rewriting queries to use `LATERAL` joins and CTEs with proper
filtering, extracting common user mapping logic into reusable functions,
and adding performance tests with SQL scripts to generate realistic test
data (10,000 mock users and activity events across 100 countries).

⏱️ Estimated Review Time: 30-90 minutes

<details>
<summary>💡 Review Order Suggestion</summary>

| Order | File Path |
|-------|-----------|
| 1 | `apps/e2e/tests/backend/performance/mock-users.sql` |
| 2 | `apps/e2e/tests/backend/performance/mock-metric-events.sql` |
| 3 | `apps/e2e/tests/backend/performance/users-list.test.ts` |
| 4 | `apps/backend/src/app/api/latest/users/crud.tsx` |
| 5 | `apps/backend/src/app/api/latest/internal/metrics/route.tsx` |
</details>



[![Need help? Join our
Discord](https://img.shields.io/badge/Need%20help%3F%20Join%20our%20Discord-5865F2?style=plastic&logo=discord&logoColor=white)](https://discord.gg/n3SsVDAW6U)


[![Analyze latest
changes](f22b2c44a1/?repo_owner=stack-auth&repo_name=stack-auth&pr_number=966)
<!-- RECURSEML_SUMMARY:END -->
<!-- ELLIPSIS_HIDDEN -->


----

> [!IMPORTANT]
> Optimize metrics and user list endpoints with SQL refactoring,
caching, and performance tests, adding a `CacheEntry` model and mock
data scripts.
> 
>   - **Performance Optimization**:
> - Refactor SQL queries in `route.tsx` to use `LATERAL` joins and CTEs
for efficient data retrieval.
> - Implement caching in `route.tsx` using `getOrSetCacheValue()` to
reduce database load.
>   - **Database Changes**:
> - Add `CacheEntry` model to `schema.prisma` and create corresponding
table and index in `migration.sql`.
> - Remove auto-migration metadata step from
`check-prisma-migrations.yaml`.
>   - **Testing**:
> - Add performance tests in `metrics.test.ts` to benchmark metrics and
user endpoints.
> - Create mock data scripts `mock-users.sql` and
`mock-metric-events.sql` for testing with 10,000 users and events across
100 countries.
>   - **Miscellaneous**:
> - Update `db-migrations.ts` to include new migration file generation
logic.
>     - Add `cache.tsx` for caching logic implementation.
> 
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=stack-auth%2Fstack-auth&utm_source=github&utm_medium=referral)<sup>
for 4d9be71063. You can
[customize](https://app.ellipsis.dev/stack-auth/settings/summaries) this
summary. It will automatically update as commits are pushed.</sup>

----


<!-- ELLIPSIS_HIDDEN -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Metrics now use a cache layer with per-entry TTL and tenancy-aware
loaders.

* **Bug Fixes**
* Improved accuracy of daily active and related metrics with
tenancy-aware counting and more robust last-active computation.

* **Performance**
* Faster metrics responses via batched reads and cache-backed endpoints.

* **Tests**
* Added end-to-end performance benchmarks and SQL seed scripts for
metrics/user load testing.

* **Chores**
* DB migration added support for cached entries; CI migration check flow
adjusted; migration tooling improved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
2025-11-05 16:24:04 -08:00
Konstantin Wohlwend
6684db64f3 Enhance lint-and-build workflow to display uncommitted changes and diffs before exiting 2025-10-28 00:32:52 -07:00
BilalG1
0bcec0f08f
mirrored qstash image (#968)
<!--

Make sure you've read the CONTRIBUTING.md guidelines:
https://github.com/stack-auth/stack-auth/blob/dev/CONTRIBUTING.md

-->

---------

Co-authored-by: Konsti Wohlwend <n2d4xc@gmail.com>
2025-10-21 18:28:22 -07:00