mirror of
https://github.com/stack-auth/stack.git
synced 2026-06-13 21:01:21 +08:00
1d00ed2c64
### Context We encountered an out of memory error when running verify-data-integrity against the prod database. This was the error: `FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory`. This was one of the things preventing verify-data-integrity from running successfully in prod. ### Summary of Changes Local stress testing with constrained heap and memory telemetry revealed that the rise in used heap memory was directly proportional to the number of api calls. Investigation revealed that the `currentOutputData` array was growing with each api call and was kept in memory. Since it was still being appended to, it was actively kept in the heap. We refactor the script to no longer use it, and for the two flags `--save-output` and `--verify-output` that used it before, we refactor them to not need to. `--save-output` now streams responses to disk as JSONL and `--verify-output` now compares each response immediately and discards it. We also note a potential source of a future memory leak in the `allUsers` array that is populated in memory for each project. We refactor to paginate instead. Note that this didn't cause a memory leak on local, this is a preventive measure. ### Out of Scope fetching all transactions in the payments section of the script is another potential cause for concern, but since the payments section of the script will be refactored soon, we defer that discussion. |
||
|---|---|---|
| .. | ||
| api.ts | ||
| index.ts | ||
| payments-verifier.ts | ||
| recurse.ts | ||
| stripe-payout-integrity.ts | ||