Commit Graph

1602 Commits

Author SHA1 Message Date
Alex Vandiver
e104f27533 restore-backup: Handle PORT being an int, not a str.
This changed in f1ec8163ef.

(cherry picked from commit c9fccaa975)
2026-03-30 14:34:49 -04:00
Sutou Kouhei
1124dbf3a2 setup-upgrade-postgresql: Add missing --file flag to execute SQL in a file
Fix GH-36350

If we want to execute SQL in a file, we need to use `psql --file PATH`
not `psql PATH`.

(cherry picked from commit d690e589b2)
2026-03-30 14:34:49 -04:00
Anders Kaseorg
1ea4913591 install-uv: Upgrade uv from 0.9.12 to 0.9.21.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6096907263)
2026-02-06 15:21:48 -05:00
Anders Kaseorg
25c2a5a7a8 install-uv: Upgrade uv from 0.9.5 to 0.9.12.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6916c33186)
2026-02-06 15:21:48 -05:00
Alex Vandiver
a21b53e110 migrations: Move schema creation and search_path into pre_migrate hook.
(cherry picked from commit 13243be3e8)
2026-01-20 15:50:53 -08:00
Alex Vandiver
b195287b55 migrations: Fix default search_path.
PostgreSQL's `search_path` sets the order in which schemas are
searched, and the first one is the one in which unqualified objects
are created[^1].

Lower in the file, we create an explicit `zulip` schema -- so we
should not use the database name in the `search_path`.  Switch to the
correct constant `zulip`.

[^1]: https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH

(cherry picked from commit 3ee7690b0b)
2026-01-20 15:50:53 -08:00
Alex Vandiver
4ad728c786 migrations: Degrade gracefully with missing remote dictionaries.
Rather than require a configuration option, detect the `tsearch_data`
error, in conjunction with a remote PostgreSQL host, and warn and
continue.  It is pretty much impossible to arrive at that combination
and then _not_ want to try again with the flag, so we should handle
that degradation gracefully, instead of requiring an explicit
install and configuration flag.

(cherry picked from commit 1f3aed14a5)
2025-12-05 13:04:51 -05:00
Alex Vandiver
9513049815 setup-certbot: Puppet guarantees the certbot package.
It has been installed on all hosts since 01e8f752a8.

(cherry picked from commit f22592151a)
2025-12-05 13:04:51 -05:00
Alex Vandiver
07bc2f2f5e setup-certbot: Run hooks, instead of "service nginx reload".
Some checks failed
Code scanning / CodeQL (push) Has been cancelled
Zulip production suite / Ubuntu 22.04 production build (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:bookworm, true, false, Debian 12 (Python 3.11, backend + documentation), bookworm) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:jammy, false, true, Ubuntu 22.04 (Python 3.10, backend + frontend), jammy) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:noble, false, false, Ubuntu 24.04 (Python 3.12, backend), noble) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:trixie, false, false, Debian 13 (Python 3.13, backend), trixie) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm, --test-custom-db, Debian 12 production install with custom db name and user, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy, , Ubuntu 22.04 production install and PostgreSQL upgrade with pgroonga, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble, , Ubuntu 24.04 production install, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:trixie, , Debian 13 production install, trixie) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-7.0, 7.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-8.0, 8.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy-6.0, 6.0 Version Upgrade, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-10.0, 10.0 Version Upgrade, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-9.0, 9.0 Version Upgrade, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:trixie-11.0, 11.0 Version Upgrade, trixie) (push) Has been cancelled
`certbot certonly`, when obtaining a new certificate, does not
run deploy hooks that were installed into
`/etc/letsencrypt/renewal-hooks` (certbot/certbot#9978).  Because of
this, we added an explicit `service nginx reload`.

This turns out to be awkward for a few reasons -- firstly, this does
not handle the restarting of our SMTP server, which also needs to pick
up the new certificates.  It requires that Docker installs explicitly
restart nginx their own way.  And it also means that if certbot
already had a certificate, we reload nginx twice.

We would ideally just have certbot just run all of the deploy hooks,
as usual, no matter if the certificate is new or a renewal; however,
it has no flag to do so.

We move to approximating running the deploy hooks, if the certificate
is new.  This makes it easier to have identical behaviour between
initial install and later renewals.

This requires moving the symlinking step itself to a deploy hook, so
it can run before nginx and the email server are restarted.  Our one
consumer of the `--skip-symlink` flag, Docker, can work around its
removal.  The symlink step is a no-op if `$ZULIP_DOMAIN` is unset,
i.e. during automated renewals.  This ensures that system that
installed certbot, but has since changed to manual certificate
management, does not have those certificates overridden next time
certbot renews itself.

(cherry picked from commit 0c953f9870)
2025-11-26 14:18:48 -05:00
Alex Vandiver
2ccb57d368 setup-certbot: More reasonable non-interactive behaviour.
The use of `--force-interactive` was because it "suppresses a
warning", which current certbot shows no sign of displaying without
the flag.

We remove `--force-interactive`, and instead add
`--keep-until-expiring` if we are non-interactive, to allow
`setup-certbot` to default to not unnecessarily renewing certificates.
When run interactively, it may still prompt if a not-near-expiry
certificate is to be force-renewed.

(cherry picked from commit 52b1bd69a9)
2025-11-26 14:18:48 -05:00
Alex Vandiver
94728020fc letsencrypt: Remove danging fixes from Zulip Server 5.0.
It is no longer possible to upgrade directly from 5.0, so these steps
are no longer necessary.

(cherry picked from commit 150b81f3ad)
2025-11-26 14:18:48 -05:00
Alex Vandiver
7dc20c716a checks: Move checkconfig to standard check framework.
(cherry picked from commit 6c9705a6bb)
2025-11-20 15:38:06 -08:00
Anders Kaseorg
64461c0033 install-uv: Upgrade uv from 0.8.22 to 0.9.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 69f2e95e49)
2025-11-20 15:38:06 -08:00
Alex Vandiver
f82cde7f4d restart-server: All realm Tornado ports need a restart when they change.
(cherry picked from commit a8f0cb2cf9)
2025-11-20 15:38:06 -08:00
Alex Vandiver
8242941b12 restart-server: --tornado-reshard implies --skip-client-reloads.
The flags are marked mutually exclusive, so don't pass both; have the
former imply the latter.

(cherry picked from commit 21f08265de)
2025-11-20 15:38:06 -08:00
Alex Vandiver
9126e38fcd sharding: Swap new config into place during restart-server.
This allows restart-server, before moving the new config into place,
to perform a diff and only restart the affected Tornado ports.

(cherry picked from commit 01664a1a10)
2025-11-20 15:38:06 -08:00
Alex Vandiver
6099ec8867 sharding: Use restart-server after setting up new sharding config.
restart-server has grown more complex since
efresh-sharding-and-restart was written -- use that complexity.  For
instance, workers are stopped one-at-a-time, rather than requiring
that they all stop sequentially before being started again.

It also means that Django restarts will be rolling, which trades off
500s during login for realms which are moving, in exchange for leaving
other realms unaffected.

(cherry picked from commit e18b0fcd16)
2025-11-20 15:38:06 -08:00
Alex Vandiver
f1fb6b63c6 sharding: Move chown/chmod into where the files are written.
(cherry picked from commit 2bd4c07fdb)
2025-11-20 15:38:06 -08:00
Alex Vandiver
4ae1b9e364 sharding: Make nginx shard map owned by zulip:zulip.
This will make it easier to manipulate in restart-server.

(cherry picked from commit 6249c2f970)
2025-11-20 15:38:06 -08:00
Anders Kaseorg
f72ba77718 install-uv: Upgrade uv from 0.8.15 to 0.8.22.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0b400a66cc)
2025-10-22 16:08:09 -04:00
Anders Kaseorg
582bc7d303 install-uv: Upgrade uv from 0.8.10 to 0.8.15.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 270c50651c)
2025-10-22 16:08:09 -04:00
Anders Kaseorg
5963e5c6a5 install-uv: Upgrade uv from 0.7.21 to 0.8.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 26dc355bf5)
2025-09-10 14:50:32 -07:00
Alex Vandiver
9a2194ef6c run-hooks: Handle hooks after OS upgrade on fresh install.
(cherry picked from commit fc962b8b0e)
2025-09-03 17:31:37 -07:00
Alex Vandiver
6dd8e003da reload-clients: Ensure that Smokescreen does not interfere with reloads.
When run from cron, reload-server (and thus reload-clients) picks up
the `HTTP_proxy` environment variable, which redirects HTTP requests
through Smokescreen -- which prevents localhost requests.  This
results in clients never getting sent reload events.

Explicitly unset proxies when talking to localhost in reload-clients.

(cherry picked from commit bef864251b)
2025-08-28 16:17:08 -07:00
Anders Kaseorg
ff15d746c3 install: Support Debian 13.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2025-08-13 14:11:05 -07:00
Anders Kaseorg
fd1543a67c apt-repos: Use signed-by instead of globally trusted keys.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2025-08-13 14:11:05 -07:00
Anders Kaseorg
60fe9d438f build-pgroonga: Use msgpack-c on Debian 13.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2025-08-13 14:11:05 -07:00
Anders Kaseorg
2e97637d8e build-pgroonga: Upgrade PGroonga from 3.1.8 to 4.0.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-08-13 14:11:05 -07:00
Alex Vandiver
e132af28fc start-server: Warn on deploy mismatch if it is a noop. 2025-08-12 10:50:30 -07:00
Alex Vandiver
f6eba9c396 log-search: Add --extra filter, for [dm] and similar filters. 2025-08-12 10:25:44 -07:00
Shubham Padia
3e60b16ac1 starlight_help: Move help-beta over to starlight_help.
We are starting the cutover process and starlight_help is the directory
we have agreed on to place our new help center project. We do not want
to use `starlight_help` as the URL for the project, but this commit
changes the url from `help-beta` to `starlight_help` temporarily since
we can only change URL once we get rid of the current help center
project. That will be done in a future commit.
2025-08-08 11:59:29 -07:00
Alex Vandiver
fd390b9eb1 settings: Enable Altcha in development if on localhost.
Some checks failed
Code scanning / CodeQL (push) Has been cancelled
Zulip production suite / Ubuntu 22.04 production build (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:bookworm, true, false, Debian 12 (Python 3.11, backend + documentation), bookworm) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:jammy, false, true, Ubuntu 22.04 (Python 3.10, backend + frontend), jammy) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:noble, false, false, Ubuntu 24.04 (Python 3.12, backend), noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm, --test-custom-db, Debian 12 production install with custom db name and user, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy, , Ubuntu 22.04 production install and PostgreSQL upgrade with pgroonga, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble, , Ubuntu 24.04 production install, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-7.0, 7.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-8.0, 8.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy-6.0, 6.0 Version Upgrade, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-10.0, 10.0 Version Upgrade, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-9.0, 9.0 Version Upgrade, noble) (push) Has been cancelled
We only do this on localhost, because users of remote development
servers will run afoul of "Secure context is required"[^1].

[^1]: https://altcha.org/docs/troubleshooting/#secure-context
2025-07-25 22:29:51 -07:00
Anders Kaseorg
c911dcf279 install-node: Upgrade Node.js from 22.17.0 to 22.17.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-07-22 14:19:56 -07:00
Alex Vandiver
43034a79b6 run_hooks: Make post-deploy hooks be best-effort. 2025-07-16 18:00:47 -07:00
Alex Vandiver
e9a2ee56c3 log-search: Add --stats flag. 2025-07-16 09:22:02 -07:00
Alex Vandiver
2616b7d030 log-search: Add a missing new filter flag to -L. 2025-07-16 09:22:02 -07:00
Alex Vandiver
6d80d4c77f restart-server: Skip weekly restart-server if a deploy is in process.
This is unlikely, but cron running `restart-server` at the random
wrong time mid-deploy could be bad.
2025-07-15 09:07:36 -07:00
Anders Kaseorg
996eb72e2a install-uv: Upgrade uv from 0.7.15 to 0.7.21.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-07-15 07:32:44 -07:00
Prakhar Pratyush
8b3cef554b settings: Add push_registration_encryption_keys map.
The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.

The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.

- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
2025-07-06 21:11:26 -07:00
Anders Kaseorg
7959a1853c install-node: Upgrade Node.js from 22.16.0 to 22.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-25 16:53:09 -07:00
Anders Kaseorg
9f8f6e60d9 install-uv: Upgrade uv from 0.7.11 to 0.7.15.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-25 16:52:39 -07:00
Anders Kaseorg
cdbe2d157f flush_memcached: Respect DJANGO_SETTINGS_MODULE.
We don’t need to flush anything for zproject.test_settings, which
disables memcached.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-25 11:49:02 -07:00
Alex Vandiver
b924169d17 setup-apt-repo: Add libheif PPA, and debian bookworm backport.
libheif 1.18 is required to be able to parse images generated by iOS
18; none of Zulip's supported distributions package libheif 1.18, so
we pull new version of the package from PPA (Ubuntu) or backports
(Debian).
2025-06-25 11:39:18 -07:00
Alex Vandiver
a0683927ef check_rabbitmq_queue: Relax paging thresholds for email_senders. 2025-06-18 12:29:57 -07:00
Anders Kaseorg
acd6c51b6f manage: Delete custom PYTHONSTARTUP.
In Django 5.2, manage.py shell automatically imports models.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-12 09:28:00 -07:00
Anders Kaseorg
927ea011d3 upgrade-postgresql: Get PostgreSQL version without manage.py shell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-11 16:25:25 -07:00
Alex Vandiver
6f1950ac0e restart-server: Send client reload events in the background.
For deploys with --skip-puppet, this makes the output visible much
more promptly.
2025-06-11 10:16:46 -07:00
Anders Kaseorg
56470bba8d install-uv: Upgrade uv from 0.7.2 to 0.7.11.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-06 11:35:32 -07:00
Tim Abbott
0ec07fe4c8 queue: Allow sharding user_activity worker.
This follows the existing patterns for the sharded mobile
notifications worker.
2025-06-06 10:33:20 -07:00
Anders Kaseorg
f6be163bcc install-node: Upgrade Node.js from 22.15.0 to 22.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-04 16:24:47 -07:00