Commit Graph

428 Commits

Author SHA1 Message Date
Sutou Kouhei
d690e589b2 setup-upgrade-postgresql: Add missing --file flag to execute SQL in a file
Fix GH-36350

If we want to execute SQL in a file, we need to use `psql --file PATH`
not `psql PATH`.
2026-02-17 10:57:33 -05:00
Alex Vandiver
13243be3e8 migrations: Move schema creation and search_path into pre_migrate hook. 2025-12-16 13:03:27 -08:00
Alex Vandiver
3ee7690b0b migrations: Fix default search_path.
PostgreSQL's `search_path` sets the order in which schemas are
searched, and the first one is the one in which unqualified objects
are created[^1].

Lower in the file, we create an explicit `zulip` schema -- so we
should not use the database name in the `search_path`.  Switch to the
correct constant `zulip`.

[^1]: https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH
2025-12-16 13:03:27 -08:00
Alex Vandiver
359e5c2b78 install: Support PostgreSQL 18. 2025-12-10 21:37:07 -08:00
Alex Vandiver
f22592151a setup-certbot: Puppet guarantees the certbot package.
It has been installed on all hosts since 01e8f752a8.
2025-11-26 11:33:27 -08:00
Alex Vandiver
0c953f9870 setup-certbot: Run hooks, instead of "service nginx reload".
`certbot certonly`, when obtaining a new certificate, does not
run deploy hooks that were installed into
`/etc/letsencrypt/renewal-hooks` (certbot/certbot#9978).  Because of
this, we added an explicit `service nginx reload`.

This turns out to be awkward for a few reasons -- firstly, this does
not handle the restarting of our SMTP server, which also needs to pick
up the new certificates.  It requires that Docker installs explicitly
restart nginx their own way.  And it also means that if certbot
already had a certificate, we reload nginx twice.

We would ideally just have certbot just run all of the deploy hooks,
as usual, no matter if the certificate is new or a renewal; however,
it has no flag to do so.

We move to approximating running the deploy hooks, if the certificate
is new.  This makes it easier to have identical behaviour between
initial install and later renewals.

This requires moving the symlinking step itself to a deploy hook, so
it can run before nginx and the email server are restarted.  Our one
consumer of the `--skip-symlink` flag, Docker, can work around its
removal.  The symlink step is a no-op if `$ZULIP_DOMAIN` is unset,
i.e. during automated renewals.  This ensures that system that
installed certbot, but has since changed to manual certificate
management, does not have those certificates overridden next time
certbot renews itself.
2025-11-26 09:10:40 -08:00
Alex Vandiver
52b1bd69a9 setup-certbot: More reasonable non-interactive behaviour.
The use of `--force-interactive` was because it "suppresses a
warning", which current certbot shows no sign of displaying without
the flag.

We remove `--force-interactive`, and instead add
`--keep-until-expiring` if we are non-interactive, to allow
`setup-certbot` to default to not unnecessarily renewing certificates.
When run interactively, it may still prompt if a not-near-expiry
certificate is to be force-renewed.
2025-11-26 09:10:40 -08:00
Alex Vandiver
6c9705a6bb checks: Move checkconfig to standard check framework. 2025-11-19 11:50:18 -08:00
Anders Kaseorg
ff15d746c3 install: Support Debian 13.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2025-08-13 14:11:05 -07:00
Anders Kaseorg
fd1543a67c apt-repos: Use signed-by instead of globally trusted keys.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2025-08-13 14:11:05 -07:00
Alex Vandiver
fd390b9eb1 settings: Enable Altcha in development if on localhost.
Some checks failed
Code scanning / CodeQL (push) Has been cancelled
Zulip production suite / Ubuntu 22.04 production build (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:bookworm, true, false, Debian 12 (Python 3.11, backend + documentation), bookworm) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:jammy, false, true, Ubuntu 22.04 (Python 3.10, backend + frontend), jammy) (push) Has been cancelled
Zulip CI / ${{ matrix.name }} (zulip/ci:noble, false, false, Ubuntu 24.04 (Python 3.12, backend), noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm, --test-custom-db, Debian 12 production install with custom db name and user, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy, , Ubuntu 22.04 production install and PostgreSQL upgrade with pgroonga, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble, , Ubuntu 24.04 production install, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-7.0, 7.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-8.0, 8.0 Version Upgrade, bookworm) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy-6.0, 6.0 Version Upgrade, jammy) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-10.0, 10.0 Version Upgrade, noble) (push) Has been cancelled
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-9.0, 9.0 Version Upgrade, noble) (push) Has been cancelled
We only do this on localhost, because users of remote development
servers will run afoul of "Secure context is required"[^1].

[^1]: https://altcha.org/docs/troubleshooting/#secure-context
2025-07-25 22:29:51 -07:00
Prakhar Pratyush
8b3cef554b settings: Add push_registration_encryption_keys map.
The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.

The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.

- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
2025-07-06 21:11:26 -07:00
Anders Kaseorg
cdbe2d157f flush_memcached: Respect DJANGO_SETTINGS_MODULE.
We don’t need to flush anything for zproject.test_settings, which
disables memcached.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-25 11:49:02 -07:00
Alex Vandiver
b924169d17 setup-apt-repo: Add libheif PPA, and debian bookworm backport.
libheif 1.18 is required to be able to parse images generated by iOS
18; none of Zulip's supported distributions package libheif 1.18, so
we pull new version of the package from PPA (Ubuntu) or backports
(Debian).
2025-06-25 11:39:18 -07:00
Anders Kaseorg
927ea011d3 upgrade-postgresql: Get PostgreSQL version without manage.py shell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-06-11 16:25:25 -07:00
Alex Vandiver
0442bb6f0e upgrade-postgresql: Slightly better error-proof post-upgrade scripts. 2025-05-16 11:33:20 -07:00
Alex Vandiver
3ab6be650b upgrade-postgresql: Explicitly ask to not start the new cluster.
Recent versions of postgresql-common's `pg_upgradecluster`, starting
with version 254, (i.e. on Ubuntu 24.04, but not 22.04) will not just
_suggest_ running the analyze, but will do so automatically.  While
somewhat helpful, it always does so with `--analyze-in-stages`, which
as noted in f77bbd3323, is actually the incorrect choice for us.
Passing `--no-start` ensures that `pg_upgradecluster` consistently
does not do any analyzing, allowing us to start the cluster manually
and then perform the analyze correctly ourselves.
2025-05-16 11:33:20 -07:00
Alex Vandiver
e13f82f048 upgrade-postgresql: Use tags to partially-apply configuration.
This uses the same technique used in 840884ec89, to only apply select
parts of the Puppet configuration.  This is more correct, and simpler,
than attempting to chop out some base puppet roles, and hack around
the `purge => true` supervisor.d configuration.
2025-05-16 11:33:20 -07:00
Alex Vandiver
2dc5c6c50e upgrade-postgresql: Only touch pgroonga_setup.sql.applied if required.
Since c8ec3dfcf6, the file must contain the version that was
configured, or we run `ALTER EXTENSION pgroonga UPDATE`; if the file
is missing, and pgroonga was previously installed, it run `CREATE
EXTENSION pgroonga` which will be an error.  If the file is present
but pgroonga was not configured, a later attempt to enable pgroonga
will incorrectly run `ALTER EXTENSION pgroonga UPDATE` instead of
`CREATE EXTENSION pgroonga`.

If the file existed on the previous version, touch it in the new
PostgreSQL version.  This will ensure that puppet will *always* run
the pgroonga update, which may be necessary in case the pgroonga
version also changed.  At worst, if the pgroonga version has not
changed, this will be a safe no-op.
2025-05-16 11:33:20 -07:00
Anders Kaseorg
8e9de0b053 configure-rabbitmq: Restore startup retry loop.
‘rabbitmqctl await_startup’ does not retry to wait for the Erlang
runtime to start, only to wait for the RabbitMQ application to start
once Erlang is running.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-05-15 16:59:27 -07:00
Anders Kaseorg
818742c62b install: Support PostgreSQL 17.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-04-07 16:42:19 -07:00
Alex Vandiver
ba9569a6fe sha256-tarball-to: Support zipfiles. 2025-03-27 21:56:54 -07:00
Alex Vandiver
e2df4f52ef kandra: Update Teleport version. 2025-02-21 10:16:33 -08:00
Anders Kaseorg
8dd0d7f48d reindex-textual-data: Remove PostgreSQL ≥ 11 check.
We removed PostgreSQL 10 support long ago in 6.0-beta1~88.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-11-01 15:33:31 -07:00
Anders Kaseorg
8b147d92a8 apt-repos: Use PGroonga binaries on Ubuntu 24.04.
These did not exist when we first added Ubuntu 24.04 support; now they
do.  Fixes #31261.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-08-29 16:04:44 -07:00
Alex Vandiver
a5a898ba21 install-aws-server: Switch to 24.04. 2024-08-20 10:04:37 -07:00
Alex Vandiver
1e5c0dc259 kandra: Bump Teleport version to 16.
Some checks are pending
Code scanning / CodeQL (push) Waiting to run
Zulip production suite / Ubuntu 22.04 production build (push) Waiting to run
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm, --test-custom-db, Debian 12 production install with custom db name and user, bookworm) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy, , Ubuntu 22.04 production install and PostgreSQL upgrade with pgroonga, jammy) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble, , Ubuntu 24.04 production install, noble) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-7.0, 7.0 Version Upgrade, bookworm) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:bookworm-8.0, 8.0 Version Upgrade, bookworm) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:jammy-6.0, 6.0 Version Upgrade, jammy) (push) Blocked by required conditions
Zulip production suite / ${{ matrix.name }} (zulip/ci:noble-9.0, 9.0 Version Upgrade, noble) (push) Blocked by required conditions
Zulip CI / ${{ matrix.name }} (zulip/ci:bookworm, true, false, Debian 12 (Python 3.11, backend + documentation), bookworm) (push) Waiting to run
Zulip CI / ${{ matrix.name }} (zulip/ci:jammy, false, true, Ubuntu 22.04 (Python 3.10, backend + frontend), jammy) (push) Waiting to run
Zulip CI / ${{ matrix.name }} (zulip/ci:noble, false, false, Ubuntu 24.04 (Python 3.12, backend), noble) (push) Waiting to run
2024-08-16 08:51:15 -07:00
Anders Kaseorg
c8eaceff21 flush-memcached: Explicitly close memcached connection.
Fixes warnings like “ResourceWarning: unclosed <socket.socket fd=3,
family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
laddr=('127.0.0.1', 33332), raddr=('127.0.0.1', 11211)>” with warnings
enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:50:09 -07:00
Anders Kaseorg
531b34cb4c ruff: Fix UP007 Use X | Y for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg
e08a24e47f ruff: Fix UP006 Use list instead of List for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Lauryn Menard
80b86c7b16 documentation: Fix "you you" typos in docs/comments/logs. 2024-07-02 11:57:44 -07:00
Sayam Samal
a7772f86a2 docs: Update CZO links from stream to channel in docs and comments.
Updates all the https://chat.zulip.org/#narrow/stream/ links in the
docs and comments to use the new /channel/ path. All these links are
for documentation/reference purposes only and thus, can be bulk-updated.

This commit is a part of the effort to rename stream to channel.
2024-06-11 10:44:31 -07:00
Alex Vandiver
5814583694 rabbitmq: Support non-/ vhosts.
Fixes: #30255.
2024-05-31 15:41:19 -07:00
Alex Vandiver
a4ff6f9ae5 configure-rabbitmq: Switch to await_startup. 2024-05-31 15:41:19 -07:00
Anders Kaseorg
b545abe1e2 typos: Fix typos caught by mwic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-05-20 13:55:00 -07:00
Anders Kaseorg
18579755a8 apt-repos: Deprioritize libgroonga0 14.0.3-1.
It depends libarrow1600, which is missing from the PGroonga repository
on Debian 12.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-05-10 12:04:56 -07:00
Anders Kaseorg
36aa0177bd Revert "apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04) again."
This reverts commit 13e28fc3ac.
2024-04-10 16:07:25 -07:00
Alex Vandiver
acc94a5f32 create-database: Skip stop-server and flush-memcached on standalone Pg.
If running on a stand-alone PostgreSQL server, then supervisor does
exist -- but `stop-server` is useless, and in fact cannot run because
the Zulip directory may not be readable by the `zulip` user.

Detect if this is an application front-end server by looking for
`/home/zulip/deployments`, and use the stop-server and flush-memcached
from there if it exists.  The `create-db.sql` and
`terminate-psql-sessions` files are still read from the local
directory, but those already have precautions from being from a
non-world-readable directory, and are more obviously important to keep
in sync with the `create-database` script.
2024-04-04 16:48:58 -07:00
Anders Kaseorg
ad5abbe54f apt-repos: Remove Ubuntu 24.04 gnupg workaround.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 10:48:22 -07:00
Anders Kaseorg
13e28fc3ac apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04) again.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-03 11:28:49 -07:00
Anders Kaseorg
70914b0475 Remove support for Ubuntu 20.04 and Debian 11.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2024-04-01 13:27:39 -07:00
Anders Kaseorg
4ebc734632 apt-repos: Temporarily work around Ubuntu 24.04 non-installable gnupg.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-31 13:51:31 -07:00
Anders Kaseorg
5d6e616808 Revert "apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04)."
This reverts commit 44e38e8ea7.
2024-03-30 11:29:07 -07:00
Anders Kaseorg
44e38e8ea7 apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04).
This package is replaced by libmagic1t64 1:5.45-3 for the Ubuntu
64-bit time_t transition, but hasn’t been deleted from the archive
yet.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 16:47:30 -07:00
Anders Kaseorg
9015cd7da3 apt-repos: Use Ubuntu 22.04 Teleport repo on Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2024-03-25 17:57:24 -04:00
Tim Abbott
2980d14cb4 scripts: Use Apache Arrow from Groonga repository.
This lets us avoid CI failures related to that repository's signing
key having expired. See https://github.com/apache/arrow/issues/40744.
2024-03-24 20:46:24 -07:00
Alex Vandiver
b0283c42f9 create-database: Treat "zulip" db without "zerver_messages" as empty.
A user who somehow got an empty `zulip` database, but without a
`zerver_messages` table in it, would get stuck in the installer at:

```
++ su postgres -c 'cd / && psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
ERROR:  relation "zulip.zerver_message" does not exist
LINE 1: SELECT COUNT(*) FROM zulip.zerver_message;
                             ^
+ records=
```

Treat a failure to select from `zerver_messages` as having 0 messages,
and continue with the `DROP DATABASE IF EXISTS` / `CREATE DATABASE`
that `create-db.sql` usually does.

Fixes: #29110.
2024-03-08 10:10:19 -08:00
Anders Kaseorg
066ea3ebf9 install: Support Ubuntu 24.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg
553f268b04 ruff: Fix RUF027 Possible f-string without an f prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Alex Vandiver
24d3832926 teleport: Upgrade to Teleport 14. 2024-01-31 16:41:04 -08:00