The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.
The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.
- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
libheif 1.18 is required to be able to parse images generated by iOS
18; none of Zulip's supported distributions package libheif 1.18, so
we pull new version of the package from PPA (Ubuntu) or backports
(Debian).
Using postfix to handle the incoming email gateway complicates things
a great deal:
- It cannot verify that incoming email addresses exist in Zulip before
accepting them; it thus accepts mail at the `RCPT TO` stage which it
cannot handle, and thus must reject after the `DATA`.
- It is built to handle both incoming and outgoing email, which
results in subtle errors (1c17583ad5, 79931051bd, a53092687e,
#18600).
- Rate-limiting happens much too late to avoid denial of
service (#12501).
- Mis-configurations of the HTTP endpoint can break incoming
mail (#18105).
Provide a replacement SMTP server which accepts incoming email on port
25, verifies that Zulip can accept the address, and that no
rate-limits are being broken, and then adds it directly to the
relevant queue.
Removes an incorrect comment which implied that missed-message
addresses were only usable once. We leave rate-limiting to only
channel email addresses, since missed-message addresses are unlikely
to be placed into automated systems, as channel email addresses are.
Also simplifies #7814 somewhat.
Recent versions of postgresql-common's `pg_upgradecluster`, starting
with version 254, (i.e. on Ubuntu 24.04, but not 22.04) will not just
_suggest_ running the analyze, but will do so automatically. While
somewhat helpful, it always does so with `--analyze-in-stages`, which
as noted in f77bbd3323, is actually the incorrect choice for us.
Passing `--no-start` ensures that `pg_upgradecluster` consistently
does not do any analyzing, allowing us to start the cluster manually
and then perform the analyze correctly ourselves.
This uses the same technique used in 840884ec89, to only apply select
parts of the Puppet configuration. This is more correct, and simpler,
than attempting to chop out some base puppet roles, and hack around
the `purge => true` supervisor.d configuration.
Since c8ec3dfcf6, the file must contain the version that was
configured, or we run `ALTER EXTENSION pgroonga UPDATE`; if the file
is missing, and pgroonga was previously installed, it run `CREATE
EXTENSION pgroonga` which will be an error. If the file is present
but pgroonga was not configured, a later attempt to enable pgroonga
will incorrectly run `ALTER EXTENSION pgroonga UPDATE` instead of
`CREATE EXTENSION pgroonga`.
If the file existed on the previous version, touch it in the new
PostgreSQL version. This will ensure that puppet will *always* run
the pgroonga update, which may be necessary in case the pgroonga
version also changed. At worst, if the pgroonga version has not
changed, this will be a safe no-op.
‘rabbitmqctl await_startup’ does not retry to wait for the Erlang
runtime to start, only to wait for the RabbitMQ application to start
once Erlang is running.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This code was originally added in 608173657d in Zulip Server 2.0;
since we can only directly upgrade from 5.0 or later, this code is
guaranteed to have run already. Remove it.
This code was originally added in 5f3765b872 in Zulip Server 4.0;
since we can only directly upgrade from 5.0 or later, this code is
guaranteed to have run already. Remove it.
This code was originally added in 2b146012e1 in Zulip Server 1.7.0;
since we can only directly upgrade from 5.0 or later, this code is
guaranteed to have run already. Remove it.
The python3-yaml dependency was added at install time in 3314fefaec
in Zulip Server 4.0, and this workaround was added in de41a10d38,
also in 4.0. Since we can only directly upgrade from 5.0 or later,
the dependency is guaranteed to be installed already, by one or the
other of those ways. Remove this workaround.
This code was originally added in 382261dc72 in Zulip Server 3.0;
since we can only directly upgrade from 5.0 or later, this code is
guaranteed to have run already. Remove it.
This code was originally added in e705883857 in Zulip Server 5.0;
since we can only directly upgrade from 5.0 or later, this code is
guaranteed to have run already. Remove it.
The current stable branch is on uv, so we no longer need to preserve
the old-style zulip-venv-cache directories from the last 14 days.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Build warnings are unfortunately very common in third-party packages.
They’re difficult to reliably detect since packages don’t always build
from source, and they can’t be whitelisted on a per-package basis
since they’re all attributed to setuptools or an anonymous code
string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This generalizes from thumbnail_workers, to include any other queue.
We only additionally choose to document `email_senders_workers`,
however, since other queues are not guaranteed to work correctly with
multiple consumers.
c5200e8b05 switched `digest_emails` from sending emails by inserting
into the ScheduledEmail table, and being processed later by
`deliver_scheduled_emails`, to inserting into the
`deferred_email_senders` RabbitMQ queue. This moved it from being in
an unmonitored table, to a monitored queue.
This slightly improved throughput -- but began paging, since the
backlog was now in a monitored form. Increase the paging thresholds
to not page for expected behaviour.
The cron jobs are potentially wrapped by Sentry, which logs "cron
failures" and sends emails. We would like those failures to only be
when the cron job itself failed to run successfully -- not when the
underlying metric is outside of its normal range. We would like to
differentiate a failure of the monitoring infrastructure from a
failure of what it is monitoring.
Swap to return 0 on everything except "unknown" results.
`deliver_scheduled_emails` tries to deliver the email synchronously,
and if it fails, it retries after 10 seconds. Since it does not track
retries, and always tries the earliest-scheduled-but-due message
first, the worker will not make forward progress if there is a
persistent failure with that message, and will retry indefinitely.
This can result in excessive network or email delivery charges from
the remote SMTP server.
Switch to delivering emails via a new queue worker. The
`deliver_scheduled_emails` job now serves only to pull deferred jobs
out of the table once they are due, insert them into RabbitMQ, and
then delete them. This limits the potential for head-of-queue
failures to failures inserting into RabbitMQ, which is more reasonable
than failures speaking to a complex external system we do not control.
Retries and any connections to the SMTP server are left to the
RabbitMQ consumer.
We build a new RabbitMQ queue, rather than use the existing
`email_senders` queue, because that queue is expected to be reasonably
low-latency, for things like missed message notifications. The
`send_future_email` codepath which inserts into ScheduledEmails is
also (ab)used to digest emails, which are extremely bursty in their
frequency -- and a large burst could significantly delay emails behind
it in the queue.
The new queue is explicitly only for messages which were not initiated
by user actions (e.g., invitation reminders, digests, new account
follow-ups) which are thus not latency-sensitive.
Fixes: #32463.
The refactoring in 4e28e1d3ff incorrectly switched a check for
`if args.from_git` into `if NEW_ZULIP_MERGE_BASE`, which is
incorrect -- the merge-base is always defined, it may just match the
version. This led to errors when installing from tarball, without a
git repo.
Since the run_hooks command was already set up to take a `--from-git`
argument, but was ignoring it, pass down that flag from
upgrade-zulip-stage-3 when necessary, and swap the run_hooks logic
back to basing the version-resolution logic on that flag.
When flushing caches, we want to ensure that even processes which may
have a wrong cache-key-prefix know to fetch the latest data from the
database. This is complicated by the cache-key-prefixes being stored
on disk, and thus checking that every cache delete is not sufficiently
performant.
We store the list of cache-key-prefixes in the cache, itself, with no
prefix. This cache is updated when a new cache-key is written, and is
also allowed to lapse after 24 hours. Updating this global cache
entry on new prefix creation ensures that even a
not-yet-restarted-into deployment will have its caches appropriately
purged if changes are made to the underlying data.
However, this both adds a cache-get, as well as multiplies the size of
all cache clears; for large bulk clears (e.g. for stream renames,
which clear the cache for all message-ids in them) this may prove
untenable.
The old /srv/zulip-npm-cache system has been unused for two
years (Zulip Server ≥ 7.0). We can just delete this directory.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
