8ab400b95d adds a couple guardrails
againts generating an exporting realm with no usable owner and admin
accounts.
As per #34275, this commit lifts these guardrails to allow generating
export with no usable owner accounts. Export data without usable user
account might still have other uses other than for import.
This adds infrastructure to prevent duplicate custom emails from being
sent to users when the management command is run multiple times
with identical email template content.
The implementation uses RealmAuditLog tracking with a new
CUSTOM_EMAIL_SENT event type (810) that stores a hash of the email
template content.
Fixes: #19529.
This commit includes the following changes:
- Add an administrator setting to customize the Welcome Bot
message when sending an invitation.
- Add an API endpoint to test the customized Welcome Bot message
by sending a copy of the message to the administrator.
Fixes#27663.
Co-authored-by: Akarsh Jain <akarsh.jain.790@gmail.com>
Fixes#20028.
There's no reason to have a special `RealmCreationKey` class - the
`Confirmation` system already does this job.
This is somewhat complicated by the need to write a migration for
`RealmCreationKey`->`Confirmation` for pre-existing, valid objects, to
avoid breaking realm creation links that haven't been used yet.
This is most important for `send_zulip_update_announcements`, which
can race with the version run as a post-deploy hook. However, all of
these crons can tolerate being slightly delayed, and there's little
benefit to them taking CPU or possibly hitting odd borderline race
conditions when the deploy is in progress.
For safety, we only trust the deploy lockfile if it was created
within the last hour -- deploys should not take more than an hour, and
failing to ever run hourly crons is much worse than perhaps running
them during a real very-long deploy.
Adds delete_expired_demo_organizations to the archive_messages
management command, which is run as a cron job.
Adds "demo_expired" as a `RealmDeactivationReasonType` to be
used for this specific case of calling do_deactivate_realm.
The function loops through non-deactivated realms that have a
demo organization scheduled deletion datetime set that is less
than the current datetime.
Previously, the export_search command only supported the legacy
Personal recipient type for 1:1 direct messages. With this change,
the command first checks for a DirectMessageGroup for the two users
and uses it if present. If not, we fall back to the legacy logic. This
keeps compatibility as we migrate 1:1 DMs to use DirectMessageGroup.
This pulls in changes from the latest django-jinja[^1]
`makemessages.py` monkey-patching. Specifically, it adds support for
`trimmed`, `notrimmed`, and the `ext.i18n.trimmed` policy. We enable
that, which removes unsightly and unnecessary whitespace inside of
`{% trans %}` blocks.
[^1]: aac828ca63/django_jinja/management/commands/makemessages.py
This is done in a best-effort way -- we rely on the final ACL check
happening when the message is sent. However, by then we will be in
the `email_mirror` worker, with no way to convey to the original SMTP
client that the sending failed; giving the client the information will
allow them to provide it back to the end-user, instead of merely being
swallowed up in the worker.
Using postfix to handle the incoming email gateway complicates things
a great deal:
- It cannot verify that incoming email addresses exist in Zulip before
accepting them; it thus accepts mail at the `RCPT TO` stage which it
cannot handle, and thus must reject after the `DATA`.
- It is built to handle both incoming and outgoing email, which
results in subtle errors (1c17583ad5, 79931051bd, a53092687e,
#18600).
- Rate-limiting happens much too late to avoid denial of
service (#12501).
- Mis-configurations of the HTTP endpoint can break incoming
mail (#18105).
Provide a replacement SMTP server which accepts incoming email on port
25, verifies that Zulip can accept the address, and that no
rate-limits are being broken, and then adds it directly to the
relevant queue.
Removes an incorrect comment which implied that missed-message
addresses were only usable once. We leave rate-limiting to only
channel email addresses, since missed-message addresses are unlikely
to be placed into automated systems, as channel email addresses are.
Also simplifies #7814 somewhat.
This works around tus/tusd#322, which in turn is caused by
aws/aws-sdk-go-v2#1816. This requires separate authentication via
service account key.
Fixes: #34186.
The tusd protocol allows DELETE requests ("terminations") at any
point, including after a file has successfully been uploaded. This
can allow tusd to remove a file from the bucket, out from under Zulip.
We use the new-in-2.7.0 pre-terminate hook to look up the file which
the client is requesting to terminate, and reject the termination if
it is a file that the Zulip database is already aware of.
In deploys `nginx_listen_port` set, tusd would fail to send its hook
requests, as it assumed that nginx would always be listening on
127.0.0.1:80.
Set the `nginx_listen_port` on the hook URL, if necessary.
This allows us to get rid of the call to `get_consented_user_ids` in
`fetch_usermessages`. Now it's only called at the beginning of the
export, eliminating the redundant db query and also resolving the
potential for data consistency issues, if some users change their
consent setting after the export starts.
Now the full export process operates with a single snapshot of these
consenting user ids.
These ids need to be plumbed through via a file rather than normal arg
passing, because this is a separate management command, run in
subprocesses during the export.
We already do this in computed_settings.py, but only if the
S3 (secret) key is set. Those aren't required to be set, and tusd
_requires_ a region, so we try again to suss it out here.
Removed `is_billing_admin` user property as it is no longer used since
billing permissions are now determined by `can_manage_billing_group`
realm setting.
`deliver_scheduled_emails` tries to deliver the email synchronously,
and if it fails, it retries after 10 seconds. Since it does not track
retries, and always tries the earliest-scheduled-but-due message
first, the worker will not make forward progress if there is a
persistent failure with that message, and will retry indefinitely.
This can result in excessive network or email delivery charges from
the remote SMTP server.
Switch to delivering emails via a new queue worker. The
`deliver_scheduled_emails` job now serves only to pull deferred jobs
out of the table once they are due, insert them into RabbitMQ, and
then delete them. This limits the potential for head-of-queue
failures to failures inserting into RabbitMQ, which is more reasonable
than failures speaking to a complex external system we do not control.
Retries and any connections to the SMTP server are left to the
RabbitMQ consumer.
We build a new RabbitMQ queue, rather than use the existing
`email_senders` queue, because that queue is expected to be reasonably
low-latency, for things like missed message notifications. The
`send_future_email` codepath which inserts into ScheduledEmails is
also (ab)used to digest emails, which are extremely bursty in their
frequency -- and a large burst could significantly delay emails behind
it in the queue.
The new queue is explicitly only for messages which were not initiated
by user actions (e.g., invitation reminders, digests, new account
follow-ups) which are thus not latency-sensitive.
Fixes: #32463.
Previously, tusd printed unnecessary logs on startup while running
the tools/run-dev script. This commit resolves the issue by setting
the verbose flag to false, which defaults to true if not specified.
The required PR adding this flag was introduced in
https://github.com/tus/tusd/pull/1218.
Fixes#32301.
