Only clear_scheduled_emails previously took a lock on the users before
removing them; make deliver_scheduled_emails do so as well, by using
prefetch_related to ensure that the table appears in the SELECT. This
is not necessary for correctness, since all accesses of
ScheduledEmailUser first access the ScheduledEmail and lock it; it is
merely for consistency.
Since SELECT ... FOR UPDATE takes an UPDATE lock on all tables
mentioned in the SELECT, merely doing the prefetch is sufficient to
lock both tables; no `on=(...)` is needed to `select_for_update`.
This also does not address the pre-existing potential deadlock from
these two use cases, where both try to lock the same ScheduledEmail
rows in opposite orders.
(cherry picked from commit 4c518c2bba)
No codepath except tests passes in more than one user_profile -- and
doing so is what makes the deduplication necessary.
Simplify the API by making it only take one user_profile id.
(cherry picked from commit ebaafb32f3)
`deliver_scheduled_emails` and `deliver_scheduled_messages` use their
respective tables like a queue, but do not have guarantees that there
was only one consumer (besides the EMAIL_DELIVERER_DISABLED setting),
and could send duplicate messages if multiple consumers raced in
reading rows.
Use database locking to ensure that the database only feeds a given
ScheduledMessage or ScheduledEmail row to a single consumer. A second
consumer, if it exists, will block until the first consumer commits
the transaction.
(cherry picked from commit 1e67e0f218)
Sometimes the Slack import zip file we get isn't quite the canonical
form that Slack produces -- often because the user has unzip'd it,
looked at it, and re-zip'd it, resulting in extra nested directories
and the like.
For such cases, support passing in a path to an unpacked Slack export
tree.
Slack bot emails generated by us can be duplicate for two bots.
If such a case occur, append a counter to the email to make it
unique.
For maintaining the counter of duplicate emails and the final
email assigned to each bot, a class based approach is used with
static variables and static (class) methods. This keeps all the
data related to slack bot emails at the same place and easily
accessible from anywhere inside the module (without defining any
class object and passing it around).
Fixes: #16793
Zulip attempts to validate that the regular expressions that admins
enter for linkifiers are well-formatted, and only contain a specific
subset of regex grammar. The process of checking these
properties (via a regex!) can cause denial-of-service via
backtracking.
Furthermore, this validation itself does not prevent the creation of
linkifiers which themselves cause denial-of-service when they are
executed. As the validator accepts literally anything inside of a
`(?P<word>...)` block, any quadratic backtracking expression can be
hidden therein.
Switch user-provided linkifier patterns to be matched in the Markdown
processor by the `re2` library, which is guaranteed constant-time.
This somewhat limits the possible features of the regular
expression (notably, look-head and -behind, and back-references);
however, these features had never been advertised as working in the
context of linkifiers.
A migration removes any existing linkifiers which would not function
under re2, after printing them for posterity during the upgrade; they
are unlikely to be common, and are impossible to fix automatically.
The denial-of-service in the linkifier validator was discovered by
@erik-krogh and @yoff, as GHSL-2021-118.
re2[1] compiles (strictly) regular expressions to deterministic finite
automata, which guarantees linear-time behavior; `google-re2` is a
drop-in replacement for the `re` module which uses re2 under the hood.
[1]: https://github.com/google/re2/
This removes some steps which are no longer necessary to be run
in the production upgrade script. The steps were used due to
errors related to supervisor failing to restart which was resolved
in the commit 08c39a7388.
(cherry picked from commit dc2066c7e8)
We currently configure ‘APT::Get::Assume-Yes’ in our custom Docker
image, but this is the only place we rely on it (outside of the
Dockerfile itself), and it’s better not to.
Also ‘apt-get remove && apt-get purge’ is the same as just ‘apt-get
purge’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit db476bdc51)
When Github Actions run in Docker, the default pid 1 entrypoint is
`tail -f /dev/null`. PID 1 is responsible for propagating signals to
its children, and calling `waitpid()` on defunct processes; `tail`
does not do these things. This results in zombie processes piling up
inside the container, which is not an issue in most contexts.
However, it affects `start-stop-daemon`, which hangs when stopping
daemon processes, as they are never reaped. This appears in CI as
`/etc/init.d/supervisor restart` never being able to succeed.
Run the docker container with `--init`, which spawns a
`/sbin/docker-init` PID 1 to handle the job of an init process.
(cherry picked from commit 2daad58afa)
This adds a check in the current production suite of
CI that upgrades a previous release of zulip server
with a newer one.
Fixes#18346.
(cherry picked from commit e648ad3477)
It feels like the "Same as" content was unnecessarily requiring the
user to bounce around in these cases.
(I've left the "Same as" text for the Ubuntu ones, where it's two
steps in a row to follow).
It recently started failing on Debian 10 (buster). We immediately
follow this by replacing these packages with our own versions from
pip.txt, anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 902883d818)
Apparently, our slack compatible outgoing webhook format didn't
exactly match Slack, especially in the types used for values. Fix
this by using a much more consistent format, where we preserve their
pattern of prefixing IDs with letters.
This fixes a bug where Zulip's team_id could be the empty string,
which tripped up using GitLab's slash commands with Zulip.
Fixes#19588.
As of yesterday, the GitHub Actions ubuntu-16.04 environment has been
removed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e96abc3c5a)
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.
Since now bullseye is released as a stable
version, it's support can be documented.
(cherry picked from commit 502697d239)
With copy-editing from tabbott, and also a migration to use
LATEST_RELEASE_VERSION, which will be correct even on the /latest/
paths.
Fixes#19695.
(cherry picked from commit 3b1cb0b25a)
This fixes a regression where one could end up deactivating all owners
of a realm when trying to synchronize LDAP with the `is_realm_admin`
flag configured in `AUTH_LDAP_USER_FLAGS_BY_GROUP`.
With tweaks by tabbott to add is_moderator as well.
Fixes#18677.
Our backend processor is not yet sufficiently CommonMark compliant to
accept Prettier formatted Markdown files.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7df2be0965)
To avoid confusing the linter later when Prettier lowercases these.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit fdb7ec8c9e)
