Commit Graph

55915 Commits

Author SHA1 Message Date
Anders Kaseorg
4c1a7b9938 copy_and_paste: Fix HTML injection bug in construct_copy_div.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e1029b59ed)
2024-05-07 12:55:13 -07:00
sujal shah
9dd8b2bb82 setting_ui: Fix sorting by "Invited by" column.
Previously, the sorting was broken due to
incorrect referencing of the property.
The code has been updated to use the "referrer_name"
property instead of "ref".

(cherry picked from commit 01e898252b)
2024-05-07 12:52:51 -07:00
Alex Vandiver
b9ff0c4bf0 django: Skip running config and database checks on process start.
The process of running Django's built-in database and config checks
can be very heavy-weight, potentially taking multiple seconds:

```
$ hyperfine './manage.py print_initial_password iago@zulip.com'  './manage.py print_initial_password iago@zulip.com --skip-checks'
Benchmark 1: ./manage.py print_initial_password iago@zulip.com
  Time (mean ± σ):      4.943 s ±  0.722 s    [User: 4.434 s, System: 0.311 s]
  Range (min … max):    4.415 s …  6.835 s    10 runs

Benchmark 2: ./manage.py print_initial_password iago@zulip.com --skip-checks
  Time (mean ± σ):      1.786 s ±  0.113 s    [User: 1.598 s, System: 0.162 s]
  Range (min … max):    1.576 s …  1.999 s    10 runs

Summary
  './manage.py print_initial_password iago@zulip.com --skip-checks' ran
    2.77 ± 0.44 times faster than './manage.py print_initial_password iago@zulip.com'
```

This extends the window during which nginx is forced to serve 502's to
clients.  f5f6a3789b added an explicit `manage.py check` during
server restarts, and fa77be6e6c added one during upgrades; as such,
we expect that any check failures will already have been caught when
performing a restart or upgrade, and there is no point in running them
on process startup.

(cherry picked from commit 48b925931e)
2024-05-07 12:52:51 -07:00
Pratik Chanda
783a2a3f27 left_sidebar: Fix DM row highlight linger on narrowing away.
Earlier, DM row remained highlighted when narrowed to different
part of left sidebar other than recent and inbox since on narrow
event, `handle_narrow_activate` wasn't properly handling unnarrow
events. It wasn't calling `update_private_messages` to update the
state of the DM list.

This commit fixes the behaviour by updating the state on narrowing
away from DM rows.

Fixes: zulip#27698.

Co-authored-by: Rinwaoluwa <peterodejobi9@gmail.com>
(cherry picked from commit 0708505acb)
2024-05-07 12:52:51 -07:00
Pratik Chanda
5a61c2dde0 views: Fix message_list.current not updating to undefined on narrow.
Earlier when users narrowed to recents or inbox, message_list.current
was updating to last dm row selected for first few filter checks in
'narrow_state' which in turn would set active_user_ids to last dm
row selected. This resulted in DM highlight lingering.
The narrow_state.filter is supposed to return undefined when narrowed
to recents or inbox.

This commit fixes the behaviour by setting 'current' undefined before
filter checks on narrow.

Fixes part of #27698.

(cherry picked from commit 8c89725fcc)
2024-05-07 12:52:51 -07:00
Alex Vandiver
ed27910bc1 import: Skip "active" check, and analytics, for crossrealm bots.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.

(cherry picked from commit a7ec7e924e)
2024-05-07 12:52:51 -07:00
Mateusz Mandera
da4ce971eb register_server: Check write access to secrets file when rotate_key.
Using --rotate-key without write access to the secrets file is currently
quite painful, since you end up rotating your registration's secret with
no local record of it; so effectively you lose your registration and
need help from support. We should just prevent this failure mode.

(cherry picked from commit 89a22fb076)
2024-05-07 12:52:51 -07:00
Alex Vandiver
ca7cb771fb release: Fix changelog output.
9bd340957f changed the headers; while we no longer store the
changelog text in the Github release object, we should still output
the changelog to STDOUT correctly.

(cherry picked from commit db4e093ad7)
2024-05-07 12:52:51 -07:00
Mateusz Mandera
45585acc4e push_notifs: Make push_notifications_enabled more resistant to flapping.
Fixes #28403

Uses redis to remember the last time push notifications were experienced
working. This needs to work across processes, so can't be done just in
memory.
As this is transient data that's fairly harmless to lose and thus
doesn't require the persistence benefits of the database, and we're
keeping a single "row", so don't need an entire new db table, we settle
on using redis instead of postgres. This is also consistent with how we
store other kinds of such transient data.

(cherry picked from commit 962ab13203)
2024-05-07 12:52:51 -07:00
Mateusz Mandera
2deb166a23 rate_limiter: Extract KEY_PREFIX to redis_utils.
(cherry picked from commit 630335142a)
2024-05-07 12:52:50 -07:00
Aman Agrawal
0afcfccd20 inbox_row: Don't show topic visibility toggle button by default.
We emulate the same behaviour of hiding topic visibility button
by using opacity if for default topic visibility state.

(cherry picked from commit 9846c4c476)
2024-05-07 12:52:50 -07:00
Prakhar Pratyush
37af07c544 inbox: Remove 'default' visibility policy indicator.
We no longer show the 'default' visibility policy in
the inbox view. 'Unmute', 'Follow', and "Mute' are
still shown.

For topics with 'default' visibility policy, it is
still possible to change the visibility policy via the
three-dot menu.

The motivation is not to have the default state so
prominently called out.

(cherry picked from commit 6630981de8)
2024-05-07 12:49:35 -07:00
Artur Szcześniak
f5b102f146 integrations: Fix confusing wording in alertmanager integration.
We use Alertmanager as an aggregation place for example for failing CI pipelines,
and `graph` does not always reflect the source of the alert. It's called `source` originally
and I think it should stay this way.

(cherry picked from commit 706be812b9)
2024-05-07 12:46:39 -07:00
Alex Vandiver
92da597763 backport: Fix the commit-ids that are used.
The commit hashes that appear in the `pull/12345/head` ref are the
ones _before_ any final rebase occurs, and as such may not match to
any commit hashes which exist in `main`.

Use the GitHub GraphQL API to pull the last "merge commit" on the PR,
which is post-rebase, and use that as the target commit when
cherry-picking.  Then walk backwards from that commit, including every
sequential commit which is still associated with the PR; we do this
because during the merge, commits may be added or removed, so the PR
is not reliable in the commit count.

(cherry picked from commit 06543b54b1)
2024-05-07 12:46:39 -07:00
Karl Stolley
aad3e4fff0 reactions: Increase prominence of own reactions.
To address feedback that own emoji reactions were not sufficiently distinct, we increase
the prominence of one's own emoji reactions by:

- Setting a pixel-and-a-half border around own reactions.
- Reducing the alpha on the inner drop shadow of other reactions.
- Increasing the contrast on own reactions borders (and decreasing
  them, in dark mode, on other reactions).
- Space around the emoji is maintained as in the current design in the own reactions.

Other reactions benefit from an additional half pixel of padding, top and bottom,
which is necessary to keep the pills (and the hover reaction button) the same
height as each other--regardless of whether there's an own reaction among them
or not. Padding is reduced in line with the increased border on own reactions.

(cherry picked from commit 7930209929)
2024-05-07 12:43:23 -07:00
Alex Vandiver
131fe9418f install: Actually add --puppet-classes argument.
7187146422 only added it to the internal parsing and the help, not
the getopt invocation which would make the command accept the
argument.

(cherry picked from commit a853b8849f)
2024-05-07 11:18:22 -07:00
Alex Vandiver
d7bc104724 docs: Improve PostgreSQL documentation.
This removes a bunch of out-of-date documentation which is better
addressed outside of Zulip, and provides more details on running a
PostgreSQL server on a separate host.
2024-05-07 11:18:22 -07:00
Alex Vandiver
416e254a9e install: Move PUPPET_CLASSES env var to --puppet-classes argument.
`--no-init-db` is used to silence the need for `--hostname` and
`--email` arguments; it is a proxy for "this is not a frontend host."
We would ideally like to use `has_class` to know if the user's
provided puppet classes are include an `app_frontend`, and thus
`--hostname` and `--email` are required -- but doing that requires
several other steps, and we would like this feedback to be immediate.

We make the presence of `--puppet-classes` equivalent to
`--no-init-db`, since nearly every configuration with
`--puppet-classes` does not install both a database and a frontend,
which is what is required to initialize a database.
2024-05-07 11:18:22 -07:00
Alex Vandiver
bc648a8de5 install: Remove the hidden VIRTUALENV_NEEDED parameter.
We can do this behaviour better by using `has_roles`.
2024-05-07 11:18:22 -07:00
Alex Vandiver
b0bd38cae6 install: Only install node on frontend hosts. 2024-05-07 11:18:22 -07:00
Alex Vandiver
14d4900648 puppet: Split out a zulip::profile::standalone_nodb class. 2024-05-07 11:18:22 -07:00
Alex Vandiver
133c8249b1 puppet: Remove gettext, zulip::static_asset_compiler.
These came in via d0dcc8bf26, which looks like it copied the comment
from the provisioning code.  Production installs (even from git) do
not call `./manage.py makemessages`, so there is no reason to require
this for production deployments.
2024-05-07 11:18:22 -07:00
Alex Vandiver
159e962a67 docs: Factor out PostgreSQL support matrix. 2024-05-07 11:18:22 -07:00
Alex Vandiver
47c2c84bbc settings: Support multiple database replicas in REMOTE_POSTGRES_HOST.
The libpq client library, used under the hood by psycopg2, supports
passing a list of hosts; they are tried sequentially until one of them
works[^1].

In cases where this is used, it is often the case that the other
servers are read-only hot spare replicas.  Since Zulip does not expect
to be in a read-only transaction, we require that the server that we
connect to be writable, by passing `target_session_attrs`[^2].

To limit how long we may block connecting to a potentially bad host
before moving on, we set `connection_timeout` from null (meaning
forever) to 2 (the lowest supported value)[^3], so we move on quickly
in the case that the server is running but unable to handle new
connections.

[^1]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-MULTIPLE-HOSTS
[^2]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS
[^3]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT
2024-05-07 11:18:22 -07:00
Alex Vandiver
eb202446f8 postgresql: Decouple wal_g from streaming replication.
Streaming replication may be used even if `wal-g` is not -- as long as
the user can move a copy of the base backup to the replica (e.g. using
`pg_basebackup`).  Remove the warning about this combination, and move
the `primary_conninfo` setting outside of the `s3_backups_bucket`
check.
2024-05-07 11:18:22 -07:00
Alex Vandiver
c5fe866297 postgresql: Drop lowered max_wal_senders value.
This was originally added in a PostgreSQL 9.5 configuration, where the
default is 0, meaning replication is disabled[^1].  In PostgreSQL 10
and above, the default is 10[^2], and there is no reason to lower it
from that value.

[^1]: https://www.postgresql.org/docs/9.5/runtime-config-replication.html
[^2]: https://www.postgresql.org/docs/10/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER
2024-05-07 11:18:22 -07:00
Alex Vandiver
0e7fb8f56c create-database: Skip stop-server and flush-memcached on standalone Pg.
If running on a stand-alone PostgreSQL server, then supervisor does
exist -- but `stop-server` is useless, and in fact cannot run because
the Zulip directory may not be readable by the `zulip` user.

Detect if this is an application front-end server by looking for
`/home/zulip/deployments`, and use the stop-server and flush-memcached
from there if it exists.  The `create-db.sql` and
`terminate-psql-sessions` files are still read from the local
directory, but those already have precautions from being from a
non-world-readable directory, and are more obviously important to keep
in sync with the `create-database` script.
2024-05-07 11:18:22 -07:00
Alex Vandiver
2c7deef0ea puppet: Fix indentation. 2024-05-07 11:18:22 -07:00
Alex Vandiver
84d5e58e3a puppet: Switch from top-level fact variables to facts dict.
This is a cherry-pick of 57f8b48ff9 and 235e2eefc8.
2024-05-07 11:18:22 -07:00
Alex Vandiver
a8b5398013 topic: Fix history order for topic moves.
5c96f94206 mistakenly appended, rather than prepended, the edit to
the history.  This caused AssertionErrors when attempting to view the
history of moved messages, which check that the `last_edit_time`
matches the timestamp of the first edit in the list.

Fix the ordering, and update the `edit_history` for messages that were
affected.  We limit to only messages edited since the commit was
merged, since that helps bound the affected messages somewhat.

Cherry picked from commit b747ea285f.
Because we have already merged other migrations, this migration is
renumbered to 0517, which will appear as a no-op migration on `main`.
2024-05-06 09:06:47 -07:00
Alex Vandiver
32dbcc3489 puppet: Only use the cron resource on Puppet 5.x.
Puppet 6.x and above moved the `cron` resource type out of core
Puppet, into a "cron_core" module[^1]; thus Debian 12 and Ubuntu
24.04, with the newer Puppet, fail to compile if backups are
configured:

```
Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron' (file: /home/zulip/deployments/current/puppet/zulip/manifests/postgresql_backups.pp, line: 32, column: 3) on node zulip.example
```

However, we only use the `cron` package to assert that we no longer
use it, for installs which are coming up from Zulip 4.x where it _was_
used.

Since Zulip 4.x does not support Ubuntu 24.04 or Debian 12, having
Puppet >= 6.x means that the cron must have been previously removed in
a previous upgrade, and we can skip the block.

In `main`, this block is removed entirely.

[^1]: https://forge.puppet.com/modules/puppetlabs/cron_core/readme
2024-04-02 11:54:29 -07:00
Alex Vandiver
791969de3e puppet: Do not include postgresql_common for all PostgreSQL clients.
It pulls in the server.

(cherry picked from commit 80ef38757a)
2024-03-28 12:31:46 -07:00
Alex Vandiver
96dfec1f1d puppet: Install the same version of postgres-client as the server.
We require a `pg_dump` whose version matches the version of the server
we are configured against (see 3a8b4b0205).  Installing the latest
`postgresql-client` does not guarantee that we have such a binary
present.

(cherry picked from commit 23504308fb)
2024-03-28 12:31:46 -07:00
Anders Kaseorg
a387a2714b semgrep: Broaden select_related() and prefetch_related() rules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0cc40605b0)
2024-03-28 12:31:46 -07:00
Alex Vandiver
69726408eb semgrep: Enforce no bare select_related() calls.
Previously, #26419 addressed the majority of these calls, but did not
prevent more from creeping in.  Remove the one remaining
callsite (after the cleanup from the previous commits), and ban any
future use of the pattern.

(cherry picked from commit 7d0c12aa3a)
2024-03-28 12:31:46 -07:00
Alex Vandiver
ad21451d01 messages: Stop joining through every table.
Calling `.select_related()` with no arguments joins through every
possible table, recursively.  In this case, this currently produces a
query which joins through forty-three tables.

This is rather inefficient, particularly for what is a very common
call which should be very fast.

No callsite depends on having prefetched any joined table on the
object; drop all of the joins.

(cherry picked from commit 6ace34c374)
2024-03-28 12:31:46 -07:00
Alex Vandiver
105e699ac6 missedmessage_emails: Clear caches and db query tracking per-loop.
Otherwise, these accumulate and leak memory.

(cherry picked from commit 3cbce0c5c7)
2024-03-28 12:31:46 -07:00
Alex Vandiver
1e2e1b8888 version: Update version after 8.3 release. 2024-03-19 22:42:46 +00:00
Alex Vandiver
a17af4b1ea Release Zulip Server 8.3. 8.3 2024-03-19 22:07:05 +00:00
Alex Vandiver
7eeaaf4bb8 message_edit: Do set differences in QuerySets. 2024-03-19 19:32:12 +00:00
Alex Vandiver
9dca0e4b44 message_edit: Handle user, not subscription, collections.
Nothing about the subscription is relevant -- we instead use
collections of User objects for readability.
2024-03-19 19:32:12 +00:00
Alex Vandiver
5f823bf62f message_edit: Perform cross join in SQL, not Python.
The database can do this faster than we can, and it reduces the amount
of information transmitted over the wire.
2024-03-19 19:32:12 +00:00
Alex Vandiver
19472d72e3 message_edit: Pull user_ids_gaining_usermessages higher, rename. 2024-03-19 19:32:12 +00:00
Alex Vandiver
4b87e6c8d9 message_edit: Handle previous subscribers and private-history streams. 2024-03-19 19:32:12 +00:00
Alex Vandiver
3db1733310 CVE-2024-27286: Delete dangling UserMessage rows.
This cleans up dangling UserMessage rows for moved messages which were
affected by bugs in one of the previous two commits.
2024-03-19 19:32:12 +00:00
Alex Vandiver
e3b50fa373 CVE-2024-27286: Remove UserMessage rows for non-active Subscriptions.
A user who was no longer subscribed to a private stream kept their
UserMessage row for a message sent while they were in it; this is
expected.  However, they _also_ kept that row even if the message was
moved to a different private stream that they were also not subscribed
to.  This violates the invariant that users without subscriptions
never have UserMessage rows.
2024-03-19 19:32:12 +00:00
Alex Vandiver
a4141782cc CVE-2024-27286: Run usermessage modifications even for change_one.
This `if new_stream is not None` block was improperly indented,
causing it to only run if the propagation mode was not `change_one`.
Since the block controlled creation and deletion of UserMessage rows,
this led to messages being improperly still visible to members of the
old stream if they were being moved from public to private streams.
Clients also failed to receive `delete_message` events, so the
messages remained visible in their feeds until they reloaded the
application.
2024-03-19 19:32:12 +00:00
Alex Vandiver
b3eb6680ef create-database: Treat "zulip" db without "zerver_messages" as empty.
A user who somehow got an empty `zulip` database, but without a
`zerver_messages` table in it, would get stuck in the installer at:

```
++ su postgres -c 'cd / && psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
ERROR:  relation "zulip.zerver_message" does not exist
LINE 1: SELECT COUNT(*) FROM zulip.zerver_message;
                             ^
+ records=
```

Treat a failure to select from `zerver_messages` as having 0 messages,
and continue with the `DROP DATABASE IF EXISTS` / `CREATE DATABASE`
that `create-db.sql` usually does.

Fixes: #29110.
(cherry picked from commit ed70b62ea8)
2024-03-19 13:47:16 -04:00
Sayam Samal
3034ad2931 compose: Fix topics required banner closing on enter to send.
We were updating the compose banners on every `keyup` event on the
topic input. Since, `keyup` also gets triggered for the modifier and
non-printing keys such as "Enter", this lead to banner for topic
required being closed via the `check_posting_policy_for_compose_box`
when pressing "Enter" to send a message with no topic.

This bug was probably introduced in 5c993f0, which moved additional
logic into `update_on_recipient_change`.

To solve this issue, we use the `input` event instead of the `keyup`
event to update the compose banners only when the value inside the
input element changes.

This change also prevents the the compose banner from being closed
when we only press modifier keys - such as Shift.

(cherry picked from commit 44f313633f)
2024-03-19 13:47:16 -04:00
Alex Vandiver
1aabbf1d8c wal-g: Add support for incremental backups.
This only defaults to on for local-disk backups, since they are more
disk-size-sensitive, and local accesses are quite cheap compared to
loading multiple incremental backups from S3.

(cherry picked from commit 323e1e92b7)
2024-03-19 10:41:20 -07:00