Commit Graph

55956 Commits

Author SHA1 Message Date
Anders Kaseorg
bdfb0c0dc8 codespell: Fix spelling mistakes caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e53e3af0f6)
2024-07-27 09:27:23 -07:00
Anders Kaseorg
cf73bf338d requirements: Fix Python 3.12 issue in pip._vendor.pkg_resources.
76ef30913d

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0d30ae8f44)
2024-07-27 09:27:23 -07:00
Mateusz Mandera
c494a4331c register_server: Call crudini with --inplace when rotating secret key.
This is needed for the command to work in docker-zulip, where the zulip
user may not have write permissions to the parent directory of the
secrets file. By default, crudini creates a temporary file, which makes
the command fail under those conditions.
With --inplace, the secret file gets written to directly without
creation of temporary files.

(cherry picked from commit d89ab2694f)
2024-07-26 14:27:45 -07:00
Alex Vandiver
9ea1b99dbb nginx: Strip off request headers which might affect S3's behaviour.
Clients making requests to Zulip with a `Authorization: Basic ...` for
an upload in S3 pass along all of their request headers to the S3
backend -- causing errors of the form:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidArgument</Code>
  <Message>Only one auth mechanism allowed; only the X-Amz-Algorithm
  query parameter, Signature query string parameter or the
  Authorization header should be specified</Message>
  <ArgumentName>Authorization</ArgumentName>
  <ArgumentValue>Basic ...</ArgumentValue>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
</Error>
```

Strip off all request headers which AWS reports that S3 may read[^1].

Fixes: #30180.

[^1]: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRequestHeaders.html

(cherry picked from commit 549f4fe00b)
2024-07-26 14:27:45 -07:00
Tim Abbott
a932d7e99e scripts: Use Apache Arrow from Groonga repository.
This lets us avoid CI failures related to that repository's signing
key having expired. See https://github.com/apache/arrow/issues/40744.

(cherry picked from commit 2980d14cb4)
2024-07-26 14:27:45 -07:00
Prakhar Pratyush
551f382721 data_import: Remove gitter data import tool.
Gitter broke their older API as part of being integrated
into Matrix.

Their announcement blog says:
"Anything left using the Gitter APIs will need to be
updated to use the Matrix API"

This commit drops the legacy Gitter import tool and
we plan to build a new one for Matrix in future.

(cherry picked from commit b17a006bf0)
2024-07-26 10:41:29 -07:00
Tim Abbott
a73c3089c9 i18n: Update translations from Transifex.
This will be the last commit of this form before the 8.x branch is
deleted in Transifex.
2024-07-25 14:15:26 -07:00
Alex Vandiver
650d4e715f version: Update version after 8.4 release. 2024-05-09 19:06:26 +00:00
Alex Vandiver
6800b1f688 Release Zulip Server 8.4. 8.4 2024-05-09 18:35:59 +00:00
Alex Vandiver
c26baa2a8e requirements: Drop the --no-binary for lxml and xmlsec.
Building these libraries from source requires too much memory, and
causes OOMs on a host with 4GB of RAM when still running Zulip.

Building from source was enabled in `main` to work around
xmlsec/python-xmlsec#320, which does not occur with the xmlsec version
in the 8.x branch, as xmlsec/python-xmlsec#314 (incompatibilities with
Ubuntu 20.04) causes us to pin xmlsec<1.3.14.

As such, we switch back to using prebuilt wheels.  The version
mismatch from xmlsec/python-xmlsec#320, if real, is not new, so this
exposes us to no more risk than before.
2024-05-09 09:34:04 -07:00
Alex Vandiver
5e8cc22b0a i18n: Update translation data from Transifex. 2024-05-08 20:09:11 +00:00
sayyedarib
52d51294dc direct_message: Allow DMs to bots and self regardless of policy.
Previously, in DM disabled org messaging to bot was not working when
starting new conversation and adding bot as recipient because of not
updating on recipient change. And secondly, self messaging was not
allowed.
This commit ensures that the DM to bot and self are allowed irrespective
of dm restrictions.

tests: Verify DMs adhere to DM restriction policy.

Fixes #28412

Signed-off-by: sayyedarib <sayyedaribhussain4321@gmail.com>
(cherry picked from commit 5c993f0479)
2024-05-08 10:36:52 -07:00
Alex Vandiver
64189bfb7f docs: Update for wal-g not being required for replication.
(cherry picked from commit a7022bdfec)
2024-05-08 09:44:35 -04:00
Alex Vandiver
8e8f6ebcdb docs: Document PostgreSQL multiple database host failover.
Added in bd82c6edf9, this is based on libpq's functionality[^1].

[^1]: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-MULTIPLE-HOSTS

(cherry picked from commit 4e15a3969a)
2024-05-08 09:44:35 -04:00
Alex Vandiver
cf6ade4720 docs: Move warm-standby replication into postgresql.md.
(cherry picked from commit 73b760d829)
2024-05-08 09:44:35 -04:00
Anders Kaseorg
a6063b6344 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f72f156c0e)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
10b89db6a2 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f13e94d9ae)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
f058b6f946 send_email: Assert custom email payload is a str.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 8b47b5dc34)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
479f86f41f narrow: Add maybe_negate annotation needed by mypy 1.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0dff5f0a27)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
edeb17400b stripe: Add assertions to fix errors flagged by mypy 1.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 31c7b2bfd7)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
60423d3a75 stripe: Add missing stripe_customer_id assertions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit b195dc5a89)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
b331d91344 stripe: Stringify invoice metadata.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7b2a4304fc)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
5e79eff437 stripe: Pass ids, not objects, to pagination API.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 385cd038b0)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
c67f08b58a stripe: Use absent days_until_due rather than invalid None.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 228ade86be)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
1327db6e9c stripe: Use more specific type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e7a3d743d1)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
4b19baa7e8 partial: Replace returns plugin with an annotation.
The returns plugin hasn’t been updated for mypy ≥ 1.6.  This
annotation is more limited in that it only supports a fixed number of
positional arguments and no keyword arguments, but is good enough for
our purposes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit d32d4434dd)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
7228d2ed26 test_events: Switch verify_action to a context manager API.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 87f228fcb0)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
9662ce2d62 test_invite: Use context manager form of assertRaises.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 9efba89a43)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
6eb3118a9f webhooks: Use partial with positional arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6fef5c3e46)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
471282fd24 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0d1ed06762)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
e3b695f27b requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e3340774ce)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
c11af3aae6 sentry: Fix type error in add_context.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 71e1b3c91b)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
3045cfaf22 ruff: Fix S605 Starting a process with a shell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit eb622bae0a)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
95bdce69c1 ruff: Fix UP032 Use f-string instead of format call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a1a341f0ae)
2024-05-07 18:01:29 -07:00
Anders Kaseorg
b4f67e85b5 timeout: Rename to unsafe_timeout.
This timeout strategy using asynchronous exceptions has a number of
safety caveats (read the docstring!!) and should only be used in very
specific circumstances.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 72018cc26b)
2024-05-07 16:42:08 -04:00
Anders Kaseorg
607defeba7 test_helpers: Remove unused timeout_mock.
We don’t want to encourage more uses of zerver.lib.timeout.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 631c2f7d4c)
2024-05-07 16:42:08 -04:00
Anders Kaseorg
02b80db1dd streams: Remove inappropriate use of zerver.lib.timeout.
zerver.lib.timeout abuses asynchronous exceptions, so it’s only safe
to use on CPU computations with no side effects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit da979bc65c)
2024-05-07 16:42:08 -04:00
Anders Kaseorg
7384f61556 message_flags: Remove inappropriate use of zerver.lib.timeout.
zerver.lib.timeout abuses asynchronous exceptions, so it’s only safe
to use on CPU computations with no side effects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 95a1481f99)
2024-05-07 16:42:08 -04:00
Alex Vandiver
82e3b33f1e install: Check for a supported version of the OS in shell.
Testing for it in Python means that we have to worry about keeping the
`upgrade-zulip-stage-2` backwards-compatible with all versions of
Python which we could ever be upgrading from -- which is all of them.

Factor out the "supported operating systems" check, and share it
between upgrade and install codepaths.

(cherry picked from commit 7f4bc05f8e)
2024-05-07 13:26:48 -07:00
Anders Kaseorg
b36d7a0d01 ui_util: Fix HTML injection bug in replace_emoji_with_text.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 191345f9d6)
2024-05-07 12:55:13 -07:00
Anders Kaseorg
40208ee42b rendered_markdown: Fix HTML injection bug in update_elements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 4dc27216f4)
2024-05-07 12:55:13 -07:00
Anders Kaseorg
4c1a7b9938 copy_and_paste: Fix HTML injection bug in construct_copy_div.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e1029b59ed)
2024-05-07 12:55:13 -07:00
sujal shah
9dd8b2bb82 setting_ui: Fix sorting by "Invited by" column.
Previously, the sorting was broken due to
incorrect referencing of the property.
The code has been updated to use the "referrer_name"
property instead of "ref".

(cherry picked from commit 01e898252b)
2024-05-07 12:52:51 -07:00
Alex Vandiver
b9ff0c4bf0 django: Skip running config and database checks on process start.
The process of running Django's built-in database and config checks
can be very heavy-weight, potentially taking multiple seconds:

```
$ hyperfine './manage.py print_initial_password iago@zulip.com'  './manage.py print_initial_password iago@zulip.com --skip-checks'
Benchmark 1: ./manage.py print_initial_password iago@zulip.com
  Time (mean ± σ):      4.943 s ±  0.722 s    [User: 4.434 s, System: 0.311 s]
  Range (min … max):    4.415 s …  6.835 s    10 runs

Benchmark 2: ./manage.py print_initial_password iago@zulip.com --skip-checks
  Time (mean ± σ):      1.786 s ±  0.113 s    [User: 1.598 s, System: 0.162 s]
  Range (min … max):    1.576 s …  1.999 s    10 runs

Summary
  './manage.py print_initial_password iago@zulip.com --skip-checks' ran
    2.77 ± 0.44 times faster than './manage.py print_initial_password iago@zulip.com'
```

This extends the window during which nginx is forced to serve 502's to
clients.  f5f6a3789b added an explicit `manage.py check` during
server restarts, and fa77be6e6c added one during upgrades; as such,
we expect that any check failures will already have been caught when
performing a restart or upgrade, and there is no point in running them
on process startup.

(cherry picked from commit 48b925931e)
2024-05-07 12:52:51 -07:00
Pratik Chanda
783a2a3f27 left_sidebar: Fix DM row highlight linger on narrowing away.
Earlier, DM row remained highlighted when narrowed to different
part of left sidebar other than recent and inbox since on narrow
event, `handle_narrow_activate` wasn't properly handling unnarrow
events. It wasn't calling `update_private_messages` to update the
state of the DM list.

This commit fixes the behaviour by updating the state on narrowing
away from DM rows.

Fixes: zulip#27698.

Co-authored-by: Rinwaoluwa <peterodejobi9@gmail.com>
(cherry picked from commit 0708505acb)
2024-05-07 12:52:51 -07:00
Pratik Chanda
5a61c2dde0 views: Fix message_list.current not updating to undefined on narrow.
Earlier when users narrowed to recents or inbox, message_list.current
was updating to last dm row selected for first few filter checks in
'narrow_state' which in turn would set active_user_ids to last dm
row selected. This resulted in DM highlight lingering.
The narrow_state.filter is supposed to return undefined when narrowed
to recents or inbox.

This commit fixes the behaviour by setting 'current' undefined before
filter checks on narrow.

Fixes part of #27698.

(cherry picked from commit 8c89725fcc)
2024-05-07 12:52:51 -07:00
Alex Vandiver
ed27910bc1 import: Skip "active" check, and analytics, for crossrealm bots.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.

(cherry picked from commit a7ec7e924e)
2024-05-07 12:52:51 -07:00
Mateusz Mandera
da4ce971eb register_server: Check write access to secrets file when rotate_key.
Using --rotate-key without write access to the secrets file is currently
quite painful, since you end up rotating your registration's secret with
no local record of it; so effectively you lose your registration and
need help from support. We should just prevent this failure mode.

(cherry picked from commit 89a22fb076)
2024-05-07 12:52:51 -07:00
Alex Vandiver
ca7cb771fb release: Fix changelog output.
9bd340957f changed the headers; while we no longer store the
changelog text in the Github release object, we should still output
the changelog to STDOUT correctly.

(cherry picked from commit db4e093ad7)
2024-05-07 12:52:51 -07:00
Mateusz Mandera
45585acc4e push_notifs: Make push_notifications_enabled more resistant to flapping.
Fixes #28403

Uses redis to remember the last time push notifications were experienced
working. This needs to work across processes, so can't be done just in
memory.
As this is transient data that's fairly harmless to lose and thus
doesn't require the persistence benefits of the database, and we're
keeping a single "row", so don't need an entire new db table, we settle
on using redis instead of postgres. This is also consistent with how we
store other kinds of such transient data.

(cherry picked from commit 962ab13203)
2024-05-07 12:52:51 -07:00