Commit Graph

60163 Commits

Author SHA1 Message Date
Anders Kaseorg
73cc8efb05 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit df7ad517c5)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
c0aeb24692 python: Replace deprecated BeautifulSoup.findAll.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a1983d9052)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
69fb1828ca codespell: Fix typos caught by codespell.
“Brunch” wasn’t a typo, but, whatever.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 949e5004cb)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
1042fe86be slack_incoming: Fix slack_error_handler typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 778a45d41e)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
224e58a2ac backends: Fix type errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit ec3d187659)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
4455ddd513 backends: Add missing @override declarations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 302b961ec1)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
75a74eb407 backends: Rename SAMLAuthBackend.process_logout override.
Our method has an incompatible signature and cannot be validly typed
as an override.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 4703aed86f)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
be8142c9c2 ruff: Remove ISC001 ignore.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2aff7ce9b5)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
d28639c08e requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f223251ffe)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
cd485ef162 ruff: Partially reformat Python with Ruff 0.9 (2025 style).
These are the changes that are backwards compatible with the 2024
style.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 653b0b0436)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
680f318bc0 python: Simplify with str.removeprefix, str.removesuffix.
These are available in Python ≥ 3.9.
https://docs.python.org/3/library/stdtypes.html#str.removeprefix

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 91ade25ba3)
2025-03-17 14:10:29 -07:00
Anders Kaseorg
61e6723a51 tests: Fixes for Node.js upgrade.
These were missed in the previous commit, as they were from the
intermediate upgrade in 0bd9f4f674.
2025-03-13 15:49:07 +00:00
Alex Vandiver
9d0c065ab5 install-node: Upgrade Node.js from 20.15.1 to 22.14.0.
This is the equivalent of 3af4900891, as a better fix than
7c2212caa3 for the provisioning errors caused by key rotation in
https://github.com/nodejs/corepack/issues/612
2025-03-13 14:54:01 +00:00
Tim Abbott
50ecb8bb3f rocketchat: Harden parsing of custom emoji exports.
Apparently some exports have a weird `ObjectId("idstring")` typed
value here, that stringify to the actual string.

(cherry picked from commit b249830715)
2025-03-12 13:48:08 +00:00
Mateusz Mandera
b7eca1c55e slack: Fetch workspace users from /users.list in the correct manner.
1. Fetching from the `/users.list` endpoint is supposed to use
   pagination. Slack will return at most 1000 results in a single
   request. This means that our Slack import system hasn't worked
   properly for workspaces with more than 1000 users. Users after the
   first 1000 would be considered by our tool as mirror dummies and thus
   created with is_active=False,is_mirror_dummy=True.
   Ref https://api.slack.com/methods/users.list

2. Workspaces with a lot of users, and therefore requiring the use of
   paginated requests to fetch them all, might also get us to run into
   Slack's rate limits, since we'll be doing repeating requests to the
   endpoint.
   Therefore, the API fetch needs to also handle rate limiting errors
   correctly.
   Per, https://api.slack.com/apis/rate-limits#headers, we can just read
   the retry-after header from the rsponse and wait the indicated number
   of seconds before repeating the requests. This is an easy approach to
   implement, so that's what we go with here.

(cherry picked from commit f81e514d07)
2025-01-28 09:41:16 -08:00
Alex Vandiver
d8643becbc version: Update version after 9.4 release. 2025-01-16 17:58:45 +00:00
Alex Vandiver
b762cdde48 Release Zulip Server 9.4. 9.4 2025-01-16 17:22:09 +00:00
Mateusz Mandera
c6334a765b CVE-2024-56136: Don't leak information via "invalid subdomain" error.
The bug we're fixing here leaks information by returning an "invalid
subdomain" error when an attempt is made to log in to user@example.com
on a subdomain X when user@example.com does not exist on X, but does
on another subdomain Y.

This allows an attacker to determine that a certain email address has an
account on the server.

Instead, this should just return a regular authentication error.
2025-01-14 18:36:30 +00:00
Alex Vandiver
ff5512e5a9 i18n: Update translations from Transifex.
This only updates translations.json, since Transifex limitations mean
that we do not have updated Django translations.
2025-01-14 18:10:18 +00:00
PieterCK
59ac4612e0 integrations: Change Slack private channel mention string.
In Slack webhook integration setup, the only scenario where a message
payload contains a channel mention without the channels name is when the
user mentions a private channel.
  e.g <#C07AVLQ3AUQ|>

This commit updates the string we use for such mention to "private Slack
channel" for better clarity.

(cherry picked from commit f89881d843)
2025-01-09 09:14:15 -08:00
PieterCK
b60d71ea9c integrations: Improve Slack integrations setup notification.
Previously, the notification message only informed users that the
integration URL was registered with Slack's Event API. However, it might
be misleading to send an "integration is successful" message when this
happens because we didn't verify that the token has the required scopes
or if the Slack token is even added to the URL at all.

Now that the integration also verifies the `slack_app_token`'s scope,
it's now more appropriate to send a notification message like
`get_setup_webhook_message` to let the user know that the setup is
indeed successful.

(cherry picked from commit 059782bb9f)
2025-01-09 09:14:15 -08:00
PieterCK
14cf851f0e integrations: Clean up duplicate functions.
This commit makes the Slack incoming webhook use the original
`convert_slack_formatting` and `convert_slack_workspace_mentions` from
`slack_message_conversion.py`. Previously those were not refactored out
into two smaller functions yet.

Fixes part of #30827.

(cherry picked from commit c6fd0ba424)
2025-01-09 09:14:15 -08:00
PieterCK
32b68553ba integration-doc: Fix typo/wrong in Slack token scope list.
This commit deletes the duplicate `users:read` token scope from the list
of token scopes.

(cherry picked from commit a194d62093)
2025-01-09 09:14:15 -08:00
PieterCK
a0eb033731 integrations: Do check_token_access only initially.
Previously the `check_token_access` is called for every request we get
from Slack webhook, this may introduce significant latency.

This commit moves `check_token_access` to the same condition for when we
need to handle Slack challenge handshake so that we only do API token
check once per URL registered.

Additionally, we now check for the specific scopes that we need to run
the Slack webhook integration (SLACK_INTEGRATION_TOKEN_SCOPES).

Fixes part of #30827.

(cherry picked from commit f2599bf33d)
2025-01-09 09:14:15 -08:00
PieterCK
28c09a53a1 slack_import: Make check_token_access more flexible.
Previously, the `check_token_access` function had a hardcoded
`required_parameters` variable because it was only used in the Slack
data importer. This commit refactors `required_parameters` into a
function parameter, enabling the function to check a Slack token’s scope
for other purposes, such as Slack webhook integration.

Additionally, this commit changes the Slack API call in
`check_token_access` from `teams.info` to `api.test`. The endpoint is
better suited for this purpose since we're only checking a token’s scope
using the response header here.

(cherry picked from commit a746be807f)
2025-01-09 09:14:15 -08:00
PieterCK
ee8a1a3759 slack_import: Refactor a convert_slack_formatting.
This prep commit extracts the logic for converting Slack formatted text
into Zulip friendly markdown from `convert_to_zulip_markdown` into a new
helper function, `convert_slack_formatting`.

This is done to make reformating logic be reusable else where such as in
the Slack webhook or Slack incoming webhook.

(cherry picked from commit 6543dadd0f)
2025-01-09 09:14:15 -08:00
PieterCK
2d9c167ca3 slack_import: Refactor a convert_slack_workspace_mentions.
This prep commit extracts the logic for reformatting Slack mentions into
Zulip mentions from `convert_to_zulip_markdown` into a new helper
function, `convert_slack_workspace_mentions`.

This is done to make the reformatting logic be reusable else where such
as in the Slack webhook or Slack incoming webhook.

(cherry picked from commit 4b472611d0)
2025-01-09 09:14:15 -08:00
PieterCK
3e816ac12b slack_import: Reorder functions used in convert_to_zulip_markdown.
This prep commit moves `convert_to_zulip_markdown` and
`convert_markdown_syntax` so that they appear after the various
helper functions used in the main logic for converting Slack
messages to Zulip's formatting. This adjustment aligns with code
style conventions, improving readability.

The change also prepares the codebase for reusing Slack text
reformatting logic in the Slack incoming webhook.

(cherry picked from commit ff4d5daef4)
2025-01-09 09:14:15 -08:00
PieterCK
bdaa6b8299 integrations: Refactor slack_incoming webhook.
This commit refactors `render_attachment` and `render_block`
out of slack_incoming.py to promote reusability.

The primary motivation for this refactor is to add support for
converting integration bots messages in Slack exports, which could use
the same functions.

Part of #31311.

(cherry picked from commit 92437b4ab5)
2025-01-09 09:14:15 -08:00
Alex Vandiver
5d013c218c requirements: Require some swap space if installed on < 5GB RAM.
(cherry picked from commit 3eb501fa18)
2025-01-08 09:26:02 -08:00
Anders Kaseorg
4f84a04c37 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2709e7a636)
2025-01-07 09:31:11 -08:00
Anders Kaseorg
729d1f078b send_email: Parse emails in a way mypy 1.14 understands.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 63aaafb94a)
2025-01-07 09:31:11 -08:00
Anders Kaseorg
4431bd3018 scim: Add None check for email_new_value.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0bb0d3598b)
2025-01-07 09:31:11 -08:00
Anders Kaseorg
348a9a8843 ruff: Fix PLC0206 Extracting value from dictionary without calling .items().
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 19b8cde27f)
2025-01-07 09:31:11 -08:00
Anders Kaseorg
26f09aad08 mypy: Add types-defusedxml.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c48ef926e1)
2025-01-07 09:31:11 -08:00
Anders Kaseorg
2d3484cf60 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 532aee926c)
2025-01-07 09:31:11 -08:00
PieterCK
9a0ca119e3 slack-integration: Update Slack integration documentation.
The doc now shows instructions to setup the integration using
Slack's Events API instead of the legacy Outgoing webhook
service.

Co-authored-by: Alya Abbott <alya@zulip.com>
Co-authored-by: Lauryn Menard <lauryn@zulip.com>
(cherry picked from commit 82e0468071)
2024-12-18 17:09:43 -08:00
PieterCK
7ff5040b11 slack-integration: Block requests from Slack retries.
A Slack fail condition occurs when we don't respond with HTTP 200 within
3 seconds after Slack calls our endpoint. If this happens, Slack will
retry sending the same payload. This is often triggered because we need
to perform callbacks when converting messages. To avoid sending the same
message multiple times, we block subsequent retry calls from Slack.

This commit returns early HTTP 200 response as soon as we get any retry
calls from Slack.

Part of #30465.

(cherry picked from commit 42a22e6aaa)
2024-12-18 17:09:43 -08:00
PieterCK
ad88fe7005 slack-integration: Add callback function to Slack API.
Payloads from Slack's Events API don't include human-readbale
information for a Slack username and for Slack channels.

This commit makes the Slack integration callback to the relevant
Slack API endpoints for that information.

Fixes #30074.

(cherry picked from commit c54d90e5c7)
2024-12-18 17:09:43 -08:00
PieterCK
0905513841 slack-integration: Add test fixtures for extra Slack payloads.
This commit adds 4 new payload fixtures to the Slack
integration for messages we don't process properly yet:

- Messages with code blocks.
- Messages with overlapping text formatting (e.g, bold
  and italic).
- Messages with formatted mentions.
- Messages with quote blocks.

The follow-up plan to add support for these kind of
Slack messages is tracked in #30827.

Part of #30465.

(cherry picked from commit 1ebbe2fd99)
2024-12-18 17:09:43 -08:00
PieterCK
148f7cde6c slack-integration: Update Slack integration to handle Events API.
This updates the Slack webhook integration to handle the Slack Events
API[1], while maintaining backwards compatibility with Slack's legacy
Outgoing Webhook service.

The Events API introduces the "challenge" handshake[2] to verify and
add a new webhook URL for them to call. This commit adds a handler for
the challenge handshake.

Additionally, this commit reformats incoming payloads using the Slack
text reformatter from `slack_message_conversion.py`. There is some
duplicative code here because of the difference in Slack export data
and Slack's webhook payload.

Part of #30465

[1]: https://api.slack.com/apis/events-api#using-events-api
[2]: https://api.slack.com/apis/events-api#handshake

(cherry picked from commit f29312ce03)
2024-12-18 17:09:43 -08:00
Anders Kaseorg
5e3030b072 avatar: Fix missing function call. (#32667)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a5d7ecd7e7)
2024-12-18 17:09:43 -08:00
Anders Kaseorg
88fb4279e3 integrations: Properly pass data-categories as a JSON array.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 801e267e00)
2024-12-18 17:09:43 -08:00
Tim Abbott
c887c5d4e7 integrations: Fix jQuery .data mishandling translated category names.
Apparently `.data("category")` would be come back as a single-element
array for some French translations, rather than a string.

We fix this by using `.attr("data-category")` across this file, which
is our preferred coding pattern anyway.

(cherry picked from commit 77cac403ee)
2024-12-18 17:09:43 -08:00
PieterCK
d5b5388aa0 slack_data_import: Support converting integration bot users.
Currently, we're unable to convert messages from Slack's integration
bots because this message subtype doesn't come from a Slack "user", that
is they don't have a Slack user profile.

This is a preparatory change to support converting Slack's integration
bot messages. This commit artificially creates Slack user data from the
integration bot's "profile" so that we can create a corresponding Zulip
user for them.

Part of #31311.

(cherry picked from commit f988412394)
2024-12-18 17:09:43 -08:00
PieterCK
26d13fefd6 slack_data_import: Update how Slack user avatars are processed.
Previously, the Slack export converter can only process Slack's avatar
URL from Slack's "ca.slack-edge.com" server, which looks like this:

https://ca.slack-edge.com/T0CDRA6HM3P-U06NABE26M9-1173e04f818e-512

This commit adds support for converting any public downloadable image
URLs.

This is done to support importing Slack's integration bots and their
messages, which typically have PNG type file url:

https://avatars.slack-edge.com/2024-05-01/7057208497908_a4351f6deb91094eac4c_72.png
(cherry picked from commit 10946caa3d)
2024-12-18 17:09:43 -08:00
Alex Vandiver
7d4434b67f migrations: Update user and realm preferences of zh_Hant to zh_TW.
(cherry picked from commit 2250ae3f0d)
2024-12-12 10:59:35 -08:00
Alex Vandiver
1de2ec60a2 i18n: Merge zh_Hant into zh_TW.
(cherry-picked from commit 25069c8d47)
2024-12-12 10:59:35 -08:00
Tim Abbott
ed72d1a080 version: Update version after 9.3 release. 2024-11-22 17:16:35 -08:00
Tim Abbott
0968ef91e8 Release Zulip Server 9.3. 9.3 2024-11-22 17:10:02 -08:00