Commit Graph

20381 Commits

Author SHA1 Message Date
Lauryn Menard
db0b50089b templates: Add rel-canonical link to documentation pages.
Updates templates/zerver/meta_tags.html to add a rel-canonical
link if REL_CANONICAL_LINK is in the template context dict.

We add REL_CANONICAL_LINK to the documentation context for the
help center, API and integrations documentation pages in all
cases.

For policies documentation pages, we add REL_CANONICAL_LINK to
the context only when settings.CORPORATE_ENABLED is true, so
that self-hosted servers' policies documentation do not have a
rel-canonical link set.
2025-10-06 09:25:59 -07:00
Alex Vandiver
279f3429d5 email_mirror_server: Drop privileges before attaching file handler.
(cherry picked from commit b39188f9cf)
2025-10-03 16:54:48 -07:00
Alex Vandiver
c0ce96a541 email_mirror_server: chown the logfile before dropping privileges.
This prevents the logger from failing due to an old, root-owned,
logfile already existing.

(cherry picked from commit 3064939ff7)
2025-10-03 16:54:48 -07:00
Lauryn Menard
44c6c4af4d integrations: Update GitLab doc to direct to create webhook.
GitLab has both integrations and webhooks, and our documentation
should direct users to the webhooks set up for integrating with
Zulip.

Adds a link to GitLab's create a webhook documentation as well.

(cherry picked from commit 0af56bb80b)
2025-10-03 16:54:48 -07:00
Sai
ddf582c6d4 slack: Fix bugs during import of thread messages.
Consider the following messages JSON (taken from real data, redacted):
```json
[
    {
        "subtype": "bot_message",
        "text": "",
        "attachments": [
            {
                "fallback": "Open Slack to cast your vote in this Simple Poll",
                "title": "Should we do a thing?",
                "id": 1,
                "color": "6ecadc",
                "fields": [
                    {
                        "title": "",
                        "value": "1️⃣ Yes 👍\n\n"",
                        "short": false
                    },
                    {
                        "title": "",
                        "value": "2️⃣ No 👎\n\n",
                        "short": false
                    },
                    {
                        "title": "",
                        "value": "3️⃣ Abstain :spock-hand:\n\n",
                        "short": false
                    }
                ],
                "mrkdwn_in": [
                    "fields"
                ]
            },
            {
                "callback_id": "12345678-1234-1234-1234-123456789abc",
                "fallback": "Open Slack to cast your vote in this Simple Poll",
                "id": 2,
                "color": "6ecadc",
                "actions": [
                    {
                        "id": "1",
                        "name": "vote",
                        "text": "1️⃣",
                        "type": "button",
                        "value": "1",
                        "style": ""
                    },
                    {
                        "id": "2",
                        "name": "vote",
                        "text": "2️⃣",
                        "type": "button",
                        "value": "2",
                        "style": ""
                    },
                    {
                        "id": "3",
                        "name": "vote",
                        "text": "3️⃣",
                        "type": "button",
                        "value": "3",
                        "style": ""
                    },
                    {
                        "id": "4",
                        "name": "delete-v2",
                        "text": "Delete Poll",
                        "type": "button",
                        "value": "",
                        "style": "danger",
                        "confirm": {
                            "text": "Are you sure you want to delete the Poll?",
                            "title": "Delete Poll?",
                            "ok_text": "Yes",
                            "dismiss_text": "No"
                        }
                    }
                ]
            },
            {
                "callback_id": "12345678-1234-1234-1234-123456789abc",
                "fallback": "Open Slack to cast your vote in this Simple Poll",
                "footer": "Simple Poll        <https:\/\/simplepoll.rocks\/dashboard\/redacted\/settings\/|Edit Settings>",
                "id": 3,
                "footer_icon": "https:\/\/simplepoll.rocks\/static\/main\/favicon.png",
                "color": "6ecadc"
            }
        ],
        "type": "message",
        "ts": "1234567890.123456",
        "bot_id": "B1ABCDEF1",
        "thread_ts": "1234567890.123456",
        "reply_count": 1,
        "reply_users_count": 1,
        "latest_reply": "1234567890.765432",
        "reply_users": [
            "U1ABC1234"
        ],
        "replies": [
            {
                "user": "U1ABC1234",
                "ts": "1234567890.765432"
            }
        ],
        "is_locked": false,
        "subscribed": false
    },
    {
        "user": "U1ABC1234",
        "type": "message",
        "ts": "1234567890.765432",
        "text": "Maybe do qux instead",
        "team": "T1AB23456",
        "user_team": "T1AB23456",
        "source_team": "T1AB23456",
        "user_profile": {
            "avatar_hash": "a123456789ab",
            "image_72": "https:\/\/avatars.slack-edge.com\/2017-01-01\/123456789abc_def123456789abcdef12_72.jpg",
            "first_name": "Alice",
            "real_name": "Alice Smith",
            "display_name": "a.smith",
            "team": "T1AB23456",
            "name": "a.smith",
            "is_restricted": false,
            "is_ultra_restricted": false
        },
        "thread_ts": "1234567890.123456",
        "blocks": [
            {
                "type": "rich_text",
                "block_id": "EoBdt",
                "elements": [
                    {
                        "type": "rich_text_section",
                        "elements": [
                            {
                                "type": "text",
                                "text": "Maybe do qux instead"
                            }
                        ]
                    }
                ]
            }
        ]
    }
]
```

533f177175/zerver/data_import/slack.py (L922-L924)
fails for the first message, because it lacks a 'user' key. It should fall back to the bot_id.

533f177175/zerver/data_import/slack.py (L925-L926)
fails for the second message, because it lacks a 'parent_user_id' key.
However, the thread root will have been processed earlier, so
memoization of thread parents fixes this issue. Because the original
message may not be in the same file, the memoization needs to be global
rather than in `channel_message_to_zerver_message`.

(cherry picked from commit eaa2ece4e1)
2025-10-03 16:54:48 -07:00
Alex Vandiver
1efc762dfb update_subscriber_counts: Fix calling without any flags.
(cherry picked from commit 8c37e9c7a6)
2025-09-15 15:12:59 +00:00
Alex Vandiver
23de6ae1dd update_subscriber_counts: Add tool to re-sync subscriber counts.
(cherry picked from commit f915498e48)
2025-09-11 11:10:52 -04:00
Alex Vandiver
c4d3547ddf settings: Derive RUNNING_INSIDE_TORNADO based on sys.argv.
This allows it to be used at application startup time.

(cherry picked from commit 169daa31a0)
2025-09-10 12:09:57 -07:00
Evy Kassirer
75917a9019 subscription_info: Improve subscriber fetch preformance.
Discussed here:
https://chat.zulip.org/#narrow/channel/141-kandra-ops/topic/partial.20subscribers.20query.20does.20users.20sequential.20scan/near/2255421

(cherry picked from commit 31505e90f7)
2025-09-10 12:09:57 -07:00
Lauryn Menard
cf5c77d8a5 test-subs: Make new channel name consistent in notification bot test.
Updates final check in test_notification_bot_dm_on_subscription to
use the same pattern for the new channel name, and adds a comment
to clarify what that final check is for since it doesn't look at
the Notification Bot DM or channel messages for that case.

(cherry picked from commit 1030f41884)
2025-09-10 12:09:57 -07:00
Lauryn Menard
b1b5b66bf6 test-subs: Fix comment in test_notification_bot_dm_on_subscription.
(cherry picked from commit 14581de0de)
2025-09-10 12:09:57 -07:00
Prakhar Pratyush
2d203f1743 api_docs: Fix 500 error on /api/register-remote-push-device.
`settings.ZULIP_SERVICES_URL` is used to construct curl
example for `/api/register-remote-push-device`.

On zulip cloud `settings.ZULIP_SERVICES_URL=None`, it resulted
in 500 error on visiting that page.

This commit fixes the bug by using "https://push.zulipchat.com"
as the default value if `settings.ZULIP_SERVICES_URL=None`.

The doc already mentions that the endpoint is meant to be used
by self-hosted servers, so the default value makes sense if
ZULIP_SERVICES_URL=None.

(cherry picked from commit 271aba001c)
2025-09-10 12:09:57 -07:00
Sahil Batra
1713694e3e openapi: Mark server_supported_permission_settings field as stable.
We have mostly decided on the structure of the group permission
setting object in server_supported_permission_settings field of
register response, so this commit removes the comment mentioning
it as unstable and adds the corresponding "Changes" entry.

(cherry picked from commit 6e17b05156)
2025-09-03 17:31:37 -07:00
Mateusz Mandera
2a97f6a88e import: Set Stream.subscriber_count for imported channels.
The import code wasn't setting subscriber_count at all, resulting in a
value of 0 when dealing with imports from 3rd party apps.
We run this unconditionally, also for imports from Zulip, since this
ensures we won't inherit incorrect values, if the data we're import has
them - e.g. due to some bugs that affected the server the data came from.

(cherry picked from commit 6ab30fcced)
2025-09-03 17:31:37 -07:00
Prakhar Pratyush
e7f3277931 api_docs: Deprecate the non-E2EE register push device endpoints.
Endpoints marked deprecated:
* /users/me/apns_device_token
* /users/me/android_gcm_reg_id

The older endpoints were for non-E2EE push notification case.

Fixes part of #35213.

(cherry picked from commit e8c9f4a811)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
08336b2f50 api_docs: Mark /mobile_push/test_notification as deprecated.
The older endpoint was for non-E2EE push notification case.

Fixes part of #35213.

(cherry picked from commit b157b14d76)
2025-08-28 17:34:20 -07:00
Kislay Verma
31ac5f8421 api_docs: Clarify description for send_new_subscription_messages.
The same subscription request can contain both existing channels
and newly created channels, so we clarify here that direct message
notifications are only sent when users are subscribed to existing
channels.

Also, updates this text to use "direct message" instead of "DM".

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
(cherry picked from commit 396f4d004a)
2025-08-28 17:34:20 -07:00
Kislay Verma
c068dcf74c test_subs: Add send_new_subscription_messages parameter false test case.
For the sake of completion, we add a test case ensuring that
the response does not contain `new_subscription_messages_sent` if
the parameter `send_new_subscription_messages` is `false`,
as mentioned in the API documentation.

(cherry picked from commit 9b7d6335c1)
2025-08-28 17:34:20 -07:00
Kislay Verma
eae9bb6535 add_subscriptions: Only exclude DMs and not new channel notifications.
When the number of new subscriptions crosses a threshold, we wish to
avoid sending DMs to the user as implemented in #31206. But there was
a bug causing announcement channel and new channel notification
messages to also not be sent.

This commit fixes that bug by ensuring that we only exclude direct
message notications.

(cherry picked from commit 18d9b686b7)
2025-08-28 17:34:20 -07:00
Kislay Verma
d0d54b82a5 test_subs: Add test for all notification bot messages.
In #31206, we changed `add_subscriptions_backend` to exclude the
notification bot DMs when subscribing users to a channel when the
number of subscriptions exceeds MAX_BULK_NEW_SUBSCRIPTION_MESSAGES.

This caused a bug where new channel announcement and creation
notification messages were not being sent. Here we add a test that
confirmes the current behavior, which will be corrected in a
subsequent commit.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
(cherry picked from commit 06d2ecb482)
2025-08-28 17:34:20 -07:00
Lauryn Menard
cf869a29a8 create-channel: Add test for invalid user ID in subscribers param.
(cherry picked from commit e5514d575d)
2025-08-28 17:34:20 -07:00
Lauryn Menard
5ba693446a api-docs: Revise create channel endpoint documentation.
(cherry picked from commit 8ee59e9277)
2025-08-28 17:34:20 -07:00
Lauryn Menard
183abc19be api-docs: Improve rendering of create channel python example.
(cherry picked from commit 910ca975ab)
2025-08-28 17:34:20 -07:00
Lauryn Menard
0a761cfaaa api-docs: Revise folder_id parameter for adding/updating channels.
Revises the description of the `folder_id` parameters in the
api/create-channel, api/subscribe and api/update-stream endpoints.

(cherry picked from commit 224499b58f)
2025-08-28 17:34:20 -07:00
Lauryn Menard
3965bb4d40 create-channel: Remove send_new_subscription_messages parameter.
Removes send_new_subscription_messages parameter from
`POST channel/create` endpoint as Notification Bot DMs
are never sent when users are subscribed as part of
creating a new channel.

Not considered a documentable API change, since the previous parameter
never had any effect, so this is effectively just a documentation and
error-handling bug fix, not an API change relevant to implementors.

(cherry picked from commit 699c0c6200)
2025-08-28 17:34:20 -07:00
Lauryn Menard
53aaef506c create-channel: Clarify flow of new channel and subscription notices.
Updates comments in send_user_subscribed_and_new_channel_notifications
so that it is clearer what notification each "if" block of the function
is generating.

Also, moves variables that are only used in the user subscribed DMs to
that "if" block.

(cherry picked from commit dda2c6e285)
2025-08-28 17:34:20 -07:00
Lauryn Menard
d5016b69ab create-channel: Rename send_messages_for_new_subscribers helper.
Because send_messages_for_new_subscribers also sends channel
notification messages about newly created channels, a clearer
name for is send_user_subscribed_and_new_channel_notifications.

(cherry picked from commit 8b1e536e6b)
2025-08-28 17:34:20 -07:00
Lauryn Menard
4264076e6d api-docs: Revise "description" parameter of new channel folder.
This parameter should be marked as required, but allowing an empty
string value. Updates the description of the parameter to be more
explicit of that fact.

(cherry picked from commit 6847c5bfdb)
2025-08-28 17:34:20 -07:00
Lauryn Menard
008de9883c api-docs: Use a more realistic example for a new channel folder ID.
(cherry picked from commit a497571be6)
2025-08-28 17:34:20 -07:00
Lauryn Menard
dbee13e75c api-docs: Revise descriptive text in channel folders endpoints.
Adds links to relevant help center documentation on channel
folders.

(cherry picked from commit 1d99e55bb4)
2025-08-28 17:34:20 -07:00
Lauryn Menard
28a37f82fa channel-folders: Create two channel folders in the dev database.
Adds a visible example to the PATCH /channel_folders endpoint for
reordering an organization's channel folders. So that the example
works in a normal dev environment, we also add a second channel
folder when populating the dev database.

(cherry picked from commit 4f4923ef2b)
2025-08-28 17:34:20 -07:00
Lauryn Menard
bedc06f6c1 api-docs: Mark "name" required for new channel folder.
Also, note that an empty string for the "name" parameter, when
creating or updating a channel folder, is not a valid value.

(cherry picked from commit 451e64b3e6)
2025-08-28 17:34:20 -07:00
Lauryn Menard
1aad055cd3 api-docs: Mark "order" as required for reodering channel folders.
(cherry picked from commit 4b0a9e484a)
2025-08-28 17:34:20 -07:00
Lauryn Menard
6a318bd2ef api-docs: Add admin only notes for channel folder endpoints.
(cherry picked from commit 4b7e0be1ab)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
738b3e9399 push_notifications: Log roundtrip time since worker decided to send.
Adds a log for the end-to-end latency from when the worker decided
to send push notifications & received a success response from bouncer.

Fixes part of #35368.

(cherry picked from commit c604ecb902)
2025-08-28 17:34:20 -07:00
Alex Vandiver
62bcab1a1a push_notifications: Revert parallel-device sending.
This reverts #26594, due to unexpected ConnectionClosed errors observed in Django.

(cherry picked from commit 331d210dac)
2025-08-28 17:34:20 -07:00
Mateusz Mandera
d70124be9a retention: Eliminate join with Recipient table when archiving DMs.
We can use the is_channel_message column instead of doing the join to
filter on recipient type.

(cherry picked from commit 4e3ec77a0f)
2025-08-28 17:34:20 -07:00
Mateusz Mandera
ddbe3fa17f delete_in_topic: Don't unnecessarily fetch .recipient.
This reverts commit f119c33789.
With 51cef01c29 merged, there is no need
to fetch .recipient here, as it won't be accessed by the delete messages
codepath.

(cherry picked from commit 5ef6852cfe)
2025-08-28 17:34:20 -07:00
Mateusz Mandera
8c9c39059d message: Use .is_channel_message column instead of is_stream_message().
This avoids a potential unnecessary message.recipient fetch required by
is_stream_message(). is_stream_message() methods precedes the addition
of the denormalized is_channel_message column and is now unnecessary.

In practice, we usually fetch Message objects with `.recipient` already,
so I don't expect any notable performance impact here - but it's still a
useful change to make.

(cherry picked from commit 51cef01c29)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
51711bd3d9 push_notifications: Parse push/e2ee/notify response using Pydantic.
In `send_push_notifications`, we were manually asserting types of the
fields returned by the `remotes/push/e2ee/notify` endpoint at runtime
and constructed the `response_data` data structure for further use.

Pydantic is the perfect tool for this task.

This commit updates the concerned code to use Pydantic's `TypeAdapter`.

No functional change.

Fixes part of #35368.

(cherry picked from commit 5a37942c61)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
04352cd738 push_notifications: Use SentPushNotificationResult dataclass.
Refactoring, no functional change.

This commit refactors `send_e2ee_push_notification_apple`
and `send_e2ee_push_notification_android` to return a
`SentPushNotificationResult` dataclass.

It's a cleaner protocol than passing a mutable data structure
`delete_device_ids` as argument and updating it within
functions.

Fixes part of #35368.

(cherry picked from commit 3548764e21)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
bb534fbab5 api_docs: Document /remotes/push/e2ee/register endpoint.
This commit documents the `/remotes/push/e2ee/register` endpoint.

We use auth_email="ZULIP_ORG_ID" and auth_api_key="ZULIP_ORG_KEY"
instead of "BOT_EMAIL_ADDRESS" and "BOT_API_KEY".

(cherry picked from commit f52533795b)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
5cd52a2a14 api_docs: Prep work to document zilencer endpoints.
Until now we were not documenting bouncer's REST API endpoints.

We plan to document the newly introduced "remotes/push/e2ee/register"
and "remotes/push/e2ee/notify" endpoints.

This commit does the prep work for documenting bouncer endpoints:
* mark the older endpoints related to sending non-E2EE push
  notifications as "intentionally_undocumented" - we'll remove
  them in future.
* the remaining endpoints are marked pending-to-document with
  helpful comments.

(cherry picked from commit 062a736097)
2025-08-28 17:34:20 -07:00
Prakhar Pratyush
7065f91699 render_curl_example: Remove unused 'auth_email' & 'auth_api_key'.
The 'function' variable in 'APICodeExamplesPreprocessor.generate_text'
matches the pattern of API_ENDPOINT_NAME = r"/[a-z_\-/-{}]+:[a-z]+"

It is always of the form 'endpoint_path:endpoint_method'.

There's no possibility of multiple ':'.

This commit removes the unused code block in `render_curl_example`,
which was trying to get 'auth_email' & 'auth_api_key' from 'function'.

(cherry picked from commit 9785cb4a09)
2025-08-28 17:34:20 -07:00
Alex Vandiver
770d86c288 export-search: Use background workers to download attachments.
(cherry picked from commit d15f2fb831)
2025-08-28 16:17:08 -07:00
Alex Vandiver
6042830850 export-search: Add explicit usermessage message_id range filters.
(cherry picked from commit 7714ca3ff9)
2025-08-28 16:17:08 -07:00
Alex Vandiver
570e32a771 export-search: Use chunked regions by id range.
(cherry picked from commit b9216c768d)
2025-08-28 16:17:08 -07:00
Alex Vandiver
f5097ce7ed export-search: Joins to usermessages may require adding a distinct.
(cherry picked from commit e703f65198)
2025-08-28 16:17:08 -07:00
Alex Vandiver
a9417eb4fe export-search: Limit to only the fields we need.
(cherry picked from commit e9c6856d02)
2025-08-28 16:17:08 -07:00
Alex Vandiver
270a19984a export-search: Prefetch message sender.
This avoids O(n) extra queries.

(cherry picked from commit ae090d9ff3)
2025-08-28 16:17:08 -07:00