Commit Graph

2630 Commits

Author SHA1 Message Date
theofficialvedantjoshi
194dfbc84d integrations: Add common framework for webhook signature verification.
Fixes: #19774
2025-04-09 15:05:57 -07:00
Tim Abbott
c95dd65d75 settings: Remove useless commented mypy type.
Legacy settings contained type "Dict" which were removed in zulip 9.0,
so this type was wrong, but it also serves no purpose.

(The non-commented types are checked in the development environment).
2025-04-03 18:20:45 -07:00
Niloth P
b2910aa05c integration-docs: Migrate PythonAPIIntegration docs.
To zulip/python-zulip-api, to keep them closer to their source code.

- Renamed the generate_zulip_bots_static_files to
generate_bots_integrations_static_files to accomodate the new function.
- Added a new function to
tools/setup/generate_bots_integrations_static_files to copy the
integration docs into static/generated/integrations.
- Updated integrations.py and computed_settings.py to use the new doc
paths.
- Deleted the affected integration docs.
- Updated the dependency URL.
2025-03-26 11:19:31 -07:00
Tim Abbott
37b7a32eb4 backends: Fix exception with password lengths above 72.
Apparently, while we set our own maximum password length of 100
characters, zxcvbn had a hardcoded maximum length of 72 characters,
and threw an exception if that was exceeded.

The fact that we're discovering this now would suggest that nobody has
previously attempted a password between 72 and 100 characters in
length.
2025-03-25 09:44:52 -07:00
Prakhar Pratyush
5f3896710f onboarding_steps: Add 'navigation_tour_video' for new users.
This commit adds a one-time modal to display navigation tour
video to new users.

Includes an `NAVIGATION_TOUR_VIDEO_URL` server-setting to specify
the video's URL. When set to None, the modal is not displayed.

Fixes #29304.
2025-03-13 14:38:16 -07:00
Vector73
b31024be47 saved_snippets: Add support for editing saved snippets.
Fixes #33708.
2025-03-13 10:58:36 -07:00
Alex Vandiver
721fd26442 send_email: Set the Date header according to local enqueue time.
Email clients tend to sort emails by the "Date" header, which is not
when the email was received -- emails can be arbitrarily delayed
during relaying.  Messages without a Date header (as all Zulip
messages previously) have one inserted by the first mailserver they
encounter.  As there are now multiple email-sending queues, we would
like the view of the database, as presented by the emails that are
sent out, to be consistent based on the Date header, which may not be
the same as when the client gets the email in their inbox.

Insert a Date header of when the Zulip system inserted the data into
the local queue, as that encodes when the full information was pulled
from the database.  This also opens the door to multiple workers
servicing the email_senders queue, to limit backlogging during large
notifications, without having to worry about inconsistent delivery
order between those two workers.

Messages which are sent synchronously via `send_email()` get a Date
header of when we attempt to send the message; this is, in practice,
no different from Django's default behaviour of doing so, but makes
the behaviour slightly more consistent.
2025-03-10 16:48:08 -07:00
opmkumar
c97fd1bca5 api: Rename edit typing endpoint to /messages/{message_id}/typing.
This is more consistent with how other URLs work in Zulip.

Replaces `/message_edit_typing` with `/messages/{message_id}/typing`.
The `message_id` parameter, previously passed in the request body,
is now included in the URL path.
2025-02-28 13:20:46 -08:00
Anders Kaseorg
ec3d187659 backends: Fix type errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-02-18 22:04:43 -08:00
Anders Kaseorg
302b961ec1 backends: Add missing @override declarations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-02-18 22:04:43 -08:00
Anders Kaseorg
4703aed86f backends: Rename SAMLAuthBackend.process_logout override.
Our method has an incompatible signature and cannot be validly typed
as an override.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-02-18 22:04:43 -08:00
Lauryn Menard
8561800676 video-calls: Add Zoom Serverto Server OAuth integration.
Adds a second Zoom integration that uses the Zoom Server to Server
OAuth app process. Only one of the two Zoom integrations can be
configured on a Zulip server.

Adds a cache for the access token from the Zoom server so that it
can be used by the server to create meetings for the approximate
duration of the access token

In the web-app compose box, if the user's delivery email does not
match a user on the configured Zoom account for the server to server
integration, then a compose box error banner will be shown when the
error response is received after clicking/selecting the video or
audio call button.

Also updates the production documentation for the both types of Zoom
integration apps (Server to Server and General). The General app
process for Zoom now requires unlisted apps to go through their
review process, which we now have documented.

Fixes #33117.
2025-02-13 16:35:43 -08:00
Alex Vandiver
f58c29b290 presence: Use the narrow user cache.
These two endpoints make up ~85% of requests to Zulip servers; since
presence is also a performance-critical endpoint, having them share
the same cache increases how hot it stays in memcached, in addition to
making the presence endpoint faster.

This comes at the very slightly cost of one extra field.  Checks
for the `is_bot` column are switched to the equivalent `bot_type`
check, since the columns are slightly duplicative, and we can get away
with only checking bot_type.
2025-02-13 12:40:53 -08:00
Alex Vandiver
58bf2a7935 tornado: Limit the width of the user queries, when they're needed.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them.  Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.

In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user.  However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each.  This leads to a query which is over 12k of text long, and
results in 319 columns.  In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.

We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry.  That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.

Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object.  The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
2025-02-13 12:40:53 -08:00
opmkumar
2a15da47d9 message_edit: Show typing indicator for message editing.
This commit adds typing indicators for message editing in stream
as well as in dm, if the send typing notification
for corresponding is enabled.

Based on earlier work in #28585.

Co-authored-by: Rohan Gudimetla <rohan.gudimetla07@gmail.com>

Fixes #25719.
2025-02-12 15:08:56 -08:00
Mateusz Mandera
a6f1916ab1 docs: Add endpoint for permalinks to some push notifs docs.
This will gives us the flexibility to edit our documentation, including
section names, without worrying about breaking links to docs hard-coded
in older releases / deployed self-hosted servers.
2025-02-12 09:50:52 -08:00
Aman Agrawal
2f5aea6604 message_summary: Add setting to put a monthly rate limit per user. 2025-02-06 12:03:24 -08:00
Aman Agrawal
7fb767c080 message_summary: Move cost per token configuration to settings. 2025-02-06 12:03:24 -08:00
Aman Agrawal
099bab986f settings: Remove unused setting.
Since TOPIC_SUMMARIZATION_MODEL is already defined in
dev_settings to a groq model, this is not used.
2025-02-06 12:03:24 -08:00
Simon Michalke
8de5de66fa avatar: Add option to disable gravatars per realm.
The functionality of gravatar can break anonymity if
the user has had a gravatar account set up previously.

This option allows specifically cloud instances to
have gravatar disabled selectively.
2025-01-30 18:46:02 -08:00
Mateusz Mandera
7390eb2ed0 zilencer: Rename registration takeover to registration transfer.
This is the final naming that we want, compared to the naming we merged
in #32399.
Includes renaming the API endpoints, but that should be fine as the
original PR was just merged and this isn't deployed anywhere.
2025-01-30 14:32:36 -08:00
Alex Vandiver
4dfb285b5b message_summary: Reorganize settings and secrets.
This allows arbitrary litellm-supported backends.
2025-01-29 14:55:35 -08:00
Mateusz Mandera
4e22a79e6a zilencer: Add flow for a server to reclaim its registration.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.

The flow is the following:
1. The host sends a POST request to
   /api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
   sends a POST to /remotes/server/register/verify_challenge endpoint of
   the bouncer.
4. Upon receiving the POST request, the bouncer GETS
   https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
   responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
   and thus regains its push notifications registration.

Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
2025-01-28 11:10:50 -08:00
Aman Agrawal
dc9422ec81 test_message_summary: Add a basic test. 2025-01-27 12:23:40 -08:00
Aman Agrawal
9f71f4578b portico: Replace "Find accounts" link with "Log in".
Fixes #32199

We only need a log in button since that will take users to
"/accounts/go" if we are on a non-realm specific URL.

"/accounts/go" already has link to go to "Find accounts" page.
2025-01-24 14:55:47 -08:00
Aman Agrawal
b047c4d322 message_summary: Add API endpoint to generate narrow summary.
This prototype API is disabled in production through settings not
configuring a default model.
2025-01-22 12:12:08 -08:00
Anders Kaseorg
f223251ffe requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2025-01-14 09:42:16 -08:00
Sayam Samal
75dabca291 showroom: Rename dev/design-testing -> showroom. 2025-01-13 17:46:15 -08:00
Sayam Samal
6c16adc9d8 showroom: Standardize "showroom" name for UI components testing pages.
This commit standardizes the naming convention to be used for the UI
components being used in the `/devtools/` storybook-like pages for
testing the UI components.
2025-01-13 17:46:15 -08:00
Sayam Samal
6dabfa02cb banners: Add new redesigned banner component.
This commit adds the redesigned banner component to the codebase along
with a storybook-style page in /devtools/banners to view and test the
redesigned banner component.

Any banner using the new redesigned styles, requires two classes,
- First, the base `banner` class which defines the structure and
    behavior of the banner.
- Second, a modifier class like `banner-info` which defines the styles
    for the particular banner type.

The navbar alert banners also have a custom class `navbar-alert-banner`
which is used to define the specific style and structure for these
banner types.

This commit also makes the `banner`, `action-button` and `icon-button`
components into handlebar templates to maintain consistency in their
usage in the codebase.
2025-01-13 17:46:13 -08:00
Maneesh Shukla
4362c8d3c9 password: Add password_max_length to register response.
This allows clients to render UI that disallows invalid passwords
before sending them to the server.
2025-01-13 11:47:34 -08:00
opmkumar
5b0c55fda3 realm: Add option to schedule data deletion while deactivating.
Introduce a feature to schedule realm data deletion time during realm
deactivation. This includes a server-level setting to configure the
minimum and maximum allowed deletion days.

Co-authored-by: Ujjawal Modi <umodi2003@gmail.com>
Co-authored-by: Lauryn Menard <lauryn@zulip.com>

Fixes #24677.
2024-12-18 23:06:12 -08:00
Mateusz Mandera
94cc413016 docs: Renaming AzureAD to Entra ID, part two.
This is a follow-up to #32693, which missed a bunch of places due to
varying ways of typing Azure AD. (Azure AD, Azure Active Directory etc.)

In the diff there's an apparent inconsistency with renaming, in some
places renaming to just "Microsoft Entra ID" and in others
"Microsoft Entra ID (AzureAD)".
The idea is to add (AzureAD) in ordinary documentation, as it can help
clarity since many people still think of this as AzureAD, while sticking
with brevity and cleanliness of just using the official name in
marketing focused material.
2024-12-15 08:22:30 -08:00
Sayam Samal
a8146916aa buttons: Add redesigned button styles.
This commit adds the redesigned button styles to the codebase along with
with a storybook-style page in `/devtools/buttons` to view and test the
redesigned button component.

The redesigned button component, uses the `action-button` class to
follow Zulip's no-abbreviation policy, and to avoid conflicts with the
pre-existing `button` and bootstrap `btn` classes.

A button using the new redesigned styles, required two classes,
  - First, the base `action-button` class which defines the structure
  and behavior of the button.
  - Second, a modifier class like `action-button-primary-neutral` which
  defines the styles for the particular action button type.
2024-12-04 11:08:33 -08:00
Aman Agrawal
35ffaff1f6 tusd: Allow user to upload files during preregistration.
Allow user to upload files during preregistration. This will be used
to upload import data.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2024-12-02 10:21:45 -08:00
Tim Abbott
30c54d023b settings: Write fewer values into template settings files.
This makes it easier for us to change the default values for these
settings, and also just helps keep settings files clear of values that
the relevant organization didn't choose themselves.

There's about a dozen more settings that could receive similar
treatment; I just went with the easiest ones.
2024-11-06 16:43:44 -08:00
PieterCK
7fc9fc32d1 avatars: Split email-based and user ID-based avatar endpoints.
Separate `avatars/<email_or_id>/medium?` endpoints into distinct
endpoints for email-based and user ID-based access. This change aligns
avatar endpoints with Zulip’s existing API path conventions (e.g., the
`users/` endpoint).
2024-11-05 09:06:17 -08:00
Tim Abbott
f54817d78d groups: Enable group-valued settings in production. 2024-10-31 12:04:51 -07:00
Anders Kaseorg
ec437fb770 settings: Replace deprecated STATICFILES_STORAGE.
https://docs.djangoproject.com/en/4.2/releases/4.2/#custom-file-storages
https://docs.djangoproject.com/en/5.1/releases/5.1/#features-removed-in-5-1

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-10-23 10:55:55 -07:00
Mateusz Mandera
77e7a2d30f users: Add API endpoint to update_user_backend by real email.
The old endpoint for updating a user worked only via user id. Now we add
a different entry to this functionality, fetching the user by
.delivery_email.

update_user_backend becomes the main function handling all the logic,
invoked by the two endpoints.
2024-10-22 16:36:38 -07:00
Anders Kaseorg
3ec58fd3d5 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-10-20 18:16:27 -07:00
Alex Vandiver
4b4b6c5ebe scheduled_messages: Remove separate logfile.
This is all captured in `events_deliver_scheduled_messages.log` (note
the leading `events_`) via supervisor.
2024-10-04 14:22:37 -07:00
Vector73
9e4e85e140 saved_snippets: Add backend for saved snippets.
Part of #31227.
2024-09-24 15:27:58 -07:00
Alex Vandiver
e125ad823d exports: Add a separate bucket for realm exports.
This allows finer-grained access control and auditing.  The links
generated also expire after one week, and the suggested configuration
is that the underlying data does as well.

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2024-09-20 15:43:49 -07:00
Prakhar Pratyush
86194efe7b realm_export: Add a new endpoint to fetch private data export consents.
This commit adds a new endpoint `export/realm/consents` to
fetch the consents of users for their private data exports.

Fixes part of #31201.
2024-09-20 09:10:32 -07:00
Alex Vandiver
24d110f063 settings: Increase default max file upload size to 100MB.
This also _lowers_ the default nginx client_max_body_size, since that
no longer caps the upload file size.
2024-09-19 11:37:29 -07:00
Alex Vandiver
818c30372f upload: Use tusd for resumable, larger uploads.
Currently, it handles two hook types: 'pre-create' (to verify that the
user is authenticated and the file size is within the limit) and
'pre-finish' (which creates an attachment row).

No secret is shared between Django and tusd for authentication of the
hooks endpoints, because none is necessary -- tusd forwards the
end-user's credentials, and the hook checks them like it would any
end-user request.  An end-user gaining access to the endpoint would be
able to do no more harm than via tusd or the normal file upload API.

Regardless, the previous commit has restricted access to the endpoint
at the nginx layer.

Co-authored-by: Brijmohan Siyag <brijsiyag@gmail.com>
2024-09-19 11:37:29 -07:00
Sahil Batra
5f3a8334be user_groups: Do not allow deleting user groups. 2024-09-18 13:41:13 -07:00
Alex Vandiver
5bc4d39c25 settings: Attempt to detect S3_REGION if left blank. 2024-09-17 12:51:30 -07:00
Tim Abbott
2585d209c0 user_groups: Rename misleadingly named view function. 2024-09-11 09:43:56 -07:00