Indexes on topic ("subject") are polluted by the existence of DMs,
which all have empty topics, and as such skew the statistics greatly.
This is particularly important given the new use of the empty topic
for the "general chat" function -- left as-is, the database makes bad
query plans because it believes the topic is vastly more common than
it actually is.
We move the old indexes to a new name with `_all`, and
recreate (concurrently) the same indexes but with a condition on
is_channel_message. These new indexes are unused at current, until
the query-building logic adds limits on is_channel_message; see the
following commit.
This commit separates the "resolve topic" permissions from the
topic editing permissions, through the introduction of setting -
"can_resolve_topics_group" which user group whose members
can resolve topics.
Fixes#21811
Removed `is_billing_admin` user property as it is no longer used since
billing permissions are now determined by `can_manage_billing_group`
realm setting.
Since this does impact the ability to access the channel's content, it
makes sense to permit changing subscriptions, just like other
permissions settings on the archived channel.
We do not fetch all the realm group settings using
select_related for register data now since it takes a
lot of time in planning phase. This commit updates
the code to fetch all the members and subgroups data
in user_groups_in_realm_serialized so that we do not
need to access each setting group individually.
user_groups_in_realm_serialized is updated to send the
required data accordingly.
Fixes#33656.
Zulip now supports empty string as a valid topic name.
For clients predating this feature, such messages appear
in "general chat" topic. Messages sent to "general chat" are
stored in the database as having a "" topic.
This commit adds a migration to rename the existing
"general chat" topic in the database to "".
Fixes parts of #32996.
This commit implements the backend of migrating the
`allow_edit_history` setting to
`message_edit_history_visibility_policy`.
This allows organizations, to have an intermediate setting to
view only the "Moves" history of the messages.
We still pass `realm_allow_edit_history` in `/register` response
though for older clients with its value being set depending on the
value of `realm_message_edit_history_visibility_policy`. We set
`realm_allow_edit_history` to `False` if the
`realm_message_edit_history_visibility_policy` is "None", and
`True` for "Moves only" or "All" message edit history.
Fixes part of #21398.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
We were not actually using anything but the IDs here, so it was a
bunch of wasted work to fetch these.
This essentially reverts f48e87cd3c. At
the time, something like that was required, because we needed to check
if the channel was deactivated before exposing it to the API, but more
recent reworking of the system to change the setting if the channel is
deactivated, rather than masking it in fetch_initial_state_data, means
we can do this cleanup.
Adds a second Zoom integration that uses the Zoom Server to Server
OAuth app process. Only one of the two Zoom integrations can be
configured on a Zulip server.
Adds a cache for the access token from the Zoom server so that it
can be used by the server to create meetings for the approximate
duration of the access token
In the web-app compose box, if the user's delivery email does not
match a user on the configured Zoom account for the server to server
integration, then a compose box error banner will be shown when the
error response is received after clicking/selecting the video or
audio call button.
Also updates the production documentation for the both types of Zoom
integration apps (Server to Server and General). The General app
process for Zoom now requires unlisted apps to go through their
review process, which we now have documented.
Fixes#33117.
These were likely mistakenly left off of 6098c2cebe, leading to
1847086044 likely not switching to `base_get_user_queryset` for this
function because the list differed.
These two endpoints make up ~85% of requests to Zulip servers; since
presence is also a performance-critical endpoint, having them share
the same cache increases how hot it stays in memcached, in addition to
making the presence endpoint faster.
This comes at the very slightly cost of one extra field. Checks
for the `is_bot` column are switched to the equivalent `bot_type`
check, since the columns are slightly duplicative, and we can get away
with only checking bot_type.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them. Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.
In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user. However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each. This leads to a query which is over 12k of text long, and
results in 319 columns. In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.
We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry. That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.
Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object. The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
Previously, realm and channel admins were not able to change settings
for a private channel they were not subscribed to. This commit changes
that.
We have only added the exception for can_add_subscribers_group
and not privacy settings.
We also need proper functions with proper terminologies for content
and metadata access.
Added `enable_guest_user_dm_warning` setting to decide whether
clients should show a warning when a user is composing to a guest
user in the organization.
Fixes#30078.
Co-authored-by: adnan-td <generaladnan139@gmail.com>
Refactors Realm.VIDEO_CHAT_PROIVDERS to have all the possible options
for video chat integrations, and use get_enabled_video_chat_providers
to compute the enabled options for the realm.
Prep for adding Zoom server to server video chat integration.
When a stream configured for any of these settings is deactivated,
the corresponding realm settings should be set to NULL:
* new_stream_announcements_stream
* signup_announcements_stream
* zulip_update_announcements_stream
* moderation_request_channel
Earlier, we were not updating those realm settings to NULL.
We had helper functions like 'get_new_stream_announcements_stream'
to return None if the configured stream was deactivated.
But it makes more sense to just set them to NULL in DB.
This commit also includes a migration to clear those fields
if the configured channels are deactivated.
This means that only ImageAttachment row needs to be fetched, and
removes the need to pass around an extra parameter. This
denormalization is safe, since in general Attachment rows are
read-only, so we are not concerned with drift between the Attachment
and ImageAttachment tables.
We cannot make content_type non-null, since while the both the
`content_type` column in Attachment and populating that from requests
predates the ImageAttachment table, we have both backfilled
ImageAttachment rows to consider, and imports may also leave files
with no `content_type`. Any backfill of currently-null `content_type`
values will thus need to update both tables.
This change fixes a race condition when importing. ImageAttachment
rows are imported before rendering Messages, which are both before
importing Attachment rows; if the thumbnailing finished after the
Message was imported but before Attachment rows were imported, then
the re-rendering step would not know the image's content-type.
Removed `bot_creation_policy` property, as the permission to create
bot users in the organization is now controlled by two new realm settings,
`can_create_bots_group` and `can_create_write_only_bots_group`
Added `can_create_bots_group` setting which controls who can
create any type of bots in the organization.
Added `can_create_write_only_bots_group` setting which controls
who can create incoming webhooks in the organization in additon
to those who are in `can_create_bots_group`.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.
The flow is the following:
1. The host sends a POST request to
/api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
sends a POST to /remotes/server/register/verify_challenge endpoint of
the bouncer.
4. Upon receiving the POST request, the bouncer GETS
https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
and thus regains its push notifications registration.
Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
Rename `can_subscribe_others_to_all_streams` to
`can_subscribe_others_to_all_accessible_streams` so it's clear that we
are not attempting to check basic access in this function.
The function to check relevant permissions does so for multiple streams
at once to save us database query counts. Doing it one by one for every
stream would become very expensive.
We've also added `insufficient_permission_streams` to the filter
functions return type for streams for which the current user does not
have permission to subscribe other users.
We remove `invite_to_stream_policy` from the backend wherever applicable
except deleting the field. We have just ported the existing behaviour of
`invite_to_stream_policy` to `can_add_subscribers_group` except one
change. We have added an explicit exception for admins to have this
permission whether they are part of this group or not. The reason for
this is we are adding `stream.can_add_susbcribers_group` in the future
which will grant all admins permission to subscribe other users to a
channel given they have access to a channel. So it makes sense that we
add this exception to the realm level property also.
See https://chat.zulip.org/#narrow/channel/101-design/topic/Can.20subscribe.20other.20users.20on.20user.20profile/near/2039825
This commit sets the default for user_list_style to
USER_LIST_STYLE_WITH_AVATAR so that the organization
level defaults for new users of this setting is set
to show avatar in right sidebar for new orgs.
This commit is a part of the work to support empty string
as a topic name.
Previously, empty string was not a valid topic name.
Adds a `empty_topic_name` client capability to allow client
to specify whether it supports empty string as a topic name.
Adds backward compatibility for:
- `subject` field in the `message` event type
