When flushing caches, we want to ensure that even processes which may
have a wrong cache-key-prefix know to fetch the latest data from the
database. This is complicated by the cache-key-prefixes being stored
on disk, and thus checking that every cache delete is not sufficiently
performant.
We store the list of cache-key-prefixes in the cache, itself, with no
prefix. This cache is updated when a new cache-key is written, and is
also allowed to lapse after 24 hours. Updating this global cache
entry on new prefix creation ensures that even a
not-yet-restarted-into deployment will have its caches appropriately
purged if changes are made to the underlying data.
However, this both adds a cache-get, as well as multiplies the size of
all cache clears; for large bulk clears (e.g. for stream renames,
which clear the cache for all message-ids in them) this may prove
untenable.
When these exceptions are thrown from the request-to-bouncer functions
inside of manage.py register_server/update_analytics_counts, they
shouldn't be silenced, merely calling maybe_mark_pushes_disabled in the
background.
This results in the occurrence of the error not being shown to the
user. Failure to upload analytics data when running these commands
should result in a loud, obvious error.
Failure of running register_server before this change:
```
./manage.py register_server
This command registers your server for the Mobile Push Notifications Service.
Doing so will share basic metadata with the service's maintainers:
* This server's configured hostname: zulipdev.com:9991
* This server's configured contact email address: desdemona+admin@zulip.com
* Metadata about each organization hosted by the server; see:
<https://zulip.com/doc-permalinks/basic-metadata>
Use of this service is governed by the Zulip Terms of Service:
<https://zulip.com/policies/terms>
Do you want to agree to the Zulip Terms of Service and proceed? [Y/n]
Mobile Push Notification Service registration successfully updated!
```
The occurrence of the error is not revealed to the user. Same concern
applies to the update_analytics_counts command.
After this change:
```
./manage.py register_server
This command registers your server for the Mobile Push Notifications Service.
Doing so will share basic metadata with the service's maintainers:
<...>
Do you want to agree to the Zulip Terms of Service and proceed? [Y/n]
Traceback (most recent call last):
File "/srv/zulip/./manage.py", line 150, in <module>
execute_from_command_line(sys.argv)
File "/srv/zulip/./manage.py", line 115, in execute_from_command_line
utility.execute()
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/zulip/zerver/lib/management.py", line 97, in execute
super().execute(*args, **options)
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
output = self.handle(*args, **options)
File "/srv/zulip/zerver/management/commands/register_server.py", line 137, in handle
send_server_data_to_push_bouncer(consider_usage_statistics=False, raise_on_error=True)
File "/srv/zulip/zerver/lib/remote_server.py", line 453, in send_server_data_to_push_bouncer
response = send_to_push_bouncer(
File "/srv/zulip/zerver/lib/remote_server.py", line 233, in send_to_push_bouncer
raise JsonableError(msg)
zerver.lib.exceptions.JsonableError: Duplicate registration detected.
```
The `lock=true` parameter fails, since it is not being run inside of a
transaction. We swap to not using locking, since there is no
correctness requirement for a lock here.
This is simply too time-intensive, at scale, to perform during the
critical period of an upgrade. The most performance-critical
codepaths (Tornado and presence) have moved to the narrow user cache,
which we continue to pre-fill.
This commit updates code to optimize computing users who have
metadata access via permission groups so that we do not have
to do DB query for each stream to get recursive members for
the groups having permissions.
Adds a second Zoom integration that uses the Zoom Server to Server
OAuth app process. Only one of the two Zoom integrations can be
configured on a Zulip server.
Adds a cache for the access token from the Zoom server so that it
can be used by the server to create meetings for the approximate
duration of the access token
In the web-app compose box, if the user's delivery email does not
match a user on the configured Zoom account for the server to server
integration, then a compose box error banner will be shown when the
error response is received after clicking/selecting the video or
audio call button.
Also updates the production documentation for the both types of Zoom
integration apps (Server to Server and General). The General app
process for Zoom now requires unlisted apps to go through their
review process, which we now have documented.
Fixes#33117.
This implied by its name that users could have more than one key.
They cannot, currently; make the code clearer by switching to the
explicit column access.
Only prefetching `realm` and `bot_owner` means that the cache values
filled by this process differ from the cache values that would be
stored on a cache miss -- leading to extra database queries for those
objects.
Use the same `base_get_user_queryset()` to ensure we get the same
related prefetches.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them. Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.
In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user. However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each. This leads to a query which is over 12k of text long, and
results in 319 columns. In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.
We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry. That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.
Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object. The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
1ea2f188ce mistakenly introduced a `_rate_limit` member of the
request, which was dutifully transcribed in 3f9a5e1e17. However,
`_rate_limit` was never read from, nor written to -- `_ratelimit` (with
no middle `_`) was the dict that contained rate-limiting data. This
`_ratelimit` dict was later renamed to the `_ratelimits_applied` list,
in e86cfbdbd7, which became the `ratelimits_applied` request note
field in 03693cd27e.
Remove the entirely unused `rate_limit` note, and properly copy the
`ratelimits_applied` data into the new request.
Now that we introduced an URL for serving permalinks redirecting to
docs in #33444, the docs_url mechanism is no longer needed, as we can
have a URL that's safe to hard-code in register_server.py.
The HostnameAlreadyInUseBouncerError.docs_url has been merged in main
briefly enough, that this should be safe to remove.
This commit adds typing indicators for message editing in stream
as well as in dm, if the send typing notification
for corresponding is enabled.
Based on earlier work in #28585.
Co-authored-by: Rohan Gudimetla <rohan.gudimetla07@gmail.com>
Fixes#25719.
Previously, when description for a channel -- either during its
creating or when we change its description contained a topic
permalink (through #-mention), then it was not rendered. This
is because of lack of authorization to access the channel.
This is fixed by passing the acting_user through the methods
which update or add the description, so that permissions
of the acting_user could be used to determine whether to
render the #-mention in stream description or not.
Previously when system bots used to `#-mention` a private
channel's topics in cases like moving messages of a private
channel, then the #-mentions were not rendered by the
markdown. This was because the system bots did not have
authorization to mention these channels.
This is fixed by passing down an `acting_user` parameter
in code paths involving sending these move message
notifications so that permission of acting_user to mention
the topic is verified for rendering the markdown, rather
than that of the system bot.
Previously, we needed to pass the group to the function, which sometimes
meant having 1 extra query to fetch the user group when we just needed
the group id for this function.
We were not able to use
get_user_ids_with_metadata_access_via_permission_groups in the function
in question due to a cyclic dependency to `lib/streams.py`.
Fixes https://chat.zulip.org/#narrow/channel/101-design/topic/permissions.20for.20admin.20to.20unsubscribe.20others/near/2060197
Non realm admin users were not able to view private channels they were
an administrator of but not subscribed to it. This commit changes that.
We also made changes for those users to be able to see the subscribers
list.
The increase in query count in test_home and test_event_system can be
mitigated by only fetching recursive user group ids when needed within
the `validate_user_access_to_subscribers_helper` function. But that
would require refactoring that function to handle multiple streams and
subscriptions at once, along with changing how that function is used at
different places, which might be an exercise better left as a follow up.
We have optimised the code a little bit by not fetching the group ids in
case the current user is a realm admin.
We are fetching channel_admin_ids and users belonging to
can_add_subscribers_group directly in stream_subscription.py without
using the helper function
`get_user_ids_with_metadata_access_via_permission_groups`. This is due
to a cyclic dependency and we will move `bulk_get_subscriber_peer_info`
to another file in the next commit.
Users in `can_administer_channel_group` and `can_add_subscribers_group`
have access to private channel metadata. They should be notified of
relevant events.
We've only made relevant changes to lib/streams.py in this commit to
make the changes small and reviewable.
Previously, realm and channel admins were not able to change settings
for a private channel they were not subscribed to. This commit changes
that.
We have only added the exception for can_add_subscribers_group
and not privacy settings.
We also need proper functions with proper terminologies for content
and metadata access.
We've also converted the function to check for permission to unsubscribe
others to accept a list of streams instead of checking each stream one
by one.
Earlier, we were passing the whole subscription object to the function
in order to check if the user was subscribed or not. In the future
commits, we want to check that without fetching and passing the complete
subscription object and this commit will help us do that.
Instead of referring to users being added or removed from channels,
we now use subscribe or unsubscribed from channels.
Splits the article for adding and removing users from a channel
into separate articles: /help/subscribe-users-to-a-channel and
/help/unsubscribe-users-from-a-channel.
The URL redirects for the combined add/remove articles (for both
channel and stream terminology) go to the subscribe users to a
channel article.
