If the client has passed `simplified_presence_events` as true
in the `client_capabilities` parameter of the `POST /register`
request, then the server will send `presence` events with the
`presences` field, which has the user presence data in the
modern API format. When that client capability is false, the
`presence` event will be unchanged and sent with the user
presence data in the legacy format.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them. Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.
In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user. However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each. This leads to a query which is over 12k of text long, and
results in 319 columns. In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.
We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry. That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.
Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object. The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
This commit is a part of the work to support empty string
as a topic name.
Previously, empty string was not a valid topic name.
Adds a `empty_topic_name` client capability to allow client
to specify whether it supports empty string as a topic name.
Adds backward compatibility for:
- `subject` field in the `message` event type
For an incoming 1:1 DM, the recipient’s own recipient_id is useless to
the recipient themselves. Substitute the sender’s recipient_id, so the
recipient can use recipient_id as documented to uniquely represent the
set of 2 users in this conversation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`is_archived` field is added to the stream and types.
Include a new `archived_channeels` client capability, to allow clients
to access data on archived channels, without breaking
backwards-compatibility for existing clients that don't know how to
handle these.
Also, included `exclude_archived` parameter to `/get-streams`,
which defaults to `true` as basic clients may not be interested
in archived streams.
This swaps out url_format_string from all of our APIs and replaces it
with url_template. Note that the documentation changes in the following
commits will be squashed with this commit.
We change the "url_format" key to "url_template" for the
realm_linkifiers events in event_schema, along with updating
LinkifierDict. "url_template" is the name chosen to normalize
mixed usages of "url_format_string" and "url_format" throughout
the backend.
The markdown processor is updated to stop handling the format string
interpolation and delegate the task template expansion to the uri_template
library instead.
This change affects many test cases. We mostly just replace "%(name)s"
with "{name}", "url_format_string" with "url_template" to make sure that
they still pass. There are some test cases dedicated for testing "%"
escaping, which aren't relevant anymore and are subject to removal.
But for now we keep most of them as-is, and make sure that "%" is always
escaped since we do not use it for variable substitution any more.
Since url_format_string is not populated anymore, a migration is created
to remove this field entirely, and make url_template non-nullable since
we will always populate it. Note that it is possible to have
url_template being null after migration 0422 and before 0424, but
in practice, url_template will not be None after backfilling and the
backend now is always setting url_template.
With the removal of url_format_string, RealmFilter model will now be cleaned
with URL template checks, and the old checks for escapes are removed.
We also modified RealmFilter.clean to skip the validation when the
url_template is invalid. This avoids raising mulitple ValidationError's
when calling full_clean on a linkifier. But we might eventually want to
have a more centric approach to data validation instead of having
the same validation in both the clean method and the validator.
Fixes#23124.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
Instead of using `request.POST` to access the `data` parameter used
in the internal `notify_tornado` path, adds `has_request_variables`
decorator and accesses `data` as a `REQ` parameter.
Expands `test_tornado_endpoint` in `test_event_system.py` for
`data` being a required parameter for this path.
Commit 6fd1a558b7 (#21469) introduced an
await point where get_events_backend calls fetch_events in order to
switch threads. This opened the possibility that, in the window
between the connect_handler call in fetch_events and the old location
of this assignment in get_events_backend, an event could arrive,
causing ClientDescriptor.add_event to crash on missing
handler._request. Fix this by assigning handler._request earlier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.
Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
This utilizes the generic `BaseNotes` we added for multipurpose
patching. With this migration as an example, we can further support
more types of notes to replace the monkey-patching approach we have used
throughout the codebase for type safety.
This commit adds "user_settings_object" field to
client_capabilities which will be used to determine
if the client needs 'update_display_settings' and
'update_global_notifications' event.
This prevents a memory leak arising from Python’s inability to collect
a reference cycle from a WeakKeyDictionary value to its key
(https://bugs.python.org/issue44680).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This concludes the HttpRequest migration to eliminate arbitrary
attributes (except private ones that are belong to django) attached
to the request object during runtime and migrated them to a
separate data structure dedicated for the purpose of adding
information (so called notes) to a HttpRequest.
JsonableError has two major benefits over json_error:
* It can be raised from anywhere in the codebase, rather than
being a return value, which is much more convenient for refactoring,
as one doesn't potentially need to change error handling style when
extracting a bit of view code to a function.
* It is guaranteed to contain the `code` property, which is helpful
for API consistency.
Various stragglers are not updated because JsonableError requires
subclassing in order to specify custom data or HTTP status codes.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A bug in the implementation of the all_public_streams API feature
resulted in guest users being able to receive message traffic to public
streams that should have been only accessible to members of the
organization.
This makes it much more clear that this feature does JSON encoding,
which previously was only indicated in the documentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
