This commit is a part of the work to support empty string
as a topic name.
Previously, empty string was not a valid topic name.
Adds `allow_empty_topic_name` boolean parameter to `GET /messages`
and `GET /messages/{message_id}` endpoints to decide whether the
topic names in the fetched messages can be empty strings.
If False, the topic names in the fetched message will have the
value of `realm_empty_topic_display_name` field in `POST /register`
response replacing "" for channel messages.
This commit is a part of the work to support empty string
as a topic name.
Previously, empty string was not a valid topic name.
Adds a `empty_topic_name` client capability to allow client
to specify whether it supports empty string as a topic name.
Adds backward compatibility for:
- `subject` field in the `message` event type
Earlier, when a topic was deleted then UserTopic rows corresponding
to that topic were not deleted resulting in a bug where the topic
is listed in the '/#settings/topics' panel even after deletion.
This commit fixes the incorrect behavior to delete the concerned
UserTopic rows.
We need to take special care of the case when a topic is deleted
in private channel with protected history. We delete the UserTopic
records for exactly the users for whom after the topic deletion
action, they no longer have access to any messages in the topic.
This commit completely eliminates the home-grown
data_types.py system for checking types.
Now, in our unit tests and schema checkers, we use
pydantic types to check the shape of the event
objects that are either in our JSON fixtures or
that get captured by test_events.py.
The code in event_types.py was completely
code-generated from the legacy version of
event_schema.py, so there shouldn't be any
manual errors in how I ported the types over.
(The code generation was non-trivial, though,
as I had to account for changes from
snake_case to CamelCase and deal with some
other anomalies.)
This commit adds a custom rule to check python files
and raise lint error if they have transaction.atomic used
without any argument or savepoint=True is used.
It helps to avoid creating unintended savepoints in the future.
Previously, these were only shown for Mac OS users and replaced
the "Home", "End", "PgUp" and "PgDn" shortcuts. But as this really
depends on the keyboard the user is using (there are Mac keyboards
with the above keys), we instead show both options in our web app
and help center documentation on keyboard shortcuts.
The tooltip for the "Scroll to bottom" button will now always show
"End" for all users. Previously, it showed a "Fn" key option for
Mac users.
Fixes#31815.
All endpoints have been migrated to the typed_endpoint decorator,
therefore the has_request_variables decorator and the REQ function are
no longer needed and have been removed.
The commit creates a `server_message.ts` module which
will home our `zod` schema for the messages received
from server. We can then import the schema to parse
the data in needed modules at the API boundaries.
This will also help us write better tests.
As part of the popover menu redesign, this commit redesigns the user
card popover using the new "popover-menu" tippy theme and improves
accessibility by using appropriate ARIA attributes.
Adds a linter rule for markdown files so that new cases of "e.g."
without a comma wont be introduced to user-facing documentation.
Fixes the remaining cases of "e.g." in markdown files that are not
followed by a comma.
Also, makes small updates to `next_plan_forms_support.html`.
Removes unneeded "btn" and "btn-default" classes, and updates
the placeholder text for the input as not marked for translation.
Also, renames `ad_hoc_query.html` to `activity_table.html`,
`realm_summary_table.html` to `installation_activity_table.html`,
and `activity_details_template.html` to `activity.html`.
Removes the style attribute in the installation activity template
and uses a CSS class, "installation-activity-header", to center the
h3 and p tags instead. This removes an exception from the custom
lint check.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`
The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.
The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)
There is no support for configuring the settings through the UI yet.
The goal of typed_endpoint is to replicate most features supported by
has_request_variables, and to improve on top of it. There are some
unresolved issues that we don't plan to work on currently. For example,
typed_endpoint does not support ignored_parameters_supported for 400
responses, and it does not run validators on path-only arguments.
Unlike has_request_variables, typed_endpoint supports error handling by
processing validation errors from Pydantic.
Most features supported by has_request_variables are supported by
typed_endpoint in various ways.
To define a function, use a syntax like this with Annotated if there is
any metadata you want to associate with a parameter, do note that
parameters that are not keyword-only are ignored from the request:
```
@typed_endpoint
def view(
request: HttpRequest,
user_profile: UserProfile,
*,
foo: Annotated[int, ApiParamConfig(path_only=True)],
bar: Json[int],
other: Annotated[
Json[int],
ApiParamConfig(
whence="lorem",
documentation_status=NTENTIONALLY_UNDOCUMENTED
)
] = 10,
) -> HttpResponse:
....
```
There are also some shorthands for the commonly used annotated types,
which are encouraged when applicable for better readability and less
typing:
```
WebhookPayload = Annotated[Json[T], ApiParamConfig(argument_type_is_body=True)]
PathOnly = Annotated[T, ApiParamConfig(path_only=True)]
```
Then the view function above can be rewritten as:
```
@typed_endpoint
def view(
request: HttpRequest,
user_profile: UserProfile,
*,
foo: PathOnly[int],
bar: Json[int],
other: Annotated[
Json[int],
ApiParamConfig(
whence="lorem",
documentation_status=INTENTIONALLY_UNDOCUMENTED
)
] = 10,
) -> HttpResponse:
....
```
There are some intentional restrictions:
- A single parameter cannot have more than one ApiParamConfig
- Path-only parameters cannot have default values
- argument_type_is_body is incompatible with whence
- Arguments of name "request", "user_profile", "args", and "kwargs" and
etc. are ignored by typed_endpoint.
- positional-only arguments are not supported by typed_endpoint. Only
keyword-only parameters are expected to be parsed from the request.
- Pydantic's strict mode is always enabled, because we don't want to
coerce input parsed from JSON into other types unnecessarily.
- Using strict mode all the time also means that we should always use
Json[int] instead of int, because it is only possible for the request
to have data of type str, and a type annotation of int will always
reject such data.
typed_endpoint's handling of ignored_parameters_unsupported is mostly
identical to that of has_request_variables.
**Background**
User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.
This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.
**Solution**
To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.
The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.
Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:
1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.
For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.
**Error handling**
We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.
**Notes**
An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.
User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.
**Testing**
We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).
To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.
The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).
**Security notes**
get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.
We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
