Files
zulip/zproject/dev_urls.py
Zixuan James Li a081428ad2 user_groups: Make locks required for updating user group memberships.
**Background**

User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.

This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.

**Solution**

To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.

The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.

Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:

1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.

For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.

**Error handling**

We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.

**Notes**

An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.

User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.

**Testing**

We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).

To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.

The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).

**Security notes**

get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.

We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
2023-08-24 17:21:08 -07:00

137 lines
4.9 KiB
Python

import os
from urllib.parse import urlsplit
from django.conf import settings
from django.conf.urls.static import static
from django.contrib.staticfiles.views import serve as staticfiles_serve
from django.http.request import HttpRequest
from django.http.response import FileResponse
from django.urls import path
from django.views.generic import TemplateView
from django.views.static import serve
from zerver.lib.rest import rest_path
from zerver.views.auth import config_error, login_page
from zerver.views.development.cache import remove_caches
from zerver.views.development.camo import handle_camo_url
from zerver.views.development.dev_login import (
api_dev_fetch_api_key,
api_dev_list_users,
dev_direct_login,
)
from zerver.views.development.email_log import clear_emails, email_page, generate_all_emails
from zerver.views.development.integrations import (
check_send_webhook_fixture_message,
dev_panel,
get_fixtures,
send_all_webhook_fixture_messages,
)
from zerver.views.development.registration import (
confirmation_key,
register_demo_development_realm,
register_development_realm,
register_development_user,
)
from zerver.views.development.user_groups import dev_update_subgroups
# These URLs are available only in the development environment
use_prod_static = not settings.DEBUG
urls = [
# Serve useful development environment resources (docs, coverage reports, etc.)
path(
"coverage/<path:path>",
serve,
{"document_root": os.path.join(settings.DEPLOY_ROOT, "var/coverage"), "show_indexes": True},
),
path(
"node-coverage/<path:path>",
serve,
{
"document_root": os.path.join(settings.DEPLOY_ROOT, "var/node-coverage/lcov-report"),
"show_indexes": True,
},
),
path(
"docs/<path:path>",
serve,
{"document_root": os.path.join(settings.DEPLOY_ROOT, "docs/_build/html")},
),
# The special no-password login endpoint for development
path(
"devlogin/",
login_page,
{"template_name": "zerver/development/dev_login.html"},
name="login_page",
),
# Page for testing email templates
path("emails/", email_page),
path("emails/generate/", generate_all_emails),
path("emails/clear/", clear_emails),
# Listing of useful URLs and various tools for development
path("devtools/", TemplateView.as_view(template_name="zerver/development/dev_tools.html")),
# Register new user and realm
path("devtools/register_user/", register_development_user, name="register_dev_user"),
path("devtools/register_realm/", register_development_realm, name="register_dev_realm"),
path(
"devtools/register_demo_realm/",
register_demo_development_realm,
name="register_demo_dev_realm",
),
# Have easy access for error pages
path("errors/404/", TemplateView.as_view(template_name="404.html")),
path("errors/5xx/", TemplateView.as_view(template_name="500.html")),
# Add a convenient way to generate webhook messages from fixtures.
path("devtools/integrations/", dev_panel),
path(
"devtools/integrations/check_send_webhook_fixture_message",
check_send_webhook_fixture_message,
),
path(
"devtools/integrations/send_all_webhook_fixture_messages", send_all_webhook_fixture_messages
),
path("devtools/integrations/<integration_name>/fixtures", get_fixtures),
path("config-error/<error_category_name>", config_error, name="config_error"),
path("config-error/remoteuser/<error_category_name>", config_error),
# Special endpoint to remove all the server-side caches.
path("flush_caches", remove_caches),
# Redirect camo URLs for development
path("external_content/<digest>/<received_url>", handle_camo_url),
]
testing_urls = [
rest_path(
"testing/user_groups/<int:user_group_id>/subgroups",
POST=(dev_update_subgroups, {"intentionally_undocumented"}),
),
]
urls += testing_urls
v1_api_mobile_patterns = [
# This is for the signing in through the devAuthBackEnd on mobile apps.
path("dev_fetch_api_key", api_dev_fetch_api_key),
# This is for fetching the emails of the admins and the users.
path("dev_list_users", api_dev_list_users),
]
# Serve static assets via the Django server
if use_prod_static:
urls += [
path("static/<path:path>", serve, {"document_root": settings.STATIC_ROOT}),
]
else: # nocoverage
def serve_static(request: HttpRequest, path: str) -> FileResponse:
response = staticfiles_serve(request, path)
response["Access-Control-Allow-Origin"] = "*"
return response
assert settings.STATIC_URL is not None
urls += static(urlsplit(settings.STATIC_URL).path, view=serve_static)
i18n_urls = [
path("accounts/login/local/", dev_direct_login, name="login-local"),
path("confirmation_key/", confirmation_key),
]
urls += i18n_urls