Commit Graph

15164 Commits

Author SHA1 Message Date
Anders Kaseorg
9fa2026a66 tornado: Support sharding by user ID.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e5c26eeb86)
2022-11-16 21:20:13 -08:00
Anders Kaseorg
da1f0c9f00 tornado: Raise the same error for nonexistent and unauthorized queues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7222f3fe2b)
2022-11-16 21:19:21 -08:00
Anders Kaseorg
c5de31149c event_queue: Replace sequential queue IDs with UUIDs.
This prevents ID conflicts when resharding between multiple Tornado
shards.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 31b4be3b9b)
2022-11-16 21:19:21 -08:00
Anders Kaseorg
db0d6f81c0 sharding: Support Tornado sharding by regexes.
One should now be able to configure a regex by appending _regex to the
port number:

[tornado_sharding]
9802_regex = ^[l-p].*\.zulipchat\.com$

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 987ab741f9)
2022-11-16 21:19:21 -08:00
Anders Kaseorg
59edbfa411 scim: Check SCIM tokens using constant-time comparison.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 15:41:32 +00:00
Anders Kaseorg
62730cde13 openapi: Fix “contatining” typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c6f6df986c)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
d509cd0a0f markdown: Store ZulipMarkdown in members with the right type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 8230324068)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
07e1e47db3 fenced_code: Avoid sloppy AttributeError handler.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2bd81dd5c9)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
5983770d7b markdown: Rename our Markdown subclass to ZulipMarkdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 3cf91e9e45)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
86bfc7067c markdown: Remove Optional from zulip_rendering_result type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 97be895cf0)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
140fcee48c markdown: Add missing None check in InlineInterestingLinkProcessor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit d01c99d2ee)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
864649aa3e subdomains: Fix realm=None case for is_static_or_current_realm_url.
Fixes #22636.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit eb9957aa35)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
fe372a47c7 python: Mark dict parameters with defaults as read-only.
Found by semgrep 0.115 more accurately applying the rule added in
commit 0d6c771baf (#15349).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 47c5deeccd)
2022-11-15 15:28:42 -08:00
Mateusz Mandera
e730a25754 scim: Upgrade scim2-filter-parser to 0.4.0 to fix case-sensitivity.
userName lookups are supposed to be case-insensitive per the SCIM2 RFC
and this was fixed upstream in
https://github.com/15five/scim2-filter-parser/pull/31
and included in 0.4.0 release.

(cherry picked from commit 43107fcdc3)
2022-11-15 15:28:42 -08:00
Anders Kaseorg
d91e397d5c tornado: Fix Tornado CSRF check with X-Forwarded-Proto.
Since Django factors request.is_secure() into its CSRF check, we need
this to tell it to consider requests forwarded from nginx to Tornado
as secure.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit ce9ceb7f9f)
2022-11-15 10:13:00 -08:00
Mateusz Mandera
43606b6e39 do_deactivate_user: Use .on_commit around send_event calls.
The previous commit did this for revoking sessions. send_events should
be handled similarly too, to correctly handle calling do_deactivate_user
inside a transaction.

(cherry picked from commit 470c0458e6)
2022-11-14 16:06:25 -08:00
Mateusz Mandera
a3d9297c37 do_deactivate_user: Revoke sessions in transaction.on_commit().
Fixes #21709.

(cherry picked from commit a94b2572be)
2022-11-14 16:06:25 -08:00
Anders Kaseorg
7f6bd415f0 timeout: Correct thread id type passed to PyThreadState_SetAsyncExc.
This type changed in Python 3.7:
https://github.com/python/cpython/pull/781

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 89e4233b68)
2022-11-14 16:05:05 -08:00
Anders Kaseorg
c61799aef9 test_message_flags: Fix “accessibility” spelling.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 019c9527c1)
2022-11-14 17:11:39 -05:00
Mateusz Mandera
897476d4fb middleware: Log 5xx json_errors in JsonErrorHandler.
django.request logs responses with 5xx response codes (our configuration
of the logger prevents it from logging 4xx as well which it normally
does too). However, it does it without the traceback which results in
quite unhelpful log message that look like
"Bad Gateway:/api/v1/users/me/apns_device_token" - particularly
confusing when sent via email to server admins.

The solution here is to do the logging ourselves, using Django's
log_response() (which is meant for this purpose), and including the
traceback. Django tracks (via response._has_been_logged attribute) that
the response has already been logged, and knows to not duplicate that
action. See log_response() in django's codebase for these details.

Fixes #19596.
2022-10-17 17:45:44 -04:00
Mateusz Mandera
cdd01b9f8e send_analytics_to_remote_server: Log connection errors with traceback.
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name>  while trying to connect to push notification bouncer."
2022-10-17 17:45:44 -04:00
Anders Kaseorg
55c0a15f1c middleware: Pass unhandled API exceptions through to the test suite.
This results in more useful stack traces in failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-17 17:45:44 -04:00
Anders Kaseorg
b12a5be4a0 CVE-2022-36048: Rewrite only specific local links to relative.
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-24 13:07:42 -07:00
Anders Kaseorg
09fb71f95a requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 74f7b6e4fd)
2022-08-23 15:25:01 -07:00
Anders Kaseorg
64cbd4e7c7 retention: Inline move_rows query arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit b4b8691239)
2022-08-23 15:25:01 -07:00
Anders Kaseorg
170d805d9f upload: Replace boto3.Session with boto3.session.Session.
boto3-stubs seems to have dropped the former for some reason.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2508b579a6)
2022-08-23 15:25:01 -07:00
Anders Kaseorg
540060d389 settings: Add two_factor.plugins.phonenumber to INSTALLED_APPS.
I missed this in commit feff1d0411
(#22383) for upgrading to django-two-factor-auth 1.14.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2439914a50)
2022-08-22 22:41:49 -07:00
Anders Kaseorg
90c45bd230 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit feff1d0411)
2022-08-22 22:41:49 -07:00
Anders Kaseorg
1ae9922a86 test_tornado: Avoid deprecated AsyncHTTPTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6c79b8f2f1)
2022-08-22 22:41:49 -07:00
Anders Kaseorg
4a618ed973 db: Use cursor_factory psycopg2 option.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit b4cf9ad777)
2022-08-22 22:41:49 -07:00
Anders Kaseorg
6f08dbe11b decorator: Check Tornado secret with constant-time comparison.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-22 22:23:24 -07:00
Anders Kaseorg
84064e82c7 zilencer: Check remote server API keys with constant-time comparison.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-22 22:22:19 -07:00
Anders Kaseorg
c9f6830ba6 CVE-2022-31168: Fix authorization check for changing bot roles.
Due to an incorrect authorization check in Zulip Server 5.4 and
earlier, a member of an organization could craft an API call that
grants organization administrator privileges to one of their bots.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-21 20:09:02 -04:00
Mateusz Mandera
305c13faeb test_import_export: Test exclusion of private uploads from realm export. 2022-07-12 04:36:16 +00:00
Anders Kaseorg
c6a5903280 CVE-2022-31134: Exclude private attachments from realm exports.
Zulip Server 2.1.0 and above have a UI tool, accessible only to server
owners and server administrators, which provides a way to download a
“public data” export. While this export tool is only accessible to
administrators, in many configurations server administrators are not
expected to have access to private messages and private
streams. However, the “public data” export which administrators could
generate contained the attachment contents for all attachments, even
those from private messages and streams.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-12 04:36:14 +00:00
Anders Kaseorg
82adae451e email_notifications: Give absolute path to html2text.
Our uWSGI configuration doesn’t correctly activate our virtualenv.  We
should investigate that, but until we do, we need to invoke html2text
by an absolute path.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7c992422f0)
2022-07-07 17:08:22 -07:00
Anders Kaseorg
3f0919cc65 markdown: Parse included blocks in a new parser state.
This fixes inclusion of a multi-paragraph file into a list item.

Followup to commit dc33a0ae67 (#22315).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 869fe60689)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
e61ffc5bd7 integrations: Satisfy Python-Markdown’s archaic 4-space requirement.
Followup to commit dc33a0ae67 (#22315).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a2e1d61172)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
a303c27a16 tornado: Send request_started signal in Django thread.
Django’s ASGIHandler does this too and it seems like a good idea.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit a7e10ee47e)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
6b3399d7e6 markdown: Rewrite include plugin without markdown-include.
markdown-include is GPL licensed.

Also, rewrite it as a block processor, so that it works correctly
inside indented blocks.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit dc33a0ae67)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
ad692da6aa markdown: Rename preprocessor_priorities module to priorities.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7f0e11bd06)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
cc1244afa3 mattermost: Run html2text as a subprocess.
html2text is GPL licensed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit f3254bb558)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
b6c8acbf14 email_notifications: Run html2text as a subprocess.
html2text is GPL licensed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e10b7f2f3c)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
82155e15a5 tornado: Fix race condition on handler._request.
Commit 6fd1a558b7 (#21469) introduced an
await point where get_events_backend calls fetch_events in order to
switch threads.  This opened the possibility that, in the window
between the connect_handler call in fetch_events and the old location
of this assignment in get_events_backend, an event could arrive,
causing ClientDescriptor.add_event to crash on missing
handler._request.  Fix this by assigning handler._request earlier.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e112b619cc)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
c9e00e6391 test_tornado: Call process_event on first fetch_events return.
The 0.1 second delay was sometimes not long enough to guarantee we hit
the async response path, resulting in a nondeterministic coverage
failure.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 0430705d13)
2022-07-07 13:31:32 -07:00
Anders Kaseorg
51d0886f60 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 9ee636e920)
2022-07-07 13:31:32 -07:00
Alex Vandiver
214df3ea1f compilemessages: Use a consistent ordering for the languages list.
This makes the generated data more consistent.

(cherry picked from commit 160a8b9945)
2022-06-23 23:25:51 +00:00
Anders Kaseorg
b031537fe9 CVE-2022-31017: Fix edit event exposure in protected-history streams.
When editing an old message in a private stream with protected
history, the server would incorrectly send an API event including the
edited message to all of the stream’s current subscribers, including
those who should not have access to the old message. This API event is
ignored by official clients, so it could only be observed by a user
using a modified client or their browser’s developer tools.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-21 18:23:30 +00:00
Mateusz Mandera
b5e64dd1ef get_old_unclaimed_attachments: Add docstring explaining the logic.
(cherry picked from commit a671ae9749)
2022-06-20 11:13:24 -07:00
Mateusz Mandera
b1156e6d67 do_delete_old_unclaimed_attachments: Consider ArchivedAttachment rows.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.

(cherry picked from commit 09dc166b45)
2022-06-20 11:13:24 -07:00