Commit Graph

15055 Commits

Author SHA1 Message Date
Anders Kaseorg
bf5f006971 actions: Split out zerver.actions.user_groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 372c10f5f3)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
500bd04e11 actions: Split out zerver.actions.video_calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 90cae59ea6)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
d35bdd312f actions: Split out zerver.lib.recipient_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit c136eebb33)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
51d9bbca1e actions: Split out zerver.lib.user_counts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 703186c339)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
f3c9a5019b actions: Split out zerver.lib.user_message.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 05195c02c1)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
36fa5e0385 actions: Move part into zerver.models.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 7f00aa078e)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
faea77d03f actions: Split out zerver.lib.sounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 6a70f75587)
2022-04-15 10:08:19 -07:00
Anders Kaseorg
ea9ba8b24c actions: Add zerver/actions directory.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit dd8b1aaba6)
2022-04-15 10:08:19 -07:00
Alex Vandiver
fb03c3205e timeout: Add test coverage.
(cherry picked from commit e6e4b7b3ef)
2022-04-13 20:47:34 -07:00
Alex Vandiver
662396d2c5 timeout: Minor comment cleanups.
We remove the StackOverflow link because it is now so dated as to be
irrelevant -- it does not use `self.ident`, and cargo-cults the return
value of PyThreadState_SetAsyncExc.

(cherry picked from commit 04159a674c)
2022-04-13 20:47:34 -07:00
Alex Vandiver
4a5204a967 timeout: Warn if the thread did not exit.
As noted in the docstring for this function, the timeout is
best-effort only -- if the thread is blocked in a syscall, it will not
service the exception until it returns.  It can also choose to catch
and ignore the TimeoutExpired; in either case it will still be running
even after the `timeout()` function returns.

Raising a vare TimeoutExpired it still somewhat accurate, but obscures
that the backend thread may still be running along merrily.  Notice
such cases, and log a warning about them.

(cherry picked from commit 3af2c8d9a3)
2022-04-13 20:47:34 -07:00
Alex Vandiver
44a3cd8dd3 timeout: Re-raise from where the TimeoutExpired hit the thread.
Having just thrown an exception into the thread, it is often useful to
know _what_ was the slow code that we interrupted.  Raising a bare
TimeoutExpired here obscures that information, as any `exc_info` will
end there.

Examine the thread for any exception information, and use that to
re-raise.  This exception information is not guaranteed to exist -- if
the thread didn't respond to the exception in time, or caught it, for
instance.

(cherry picked from commit e714264756)
2022-04-13 20:47:34 -07:00
Alex Vandiver
efddda2609 timeout: Remove cargo-culted and impossible-to-reach code block.
The quote in question originates in python/cpython@b8b6d0c2c6, when
the code was added.  However, the code stopped having that comment,
and was no longer able to return anything but 1 or 0, starting in
python/cpython@4643c2fda1 -- Python 2.5.

Remove the block.

(cherry picked from commit 85eeaf5f18)
2022-04-13 20:47:34 -07:00
Alex Vandiver
ecfcc20351 send_email: Only warn if EMAIL_HOST_PASSWORD is unset, not "".
Some email hosts actually do want an empty password; since the default
is `None`, we should key on that, and not just being false-y.

(cherry picked from commit ee04f42897)
2022-04-13 20:46:38 -07:00
Mateusz Mandera
9c88f6c4ce scim: Temporarily stop running SCIM change operations atomically.
do_deactivate_user can't be run in an atomic block due to concerns
around revoking session in a transaction. See
62ba8e455d for more details.

Without the change in this commit, the process of deactivating a user
via SCIM is broken.
2022-04-13 16:02:04 -07:00
Aman Agrawal
3dc29fbc76 message_edit: Fix false sub/unsub bookend on using a near link.
We were not setting the `historical` flag correctly for
messages fetched via `json_fetch_raw_message` when used didn't
have any UserMessage.

Extended relevant tests to fetch check message flags too.
2022-04-04 12:34:43 -07:00
Alex Vandiver
d9e7feae0a migrations: Remove the possibly-duplicated emoji re-uploading.
In 85e531e377, we duplicated this block
of migration code to fix a bug, but moving it (aka deleting the
original copy) is a cleaner solution.

(cherry picked from commit 35e27aef4a)
2022-04-01 22:31:54 -07:00
Tim Abbott
4db1aa75ce migrations: Repeat part of migration 0376.
The blockquote explains the motivation for this change in detail.

Fixes #21608.

(cherry picked from commit 85e531e377)
2022-04-01 15:21:05 -07:00
Alex Vandiver
9bb31433f1 data_import: Fix bot email address de-duplication.
4815f6e28b tried to de-duplicate bot
email addresses, but instead caused duplicates to crash:

```
Traceback (most recent call last):
  File "./manage.py", line 157, in <module>
    execute_from_command_line(sys.argv)
  File "./manage.py", line 122, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/management/commands/convert_slack_data.py", line 59, in handle
    do_convert_data(path, output_dir, token, threads=num_threads)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 1320, in do_convert_data
    ) = slack_workspace_to_realm(
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 141, in slack_workspace_to_realm
    ) = users_to_zerver_userprofile(slack_data_dir, user_list, realm_id, int(NOW), domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 248, in users_to_zerver_userprofile
    email = get_user_email(user, domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 406, in get_user_email
    return SlackBotEmail.get_email(user["profile"], domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 85, in get_email
    email_prefix += cls.duplicate_email_count[email]
TypeError: can only concatenate str (not "int") to str
```

Fix the stringification, make it case-insensitive, append with a dash
for readability, and add tests for all of the above.
2022-03-31 11:29:53 -07:00
jai2201
20e6315705 streams: Fix empty description in stream change notifications.
Send 'No description' text when either the old or new stream
description is empty, to avoid odd looking notifications.
2022-03-29 01:39:55 -07:00
Tim Abbott
c789097184 streams: Fix empty description handling in new stream notifications.
"No description." is conveniniently an existing translated string, in
addition to being the right content for this setting.
2022-03-29 01:39:55 -07:00
jai2201
52cf557d2b openapi: Allow empty value of stream's description.
This is important because otherwise tests that change a stream
description to the empty string don't work.
2022-03-29 01:39:55 -07:00
Alya Abbott
f04fb51ecc help: Rewrite pages on inviting users and related permissions.
Fixes #21520.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2022-03-28 17:00:10 -07:00
Lauryn Menard
cbfe2707f4 help_docs: Clarify relative link text for Subscribed streams tab. 2022-03-28 15:13:39 -07:00
Alex Vandiver
ca506f71dc push_notifications: Increase severity of APNs ConnectionError.
This has only happened when our APNs certificate expired; logging at
the error level ensures that this shows up in Sentry.
2022-03-25 18:12:14 -07:00
Alya Abbott
2a1e08759b portico: Add Asciidoctor case study. 2022-03-25 17:51:15 -07:00
Lauryn Menard
8d242f3467 help_docs: Update edit history documentation for messages.
`disable-message-edit-history`: Remove text about EDITED label
and link to `view-a-messages-edit-history` instead.

`edit-or-delete-a-message`: Reformat 'EDITED' and '(deleted)'
to be bold instead of using backticks. Make link to view
edit history clearer.

Note that the text used in the `OpenGraphTest` in
`test_middleware.py` had to be updated for the changes to
`disable-message-edit-history`.
2022-03-25 17:03:53 -07:00
Anders Kaseorg
55882fb343 python: Use modern set comprehension syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-25 10:45:12 -07:00
Alex Vandiver
eae4643cb4 message_edit: Ignore duplicates when re-muting new topic name.
This avoids an error when a user has already muted the new topic name.
We do this by ignoring duplicates, rather than catching the
IntegrityError, because this edit happens in a transaction, and that
would abort the transaction.
2022-03-24 21:27:11 -07:00
Alex Vandiver
141b0c4cec muting: Handle the case of a race muting the same user twice. 2022-03-24 21:27:11 -07:00
Alex Vandiver
781107308d muting: Add a flag to allow user duplicate mutes to silently succeed. 2022-03-24 21:27:11 -07:00
Anders Kaseorg
83c90c53df db: Fix types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-24 19:16:52 -07:00
Tim Abbott
e3600900c0 migrations: Adjust related_name settings for ArchivedAttachment.
This is necessary for the migration 0386_fix_attachment_caches to run,
and likely makes more convenient any future parallel code interacting
with both Attachment and ArchivedAttachment.
2022-03-24 19:15:27 -07:00
Tim Abbott
f9f111f950 message_edit: Only move muted topic records when moving whole topics.
Our original implementation of moving muted topic records when a topic
is moved took a shortcut of treating all change_later usage as
something with intent to move the whole topic.

This works OK when moving the whole topic via this interface, but not
when moving a last off-topic message in the topic.

Address this by changing the rule to match the existing
moved_all_visible_messages variable.
2022-03-24 17:48:52 -07:00
Tim Abbott
b493224cda management: Document logout_all_users interaction with API keys.
Fixes part of #19397, but I'm leaving that open since we'll want to
add an option to rotate all API keys for the target users.
2022-03-24 14:58:31 -07:00
Lauryn Menard
182c00248d help_docs: Update various docs related to permissions.
Adds tab for web-public streams in documentation for setting
who can create new streams, as well as some text about why
this is limited to certain roles.

Removes list of actions that can be restricted to full members
due to maintainability concerns for that type of list in the
documentation and replaces it with a short descriptive text
explaining that many settings in Zulip support this restriction.
2022-03-24 11:37:35 -07:00
Aman Agrawal
b799ec32b0 upload: Allow rate limited access to spectators for uploaded files.
We allow spectators access to uploaded files in web public streams
but rate limit the daily requests to 1000 per file by default.
2022-03-24 10:50:00 -07:00
Tim Abbott
abea1f4598 migrations: Add migration to fix Attachment cache.
This migration needs to be run after the previous commit is deployed
to a given Zulip installation, to fix any stale values of
is_realm_public and is_web_public.
2022-03-24 10:50:00 -07:00
Tim Abbott
d149af936d models: Rework Attachment.is_*_public to be a cache.
Previously, Attachment.is_realm_public and its cousin,
Attachment.is_web_public, were properties that began as False and
transitioned to True only when a message containing a link to the
attachment was sent to the appropriate class of stream, or such a link
was added as part of editing a message.

This pattern meant that neither field was updated in situations where
the access permissions for a message changed:

* Moving the message to a different stream.
* Changing the permissions for a stream containing links to the message.

This correctness issue has limited security impact, because uploaded
files are secured both by a random URL and by these access checks.

To fix this, we reformulate these fields as a cache, with code paths
that change the permissions affecting an attachment responsible for
setting these values to the `None` (uncached) state. We prefer setting
this `None` state over computing the correct permissions, because the
correct post-edit permissions are a function of all messages
containing the attachment, and we don't want to be responsible for
fetching all of those messages in the edit code paths.
2022-03-24 10:49:59 -07:00
Alex Vandiver
4f93b4b6e4 uploads: Skip the outgoing proxy if S3_KEY is unset.
When the credentials are provided by dint of being run on an EC2
instance with an assigned Role, we must be able to fetch the instance
metadata from IMDS -- which is precisely the type of internal-IP
request that Smokescreen denies.

While botocore supports a `proxies` argument to the `Config` object,
this is not actually respected when making the IMDS queries; only the
environment variables are read from.  See
https://github.com/boto/botocore/issues/2644

As such, implement S3_SKIP_PROXY by monkey-patching the
`botocore.utils.should_bypass_proxies` function, to allow requests to
IMDS to be made without Smokescreen impeding them.

Fixes #20715.
2022-03-24 10:21:35 -07:00
Anders Kaseorg
24070c7ad8 beeminder: Accept float for payload["goal"]["pledge"].
The documentation at https://api.beeminder.com/#goal says this is
“number”; empirically, we do in fact get decimal points.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-22 21:08:48 -04:00
Alex Vandiver
1ac0035f8c markdown: Allow whitespace overlaps in topic linkifiers.
`prepare_linkifier_pattern`, as of db934be064, adds a match to the
end of the regex, of either the end of string, or a non-word character
-- this is in place of a negative look-ahead, which is no longer
possible in re2.  This causes the regex to consume trailing
whitespace, and thus not be able to match twice in succession with
`pattern.finditer` -- "#1234 #5678" fails to match because the space
is consumed by the first match of the regex.

Rather than use `pattern.finditer`, write own own version, which
rewinds over the non-word character consumed after the match, if any.
This allows the same "after" non-word character to also satisfy the
"before" of the next match.

Fixes #21502.
2022-03-22 15:40:03 -07:00
Alex Vandiver
abed174b12 uploads: Add an endpoint which forces a download.
This is most useful for images hosted in S3, which are otherwise
always displayed in the browser.
2022-03-22 15:05:02 -07:00
Lauryn Menard
b67288db67 help_docs: Extend options for getting links to Zulip content.
Extends the linking to Zulip documentation to cover:

- Getting URLs to messages via the message timestamp.
- Getting links to topics via the three-dots menu.
- Getting links to streams via right-click context menu.

Creates a new tabbed section for using the browser
address bar to copy URLs.
2022-03-22 12:38:09 -07:00
Tim Abbott
ceb9dd5854 migrations: Fix confusing output for migration 0383.
* Don't print the empty list for the vast majority of realms where
  this is a noop.

* Make output a little more clear that this isn't revoking all
  Confirmations, just those associated with deactivated users.
2022-03-21 23:25:06 -07:00
Aman Agrawal
7a25a80242 message_edit: Move muted status of the topic when moved between streams.
Add support for moving MutedTopic entries to another stream where
the user has access to shared history in both streams and
`propagate_mode != "change_one"`.

Also, we delete them the current user does not have access to the
target stream.
2022-03-21 12:42:39 -07:00
Tim Abbott
4da2f154e0 create_user: Improve --help output.
This includes documenting this as not the primary way to
programatically create users in Zulip.
2022-03-21 12:05:59 -07:00
Tim Abbott
aebd81c440 management: Remove unnecessary default=SUPPRESS logic. 2022-03-21 12:05:59 -07:00
Tim Abbott
e16043547b management: Add new create_realm management command.
This is intended for rare situations where one is creating multiple
realms via a script.

After all the preparatory refactoring in this last several commits, we
can now provide a working implementation of a create_realm management
command.
2022-03-21 12:05:59 -07:00
Tim Abbott
2be2393d3e create_user: Extract get_create_user_params.
We set nocoverage for the new function. Ideally it'd eventually get an
automated test, but we don't want to block this helpful refactoring on
doing so.
2022-03-21 12:05:59 -07:00