The Slack text regexes match specific characters before and after the
formatted string to ensure that they only match at word boundaries.
However, because the capture groups consume each matching character,
including the characters used to determine word boundaries, two
formatted strings separated by a single matching character result in one
string not being matched, as the trailing whitespace is already
consumed and cannot also match as the leading pre-match whitespace for
the next character.
Switch to a look-ahead regex for the trailing word boundary
characters. This is zero-width, and as such the next match can still
also consume the same characters.
This also fixes Slack webhook integrations'
`test_message_with_complex_formatted_mentions` which was previously
expecting false output.
Fixes part of #30827.
This prep commit modifies the Slack text regex in
`slack_message_conversion.py` to use the `re.MULTILINE` flag capture
formatted strings that are adjacent to newline or end of line.
These kinds of strings are likely not uncommon to be generated by Slack
exporter itself and our Slack message block and attachment formatters
(`render_blocks` and `render_attachments`) also produces them.
This also fixes Slack webhook integration's
`test_message_with_complex_formatted_texts` which was previously
expecting false output.
Fixes part of #30827.
- Add a step to update the bot's API key if using a generic bot
for bidirectional bridging.
- Clarify setup options for how Slack channels are mapped.
- Edit for clarity and succinctness.
1. Fetching from the `/users.list` endpoint is supposed to use
pagination. Slack will return at most 1000 results in a single
request. This means that our Slack import system hasn't worked
properly for workspaces with more than 1000 users. Users after the
first 1000 would be considered by our tool as mirror dummies and thus
created with is_active=False,is_mirror_dummy=True.
Ref https://api.slack.com/methods/users.list
2. Workspaces with a lot of users, and therefore requiring the use of
paginated requests to fetch them all, might also get us to run into
Slack's rate limits, since we'll be doing repeating requests to the
endpoint.
Therefore, the API fetch needs to also handle rate limiting errors
correctly.
Per, https://api.slack.com/apis/rate-limits#headers, we can just read
the retry-after header from the rsponse and wait the indicated number
of seconds before repeating the requests. This is an easy approach to
implement, so that's what we go with here.
In Slack webhook integration setup, the only scenario where a message
payload contains a channel mention without the channels name is when the
user mentions a private channel.
e.g <#C07AVLQ3AUQ|>
This commit updates the string we use for such mention to "private Slack
channel" for better clarity.
Previously, the notification message only informed users that the
integration URL was registered with Slack's Event API. However, it might
be misleading to send an "integration is successful" message when this
happens because we didn't verify that the token has the required scopes
or if the Slack token is even added to the URL at all.
Now that the integration also verifies the `slack_app_token`'s scope,
it's now more appropriate to send a notification message like
`get_setup_webhook_message` to let the user know that the setup is
indeed successful.
This commit makes the Slack incoming webhook use the original
`convert_slack_formatting` and `convert_slack_workspace_mentions` from
`slack_message_conversion.py`. Previously those were not refactored out
into two smaller functions yet.
Fixes part of #30827.
Previously the `check_token_access` is called for every request we get
from Slack webhook, this may introduce significant latency.
This commit moves `check_token_access` to the same condition for when we
need to handle Slack challenge handshake so that we only do API token
check once per URL registered.
Additionally, we now check for the specific scopes that we need to run
the Slack webhook integration (SLACK_INTEGRATION_TOKEN_SCOPES).
Fixes part of #30827.
Previously, the `check_token_access` function had a hardcoded
`required_parameters` variable because it was only used in the Slack
data importer. This commit refactors `required_parameters` into a
function parameter, enabling the function to check a Slack token’s scope
for other purposes, such as Slack webhook integration.
Additionally, this commit changes the Slack API call in
`check_token_access` from `teams.info` to `api.test`. The endpoint is
better suited for this purpose since we're only checking a token’s scope
using the response header here.
The doc now shows instructions to setup the integration using
Slack's Events API instead of the legacy Outgoing webhook
service.
Co-authored-by: Alya Abbott <alya@zulip.com>
Co-authored-by: Lauryn Menard <lauryn@zulip.com>
A Slack fail condition occurs when we don't respond with HTTP 200 within
3 seconds after Slack calls our endpoint. If this happens, Slack will
retry sending the same payload. This is often triggered because we need
to perform callbacks when converting messages. To avoid sending the same
message multiple times, we block subsequent retry calls from Slack.
This commit returns early HTTP 200 response as soon as we get any retry
calls from Slack.
Part of #30465.
Payloads from Slack's Events API don't include human-readbale
information for a Slack username and for Slack channels.
This commit makes the Slack integration callback to the relevant
Slack API endpoints for that information.
Fixes#30074.
This commit adds 4 new payload fixtures to the Slack
integration for messages we don't process properly yet:
- Messages with code blocks.
- Messages with overlapping text formatting (e.g, bold
and italic).
- Messages with formatted mentions.
- Messages with quote blocks.
The follow-up plan to add support for these kind of
Slack messages is tracked in #30827.
Part of #30465.
This updates the Slack webhook integration to handle the Slack Events
API[1], while maintaining backwards compatibility with Slack's legacy
Outgoing Webhook service.
The Events API introduces the "challenge" handshake[2] to verify and
add a new webhook URL for them to call. This commit adds a handler for
the challenge handshake.
Additionally, this commit reformats incoming payloads using the Slack
text reformatter from `slack_message_conversion.py`. There is some
duplicative code here because of the difference in Slack export data
and Slack's webhook payload.
Part of #30465
[1]: https://api.slack.com/apis/events-api#using-events-api
[2]: https://api.slack.com/apis/events-api#handshake
Note: This doesn't change the occurences in the Zapier integration doc,
since they refer to pieces of the UI in Zapier, which may still be using
the "Stream" terminology.
Due to the channel_map_to_topics URL parameter in the Slack webhook,
it was not migrated to use the check_send_webhook_message.
By using check_send_webhook_message, any topic parameter in the
webhook URL will be prioritized over mapping Slack channels to
topics, e.g. when channel_map_to_topics is true. This is because
the default behaviour for incoming webhooks is to send a default
topic as a parameter to check_send_webhook_message in case there
is no topic specified in the URL.
In contrast, we can override the stream passed in the URL when
channel_map_to_topics is false by passing the Slack channel name
to check_send_webhook_message. The default behaviour for incoming
webhooks is to send a direct message if there is no specified
stream in the URL, so a default stream is not generally passed
to check_send_webhook_message.
Fixes#27601.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.
Currently, we use *topic as the local variable name for
topic names.
We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
Updates the Slack integration page to not describe adding a stream
or topic parameter to the URL query since that's not supported by
the current integration implementation.
Updates the Slack-compatible webhook integration page to have the
extra notes about the integration at the top of the page. Also,
removes the reference to a screenshot of the webhook since there
isn't one.
These webhooks have some URL query params that do not need additional
validation or parsing from JSON. So WebhookPaylaod is not applicable to
these webhooks.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
markdown-include is GPL licensed.
Also, rewrite it as a block processor, so that it works correctly
inside indented blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.
Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
Prior to this commit, we wrapped all incoming messages from Slack
in backticks. This led to weird formatting errors when an incom-
ing message from Slack contains backticks, to refer to a function
name, for instance.
This utilizes the generic `BaseNotes` we added for multipurpose
patching. With this migration as an example, we can further support
more types of notes to replace the monkey-patching approach we have used
throughout the codebase for type safety.
This concludes the HttpRequest migration to eliminate arbitrary
attributes (except private ones that are belong to django) attached
to the request object during runtime and migrated them to a
separate data structure dedicated for the purpose of adding
information (so called notes) to a HttpRequest.
JsonableError has two major benefits over json_error:
* It can be raised from anywhere in the codebase, rather than
being a return value, which is much more convenient for refactoring,
as one doesn't potentially need to change error handling style when
extracting a bit of view code to a function.
* It is guaranteed to contain the `code` property, which is helpful
for API consistency.
Various stragglers are not updated because JsonableError requires
subclassing in order to specify custom data or HTTP status codes.
Since FIXTURE_DIR_NAME is the name of the folder that contains the view
and tests modules of the webhook and another folder called "fixtures" that
store the fixtures, it is more appropriate to call it WEBHOOK_DIR_NAME,
especially when we want to refer to the view module using this variable.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.
Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.
We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used. So
for now I just made sure the docstrings of the two
sibling functions reference each other.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
There are only a handful of non-JSON webhooks that wouldn't
benefit from the notify_bot_owner_on_invalid_json feature.
Specifically, these are the webhooks where the third-party product
uses another format, whether it be HTML form-encoded, XML, or
something else.
Tweaked by tabbott to correc the list of excluded webhooks.
