For Slack, Mattermost and RocketChat, set the default announcement
channels if the sane default channel is available in the import.
The next commit will add a mechanism to choose an alternative default if
the expected channel name is not present.
Addresses the major part of #34984.
This updates the topic name format for Slack threads to include a
snippet of the original message. Currently the format looks like this;
"{date} Slack thread {n}", which provides little to no context about the
thread.
Currently we also use the `thread_ts_str` key to keep track of Slack
threads we're converting, this key format makes it so that it has a
small chance of combining threads with the same timestamp under one
topic. This commit updates it to use a new key format, `thread_key`.
Fixes#27661.
This logic was fundamentally broken:
```
is_realm_imported_from_other_zulip_server = RealmAuditLog.objects.filter(
realm=realm, event_type=AuditLogEventType.REALM_EXPORTED
).exists()
if not is_realm_imported_from_other_zulip_server:
send_zulip_update_announcements_to_realm(
realm, skip_delay=False, realm_imported_from_other_product=True
)
```
Because the `REALM_EXPORTED` was only created after the export completed
- meaning it couldn't be included in the export data at all. Thus
considering exports to be "not from Zulip" incorrectly.
We get around this issue by explicitly including an import_source in the
realm dict in the export data from 3rd party apps. The importer can then
rely on this value to determine if it's dealing with a Zulip-originated
export or not.
Slackbot should be correctly marked as a bot rather than a regular user
- otherwise, it confusingly appears on the Users list and in the realm
creation from Slack import flow, in the dropdown for choosing the
organization owner account.
Closes#34649
Integration bot messages in Slack may include "blocks" and
"attachments," which are Slack's messaging features.
Currently, these messages aren't processed when converting
Slack export data.
This commit adds support for converting integration bot
messages, as well as other Slack messages containing "blocks"
and "attachments".
Message payload with the block type `rich_text` is skipped because all
messages sent by users have this format.
Fixes#31162.
[1]=https://docs.slack.dev/reference/block-kit/blocks/rich-text-block/
We encountered the following two new cases with integration bots in
Slack imports:
1. Bots without the image_72 field in their data. Such bots should fall
back to gravatar.
2. Bots whose bot_id is the sender of certain messages, but querying the
bots.info endpoint returns bot_not_found error. We should create
dummy accounts in place of such bots.
Just like in the prior commit we add similar code in `get_subscription`,
here we do this for reactions. If two Slack accounts, which will be
merged into one Zulip account, posted the same emoji reaction to a
message, we need to avoid duplication.
In 1517601e9d we added merging of accounts
with the same email address; but didn't prevent duplication of
subscriptions when two Slack accounts, merged into one Zulip account,
belong to a channel.
Currently, threads in Slack direct messages will increment the
`thread_counter` variable inside the thread conversion logic. Since we
don't treat thread messages in Slack DMs differently than any other DM,
threads in DM will only falsely increment the thread topic names in
channels.
This adds a condition that checks if the Slack message is a DM or not
before executing the thread conversion logic.
Previously, `build_message` sets a message's topic name to the given
topic name, regardless of whether the message was a direct message (DM)
or a group direct message (GDM).
This change adds the `is_private` parameter to `build_message`. If
`is_private` is `True`, the `topic_name` will be overridden to an empty
string (""). Consequently, this also updates the third-party importers
to pass this parameter when calling `build_message`.
Co-authored-by: Pieter CK <pieterceka123@gmail.com>
The content `look` does not render as a
link, nor an image upload (were it to `/user_uploads/...`). The
`` syntax is intended for inline images, but unsupported in
Zulip, and as such does not link or render as _anything_.
Ensure a newline between message content and any attachments.
1. Fetching from the `/users.list` endpoint is supposed to use
pagination. Slack will return at most 1000 results in a single
request. This means that our Slack import system hasn't worked
properly for workspaces with more than 1000 users. Users after the
first 1000 would be considered by our tool as mirror dummies and thus
created with is_active=False,is_mirror_dummy=True.
Ref https://api.slack.com/methods/users.list
2. Workspaces with a lot of users, and therefore requiring the use of
paginated requests to fetch them all, might also get us to run into
Slack's rate limits, since we'll be doing repeating requests to the
endpoint.
Therefore, the API fetch needs to also handle rate limiting errors
correctly.
Per, https://api.slack.com/apis/rate-limits#headers, we can just read
the retry-after header from the rsponse and wait the indicated number
of seconds before repeating the requests. This is an easy approach to
implement, so that's what we go with here.
Previously, the `check_token_access` function had a hardcoded
`required_parameters` variable because it was only used in the Slack
data importer. This commit refactors `required_parameters` into a
function parameter, enabling the function to check a Slack token’s scope
for other purposes, such as Slack webhook integration.
Additionally, this commit changes the Slack API call in
`check_token_access` from `teams.info` to `api.test`. The endpoint is
better suited for this purpose since we're only checking a token’s scope
using the response header here.
Currently, we're unable to convert messages from Slack's integration
bots because this message subtype doesn't come from a Slack "user", that
is they don't have a Slack user profile.
This is a preparatory change to support converting Slack's integration
bot messages. This commit artificially creates Slack user data from the
integration bot's "profile" so that we can create a corresponding Zulip
user for them.
Part of #31311.
This commit updates all third-party importer tools (Slack, Mattermost,
and Rocket Chat) in the `zerver/data_import` directory to also output a
migration_status.json file in their output tarball.
This is required because all importable tarball will be checked for
migration compatibility during import.
Fixes#28443.
For resizing the icon.png files, we use resize_avatar, not resize_logo.
This is pretty confusing - sure, for icons we use the same function as
for avatars, but we should have a proper name for the function called in
the icon context. So this commit also adds resize_realm_icon, and
changes the calls to resize_avatar in icon contexts to
resize_realm_icon.
This commit makes the third-party data converters check for invalid user
emails. If it finds any, it’ll raise an Exception and show an error
message with all the bad emails listed out.
Fixes: #31783
This lets slack conversions be done on development hosts, which have a
trailing :9991 on their EXTERNAL_HOST; otherwise, we generate fake
emails like `imported-slack-bot@host.name:9991` which fail to
validate.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.
Fixes part of #25713
We use a truncated SHA256 of the id and a server-side secret to make
emoji have non-guessable filenames, while also making collisions
unlikely.
We also adjust the Slack import to use the same SHA-based name,
instead of taking the same name as it had in Slack.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.
It also renames variables and methods of type -
"huddle_message" to "group_direct_message".
This is a part of #28640
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.
Fixes part of #28640.
Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
It is possible to have multiple users with the same email address --
for instance, when two users are guests in shared channels via two
different other Slack instances.
Combine those Slack user-ids into one Zulip user, by their user-id;
otherwise, we run into problems during import due to duplicate keys.
1e5c49ad82 added support for shared channels -- but some users may
only currently exist in DMs or MPIMs, and not in channel membership.
Walk the list of MPIM subscriptions and messages, as well as DM users,
and add any such users to the set of mirror dummy users.
