As explained in the comments, if in an export with consent there are no
consenting owners or in a public export there are no owners with email
visibility set to at least ADMINS, the exported data will, upon import,
create an organization without usable owner accounts.
Adds detailed tests for the work in the prior commits fixing the
treatment of private data in various tables in exports with consent and
public exports.
This is private information, as by inspecting the DirectMessageGroup
objects and their associated Subscription objects, you could determine
which users conversed with each other in a DM group.
This did *not* leak any actual message - only the fact that at least one
of the users in the group sent a group DM.
The prior significantly restricted what data gets exported from
non-consented users. The last thing we're missing is to fix the logic
to work correctly for public exports.
Prior commits focused on addressing exports with consent. This commit
adapts it to work with public exports.:
- Do not turn user accounts into mirror dummies in the public export - or
after export->import you'll end up with a realm with no functional
accounts; as every user is non-consented and the original logic added in
the prior commits will turn them into mirror dummies.
- Some of the custom fetch/process functions were changed without
considering public exports - now they work correctly, by setting
consenting_user_ids to an empty set.
The Subscription Config is constructed in a bit of a strange way, that's
not compatible with defining a custom_fetch function.
Instead we have to extend the system to support passing a custom
function for processing just the final list of rows right before it's
returned for writing to export files.
As explained in the comment, if we turn a non-consented deactivated user
into a mirror dummy, this will violate the rule that a deactivated user
cannot restore their account by themselves after an export->import
cycle.
As explained in the comment added to the function, in terms of privacy
concerns, it is fine to export all data for these accounts. And it is
important to do - so that exporting an organization which was originally
imported e.g. from Slack doesn't result in excessively limited data for
accounts that were mirror dummies and never "activated" themselves.
Now that we severely limited the way that non-consenting users get
exported, we need to start to consider bots as consenting when
appropriate - otherwise the exported bot accounts will be unusable after
importing.
An administrator shouldn't be able to bypass a user's setting to hide
their email address from everyone, including admins.
Therefore, we should overwrite the delivery_email for such users during
export - unless the user consented to have their private data exported.
The notable consequence of this is that such user accounts will become
completely inaccessible after importing this data to a new server, due
to not having a functional email address on record.
These accounts will only be possible to reclaim via a manual
intervention to change the email address on the `UserProfile` by server
administrators.
This allows us to get rid of the call to `get_consented_user_ids` in
`fetch_usermessages`. Now it's only called at the beginning of the
export, eliminating the redundant db query and also resolving the
potential for data consistency issues, if some users change their
consent setting after the export starts.
Now the full export process operates with a single snapshot of these
consenting user ids.
These ids need to be plumbed through via a file rather than normal arg
passing, because this is a separate management command, run in
subprocesses during the export.
These users didn't consent to having their private data exported.
Therefore, correct handling of these users should involve scrubbing
their settings to just match the realm defaults.
Instead of making repeated calls to get_consented_user_ids, we can just
fetch it (mostly) once and put it in
`context["exportable_user_ids"]`. This is essentially what the
(unused until now) exportable_user_ids logic was added for after all.
The added, intended, effect of this is that non-consenting users will
now get exported as mirror dummy accounts, due to the handling of
non-exportable users in `custom_fetch_user_profile`.
The remaining additional call to `get_consented_user_ids` is in
`fetch_usermessages`. This one is tricky as this function gets called
in subprocesses via
`zerver/management/commands/export_usermessage_batch.py` management
command invoked by the export process.
It requires passing the `exportable_user_ids` in some other way. This
can be dealt with in upcoming commits.
We shouldn't export the entire Client table - it includes Clients for
all the realms on the server, completely unrelated to the realm we're
exporting. Since these contain parts of the UserAgents used by the
users, we should treat these as private data and only export the Clients
that the specific data we're exporting "knows" about.
We already do this in computed_settings.py, but only if the
S3 (secret) key is set. Those aren't required to be set, and tusd
_requires_ a region, so we try again to suss it out here.
(cherry picked from commit 9c043c6c14)
Show `You are not subscribed to #xyz. Subscribe` bookend
on channel or topic which are not subscribed and have no messages.
Fixes: #33209
(cherry picked from commit 4696c8eb67)
This commit ensures that the user emoji in the search pill
is correctly aligned with the search text. The issue was
caused by improper line height, leading to the emoji being
slightly cut off at the top. Adjusting the `line-height`
of `.pill-value` resolves this, providing a consistent
and visually balanced appearance.
(cherry picked from commit 1b260c9fd7)
Currently, imported direct messages from third-party exports might have
a non-empty string as their topic name.
This migration updates the topic names for all imported third-party DMs
and GDMs to an empty string if they aren’t already.
Fixes#29466.
Co-authored-by: Pieter CK <pieterceka123@gmail.com>
(cherry picked from commit f6b3d59c35)
Currently, threads in Slack direct messages will increment the
`thread_counter` variable inside the thread conversion logic. Since we
don't treat thread messages in Slack DMs differently than any other DM,
threads in DM will only falsely increment the thread topic names in
channels.
This adds a condition that checks if the Slack message is a DM or not
before executing the thread conversion logic.
(cherry picked from commit d5e28bcd28)
Previously, `build_message` sets a message's topic name to the given
topic name, regardless of whether the message was a direct message (DM)
or a group direct message (GDM).
This change adds the `is_private` parameter to `build_message`. If
`is_private` is `True`, the `topic_name` will be overridden to an empty
string (""). Consequently, this also updates the third-party importers
to pass this parameter when calling `build_message`.
Co-authored-by: Pieter CK <pieterceka123@gmail.com>
(cherry picked from commit 845f0d40e1)
When on_compose_select_recipient_update is called when we start
the compose box actions, then it subsequently calls
compose_recipient.update_on_recipient_change.
If there is a specified topic in the opts for the compose box,
then that should be set for various functions that are called
in update_on_recipient_change.
compose_recipient.update_topic_displayed_text is called later for
all cases, direct messages and empty topics, which will update the
compose_state.topic again.
(cherry picked from commit d7873fbc11)
As the web app is now using the "with" operator for links to channel
topic conversations, we need to filter out that operator when
checking the current narrow state.
(cherry picked from commit 24a65c1783)
This commit renames "(no topic)" to "" when used as
topic name in `POST /typing`.
Message sent in "(no topic)" is translated as being
sent in "" by the server, so it makes sense to show
the typing notification in "" when message is being composed.
(cherry picked from commit d011fb0621)
We now use `data-user-group-id` to check if a banner for the same
group already exists, preventing duplicate warnings when the same
group is mentioned multiple times.
(cherry picked from commit 35289dfe51)
In dark theme, when clicking the "Add task" button of
todo and "Add option" button of poll, the background
color incorrectly turns white.
This commit removes the white background color of the
buttons and makes it consistent woth the other green
buttons.
(cherry picked from commit eef44429e2)
When a message is moved using the "Move only this message" option
a confirmation toast is shown. This commit adds support to show
"general chat" in link of the toast when a message is moved to it.
(cherry picked from commit 939691dfed)
Searching for a word that appears in a message in a empty string
topic via the search box resulted in a message view where the
topic names in the recipient_row were empty string instead of
`realm_empty_topic_display_name`.
This commit fixes that bug.
(cherry picked from commit 8383b11526)
This was likely a longstanding issue that wasn't
caught because we required topics on CZO. The new
logic ensures topic match even for empty string
(general chat) topics.
(cherry picked from commit 8a51fa4b83)
Earlier, for topic="" and mandatory_topics=False, the inline topic
edit input field width was not set correctly when the inline topic
edit was started for the first time.
This resulted in overflowing placeholder.
This commit fixes that bug.
(cherry picked from commit b53327dabe)
Showing this banner in every view can be annoying. As a first step,
we only show it in conversation view to reduce the banner spam.
(cherry picked from commit 6c81ff61ee)
Apparently, while we set our own maximum password length of 100
characters, zxcvbn had a hardcoded maximum length of 72 characters,
and threw an exception if that was exceeded.
The fact that we're discovering this now would suggest that nobody has
previously attempted a password between 72 and 100 characters in
length.
(cherry picked from commit 37b7a32eb4)
Refactors get_recipient_label so that it's a bit clearer what the
recipient_information parameter is for and what we do when that
parameter is undefined.
In doing so, we no longer treat the constructed objects, that are
passed as the recipient_information parameter, and actual Message
objects, that we get from the current message list view, as the
same thing.
(cherry picked from commit 7d3b77e490)
Both the inbox and recent conversation views pass information about
the reply recipient to this function's caller so that the button
text can be updated for the focused row.
Therefore, the check for an undefined current message list should
be inside the case where the recipient information parameter is
undefined.
This was changed in f630272b4c when non-message list views set
undefined for the current message list.
(cherry picked from commit 4f163e5ad2)
Renames ComposeClosedMessage to ReplyRecipientInformation, and
exports the type from compose_closed_ui.ts so that the functions
that construct these objects from the recent conversations and
inbox views have the type available.
Also, renames the variables for these objects to not be "message",
so that it's clear that these are not Message objects.
(cherry picked from commit b48134a03e)
Renames update_reply_recipient_label to
update_recipient_text_for_reply_button.
This better matches the function that sets the default text for
the closed compose box button: set_standard_text_for_reply_button.
(cherry picked from commit 94fe5fc173)
