Fixes CI - it was failing due to the API test for organization exports,
which was returning an error due to there being no Organization Owners
with consent to private data export.
As explained in the comments, if in an export with consent there are no
consenting owners or in a public export there are no owners with email
visibility set to at least ADMINS, the exported data will, upon import,
create an organization without usable owner accounts.
Adds detailed tests for the work in the prior commits fixing the
treatment of private data in various tables in exports with consent and
public exports.
This is private information, as by inspecting the DirectMessageGroup
objects and their associated Subscription objects, you could determine
which users conversed with each other in a DM group.
This did *not* leak any actual message - only the fact that at least one
of the users in the group sent a group DM.
The prior significantly restricted what data gets exported from
non-consented users. The last thing we're missing is to fix the logic
to work correctly for public exports.
Prior commits focused on addressing exports with consent. This commit
adapts it to work with public exports.:
- Do not turn user accounts into mirror dummies in the public export - or
after export->import you'll end up with a realm with no functional
accounts; as every user is non-consented and the original logic added in
the prior commits will turn them into mirror dummies.
- Some of the custom fetch/process functions were changed without
considering public exports - now they work correctly, by setting
consenting_user_ids to an empty set.
The Subscription Config is constructed in a bit of a strange way, that's
not compatible with defining a custom_fetch function.
Instead we have to extend the system to support passing a custom
function for processing just the final list of rows right before it's
returned for writing to export files.
As explained in the comment, if we turn a non-consented deactivated user
into a mirror dummy, this will violate the rule that a deactivated user
cannot restore their account by themselves after an export->import
cycle.
As explained in the comment added to the function, in terms of privacy
concerns, it is fine to export all data for these accounts. And it is
important to do - so that exporting an organization which was originally
imported e.g. from Slack doesn't result in excessively limited data for
accounts that were mirror dummies and never "activated" themselves.
Now that we severely limited the way that non-consenting users get
exported, we need to start to consider bots as consenting when
appropriate - otherwise the exported bot accounts will be unusable after
importing.
An administrator shouldn't be able to bypass a user's setting to hide
their email address from everyone, including admins.
Therefore, we should overwrite the delivery_email for such users during
export - unless the user consented to have their private data exported.
The notable consequence of this is that such user accounts will become
completely inaccessible after importing this data to a new server, due
to not having a functional email address on record.
These accounts will only be possible to reclaim via a manual
intervention to change the email address on the `UserProfile` by server
administrators.
This allows us to get rid of the call to `get_consented_user_ids` in
`fetch_usermessages`. Now it's only called at the beginning of the
export, eliminating the redundant db query and also resolving the
potential for data consistency issues, if some users change their
consent setting after the export starts.
Now the full export process operates with a single snapshot of these
consenting user ids.
These ids need to be plumbed through via a file rather than normal arg
passing, because this is a separate management command, run in
subprocesses during the export.
These users didn't consent to having their private data exported.
Therefore, correct handling of these users should involve scrubbing
their settings to just match the realm defaults.
Instead of making repeated calls to get_consented_user_ids, we can just
fetch it (mostly) once and put it in
`context["exportable_user_ids"]`. This is essentially what the
(unused until now) exportable_user_ids logic was added for after all.
The added, intended, effect of this is that non-consenting users will
now get exported as mirror dummy accounts, due to the handling of
non-exportable users in `custom_fetch_user_profile`.
The remaining additional call to `get_consented_user_ids` is in
`fetch_usermessages`. This one is tricky as this function gets called
in subprocesses via
`zerver/management/commands/export_usermessage_batch.py` management
command invoked by the export process.
It requires passing the `exportable_user_ids` in some other way. This
can be dealt with in upcoming commits.
We shouldn't export the entire Client table - it includes Clients for
all the realms on the server, completely unrelated to the realm we're
exporting. Since these contain parts of the UserAgents used by the
users, we should treat these as private data and only export the Clients
that the specific data we're exporting "knows" about.
This reduces confusion amount users when they download the
intel version and it works super slow. Downloading the arm64
version on an intel mac would just not work.
Users who use intel macs have a habit now (atleast me) to look
for `Intel` version of software when downloading an app. So, made
`Intel` bold to help that process.
Changes banner text to "This conversation also has older unread
messages. Jump to first unread message?"
Updates button label from "Jump to first unread" to "Jump".
- Rename the variable "pills" to "email_pill_widget". To conform better
with other pill widgets.
- Re-use the email pill creation function.
- Directly access `item.email`, skipping function call.
We already do this in computed_settings.py, but only if the
S3 (secret) key is set. Those aren't required to be set, and tusd
_requires_ a region, so we try again to suss it out here.
Earlier, in 'EmptyTopicNameTest.test_initial_state_data',
we were not passing a short 'fetch_event_types' to 'do_events_register'
resulting in unnecessary work to fetch extra initial data which
isn't important for the test.
This commits updates the test to pass a 'fetch_event_types'
parameter with the event types required for the test.
This commit replaces repetitive 'addClass/removeClass' pair usage
with a 'toggleClass' call to manage the visibility of
$topic_not_mandatory_placeholder used in compose recipient topic
input box, inline topic edit input box, and move message new topic
input box.
Helps to reduce code duplication.
This provides access logging metrics to Prometheus. For cardinality
reasons, we cannot (nor would we want to) put every request path into
its own label value -- but we do separate out the most-frequent access
paths (as well as some low-frequency but high-interest ones) into
their own label values.
In order to differentiate accesses to https://zulip.com/ from
https://example.zulipchat.com/ (both of which appear at path `/`), we
use a `grok_exporter.realm_names_regex` value in `zulip.conf`, which
is expected to be set to match the hostname of all possible realms.
When on_compose_select_recipient_update is called when we start
the compose box actions, then it subsequently calls
compose_recipient.update_on_recipient_change.
If there is a specified topic in the opts for the compose box,
then that should be set for various functions that are called
in update_on_recipient_change.
compose_recipient.update_topic_displayed_text is called later for
all cases, direct messages and empty topics, which will update the
compose_state.topic again.
As the web app is now using the "with" operator for links to channel
topic conversations, we need to filter out that operator when
checking the current narrow state.
When we're unable to process an invoice for a customer due to a billing
or assertion error, we log the error and add the customer ID to a set
of failed customer IDs. This allows us to continuing invoicing other
customers, and at the same time not invoice any other pending plans
attached to a customer we failed to invoice.
Extracts the logic for sending an email to billing support when
a remote server's audit log data is stale and there is an active
CustomerPlan that needs to be invoiced.
This commit renames "(no topic)" to "" when used as
topic name in `POST /typing`.
Message sent in "(no topic)" is translated as being
sent in "" by the server, so it makes sense to show
the typing notification in "" when message is being composed.
