For get and filter queries of NamedUserGroup, realm_for_sharding
field is used instead of realm field, as directly using
realm_for_sharding field on NamedUserGroup makes the query faster
than using realm present on the base UserGroup table.
The import code wasn't setting subscriber_count at all, resulting in a
value of 0 when dealing with imports from 3rd party apps.
We run this unconditionally, also for imports from Zulip, since this
ensures we won't inherit incorrect values, if the data we're import has
them - e.g. due to some bugs that affected the server the data came from.
This commit includes the following changes:
- Add an administrator setting to customize the Welcome Bot
message when sending an invitation.
- Add an API endpoint to test the customized Welcome Bot message
by sending a copy of the message to the administrator.
Fixes#27663.
Co-authored-by: Akarsh Jain <akarsh.jain.790@gmail.com>
Closes#35280.
When importing from other apps, we want to generate initial channel
messages. We don't have the usual Zulip-style range of channels, so we
just settle on sending all these messages to a single channel, chosen
following the same criteria as already implemented in #35339.
The data conversion logic for exports from other apps will try to set
these announcement channels based on presence of channels with an
appropriate default name - however those sometimes do not exist.
Then it falls on the import process to choose a good default value. That
default should be the channel with the most subscribers.
This requires a tweak to the send_zulip_update_announcements_to_realm
code to respect the "Starting tomorrow, users in your organization will
receive updated" behavior promised to admins in the initial DMs.
Closes#34984.
Exports from other apps obviously don't come with Welcome Bot messages -
which isn't a great experience as it is, as those are meant to help with
onboarding. We should generate them for all users at import time.
We limit the set of users to active accounts to make the workload more
reasonable in case of large orgs with plenty of deactivated accounts.
The downside is that an imported user joining after being reactivated
will be lacking these messages. We can re-think this approach if needed.
Fixes#34820.
This logic was fundamentally broken:
```
is_realm_imported_from_other_zulip_server = RealmAuditLog.objects.filter(
realm=realm, event_type=AuditLogEventType.REALM_EXPORTED
).exists()
if not is_realm_imported_from_other_zulip_server:
send_zulip_update_announcements_to_realm(
realm, skip_delay=False, realm_imported_from_other_product=True
)
```
Because the `REALM_EXPORTED` was only created after the export completed
- meaning it couldn't be included in the export data at all. Thus
considering exports to be "not from Zulip" incorrectly.
We get around this issue by explicitly including an import_source in the
realm dict in the export data from 3rd party apps. The importer can then
rely on this value to determine if it's dealing with a Zulip-originated
export or not.
This refactors `parse_migration_status` to copy the algorithm of
Django's `showmigrations` command instead of parsing its output. This is
done so that the code is not susceptible to breaking changes if Django
modifies showmigrations's implementation.
The previous `parse_migration_status` logic has been repurposed into a
test utility function (`prase_showmigrations`). It is used to verify
that the new `parse_migration_status` generates output identical to the
actual `showmigrations` command.
The `test_clean_up_migration_status_json` is removed because
`test_parse_migration_status` has covered that behavior.
This table's export and import weren't working:
1. It didn't have a Config in export.py, so it wasn't exported at all.
2. Its `date_created` wasn't registered in `DATE_FIELDS`.
3. It wasn't registered in `ID_MAPS` in import_realm.py, so having any
SavedSnippets in the export would cause the import to fail with an
exception.
4. It was missing a `fix_datetime_fields` call in its import codepath.
The `lock=true` parameter fails, since it is not being run inside of a
transaction. We swap to not using locking, since there is no
correctness requirement for a lock here.
This removes a race which parallels 6f20c15ae9, but in the import
path; thumbnails could have been generated while the message content
was being rendered, leaving them permanently with spinners.
The act of rendering the message markdown is what enqueues the
thumbnailing of referenced images; previously, images were enqueued at
least _twice_: once when the row was created, and again for every
message they were referenced in.
The current `get_migration_by_app` has a rather naive approach to
compiling the migration status of a realm, which has led to issues like
#32826. Specifically, those flaws are:
- it does not report the complete state of the migration status of the
exporting servers, only the applied migration.
- it shows both the replaced and the squashed migrations. This would be
a problem if we decide to clean up old migration files we've
squashed(replaced) and import a slightly older realm with those still in
disk. `check_migration_status` would complain of incompatibility even
though those migration files don't matter (they are replaced, after
all).
- it does not clean up ancient/stale applied migrations (for reference,
see how `check-database-compatibility` cleans those)
This commit attempts to write a better `migration_status.json` by
parsing the output of `showmigrations` instead.
This is because Django's `showmigrations` has a lot more logic and
validations baked into it than previously thought. Ones that we care
about are:
- it does validations to make sure app names are valid
- it doesn't list replaced migrations and only squashed one
- it takes into account migrations in disk(`MigrationsLoader`) vs
applied migrations (`MigrationsRecorder`)
Which would resolve the first two points highlighted above.
This commit adds `parse_migration_status`, which takes in the string
output of `showmigrations` and parse it into key-value pair of installed
apps and a list of its migration status.
This is a prep commit to rework the check migrations function of
import/export which will parse the output of `showmigrations` to write
the `migration_status.json` file.
For import from Rocket.Chat, we set the channel
posting policy to "Admins and moderators" only
for Rocket.Chat channels that are read-only.
This commit adds the code to set the new group
setting accordingly.
This commit sorts the list of migrations to ensure the same
sets of migrations don't somehow ordered differently. This
has been reported to happen when importing Zulip Cloud to
self-host (zulip-cloud-current).
The order of migrations listed in `migration_status.json`
don't actually matter, migrations specify which other
migrations they depend on.
Fixes#32826.
This commit introduces a new non-operational
`moderation_request_channel` field to the server/API. This setting will
support a feature allowing users to flag or report abusive content
(harassment, spam, etc.).
Fixes part of #20047.
id_field_name was being used only to set the initial dummy
values for realm settings. We can directly add "_id" to the
setting name instead of having an extra id_field_name field.
Right now, the number of queries has remained the same, but when we add
more settings in the future, we won't be increasing the number of
queries when iterating over stream permission group settings.
We were using admins group as a hardcoded value for the default of
`can_remove_subscribers_group`, now we use a function to get the value
of the default group.
This change improves error handling in `do_import_realm` by replacing
the use of a generic Exception with CommandError. The updated approach
provides clearer, user-friendly error messages when there is a version
mismatch between the exported data and the Zulip server.
Fixes#32292.
When transferring a realm to a server that has a different set of
applied migrations (different Zulip versions), there is a chance that
the imported data formats appear to be compatible but data invariants
could still be violated.
This commit adds an assertion during the import process to verify
that both the exported realm and the importing server have matching
Zulip versions and have a compatible set of migrations.
We didn't have thumbnailing for images coming from data import and this
commit adds the functionality.
There are a few fundamental issues that the implementation needs to
solve.
1. The images come from an untrusted source and therefore we don't want
to just pass them through to thumbnailing without checking. For that
reason, we cannot just import ImageAttachment rows from the export
data, even for zulip=>zulip imports.
The right way to process images is to pass them to maybe_thumbail(),
which runs libvips_check_image() on them to verify we're okay with
thumbnailing, creates ImageAttachment rows for them and sends them
to the thumbnailing queue worker. This approach lets us handle both
zulip=>zulip and 3rd party=>zulip imports in the same way,
2. There is a somewhat circular dependency between the Message,
Attachment and ImageAttachment import process:
- ImageAttachments would ideally be created after importing
Attachments, but they need to already exist at the time of Message
import. Otherwise, the markdown processor doesn't know it has to add
HTML for image previews to messages that reference images. This would
mean that messages imported from 3rd party tools don't get image
previews.
- Attachments only get created after Message import however, due to the
many-to-many relationship between Message and Attachment.
This is solved by fixing up some data of Attachments pre-emptively, such
as the path_ids. This gives us the necessary information for creating
ImageAttachments before importing Messages.
While we generate ImageAttachment rows synchronously, the actual
thumbnailing job is sent to the queue worker. Theoretically, the worker
could be very backlogged and not process the thumbnails anytime soon.
This is fine - if the app is loaded and tries to display a message with
such a not-yet-generated thumbnail, the code in `serve_file` will
generate the thumbnails synchronously on the fly and the user will see
the image preview displayed normally. See:
1b47134d0d/zerver/views/upload.py (L333-L342)
We create an unnamed user group with just the group creator as it's
member when trying to set the default. The pattern I've followed across
most of the acting_user additions is to just put the user declared
somewhere before the check_add_user_group and see if the test passes.
If it does not, then I'll look at what kind of user it needs to be set
to `acting_user`.
This commit introduced 'creator' and 'date_created'
fields in user groups, allowing users to view who
created the groups and when.
Both fields can be null for groups without creator data.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.
1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.
2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
