This commit adds `parse_migration_status`, which takes in the string
output of `showmigrations` and parse it into key-value pair of installed
apps and a list of its migration status.
This is a prep commit to rework the check migrations function of
import/export which will parse the output of `showmigrations` to write
the `migration_status.json` file.
For import from Rocket.Chat, we set the channel
posting policy to "Admins and moderators" only
for Rocket.Chat channels that are read-only.
This commit adds the code to set the new group
setting accordingly.
This commit sorts the list of migrations to ensure the same
sets of migrations don't somehow ordered differently. This
has been reported to happen when importing Zulip Cloud to
self-host (zulip-cloud-current).
The order of migrations listed in `migration_status.json`
don't actually matter, migrations specify which other
migrations they depend on.
Fixes#32826.
This commit introduces a new non-operational
`moderation_request_channel` field to the server/API. This setting will
support a feature allowing users to flag or report abusive content
(harassment, spam, etc.).
Fixes part of #20047.
id_field_name was being used only to set the initial dummy
values for realm settings. We can directly add "_id" to the
setting name instead of having an extra id_field_name field.
Right now, the number of queries has remained the same, but when we add
more settings in the future, we won't be increasing the number of
queries when iterating over stream permission group settings.
We were using admins group as a hardcoded value for the default of
`can_remove_subscribers_group`, now we use a function to get the value
of the default group.
This change improves error handling in `do_import_realm` by replacing
the use of a generic Exception with CommandError. The updated approach
provides clearer, user-friendly error messages when there is a version
mismatch between the exported data and the Zulip server.
Fixes#32292.
When transferring a realm to a server that has a different set of
applied migrations (different Zulip versions), there is a chance that
the imported data formats appear to be compatible but data invariants
could still be violated.
This commit adds an assertion during the import process to verify
that both the exported realm and the importing server have matching
Zulip versions and have a compatible set of migrations.
We didn't have thumbnailing for images coming from data import and this
commit adds the functionality.
There are a few fundamental issues that the implementation needs to
solve.
1. The images come from an untrusted source and therefore we don't want
to just pass them through to thumbnailing without checking. For that
reason, we cannot just import ImageAttachment rows from the export
data, even for zulip=>zulip imports.
The right way to process images is to pass them to maybe_thumbail(),
which runs libvips_check_image() on them to verify we're okay with
thumbnailing, creates ImageAttachment rows for them and sends them
to the thumbnailing queue worker. This approach lets us handle both
zulip=>zulip and 3rd party=>zulip imports in the same way,
2. There is a somewhat circular dependency between the Message,
Attachment and ImageAttachment import process:
- ImageAttachments would ideally be created after importing
Attachments, but they need to already exist at the time of Message
import. Otherwise, the markdown processor doesn't know it has to add
HTML for image previews to messages that reference images. This would
mean that messages imported from 3rd party tools don't get image
previews.
- Attachments only get created after Message import however, due to the
many-to-many relationship between Message and Attachment.
This is solved by fixing up some data of Attachments pre-emptively, such
as the path_ids. This gives us the necessary information for creating
ImageAttachments before importing Messages.
While we generate ImageAttachment rows synchronously, the actual
thumbnailing job is sent to the queue worker. Theoretically, the worker
could be very backlogged and not process the thumbnails anytime soon.
This is fine - if the app is loaded and tries to display a message with
such a not-yet-generated thumbnail, the code in `serve_file` will
generate the thumbnails synchronously on the fly and the user will see
the image preview displayed normally. See:
1b47134d0d/zerver/views/upload.py (L333-L342)
We create an unnamed user group with just the group creator as it's
member when trying to set the default. The pattern I've followed across
most of the acting_user additions is to just put the user declared
somewhere before the check_add_user_group and see if the test passes.
If it does not, then I'll look at what kind of user it needs to be set
to `acting_user`.
This commit introduced 'creator' and 'date_created'
fields in user groups, allowing users to view who
created the groups and when.
Both fields can be null for groups without creator data.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.
1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.
2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.
Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.
Fixes part of #25928.
The emoji dir is present in the data from our export tool. This was
added in 468afe4840.
This comment hasn't been updated since
c4b886d8ae, so probably we just forgot to
refresh it when custom emoji export was added.
This commit improves the assert statements to verify
that the table name is not "usermessage' instead of
verifying that table name doesn't include a substring
"usermessage".
This prep commit will help to avoid assertion error when
importing "onboardingusermessage" table.
This commit renames the "Huddle" Django model class to
"DirectMessageGroup", while maintaining the same table --
"zerver_huddle".
Fixes part of #28640.
Hash the salt, user-id, and now avatar version into the filename.
This allows the URL contents to be immutable, and thus to be marked as
immutable and cacheable. Since avatars are served unauthenticated,
hashing with a server-side salt makes the current and past avatars not
enumerable.
This requires plumbing the current (or future) avatar version through
various parts of the upload process.
Since this already requires a full migration of current avatars, also
take the opportunity to fix the missing `.png` on S3 uploads (#12852).
We switch from SHA-1 to SHA-256, but truncate it such that avatar URL
data does not substantially increase in size.
Fixes: #12852.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.
It also renames variables and methods of type -
"huddle_message" to "group_direct_message".
This is a part of #28640
Fixes#26369.
There are two important fixes to make to the dicts in edit_history:
1. Update the user_id so that it points to the imported sender.
2. Apply fix_message_rendered_content to the prev_rendered_content data
to fix up mentions and other such syntax.
The docstring was misleading talking about the import from non-Zulip
platforms, when this function is also very much applicable for
Zulip-to-Zulip imports.
