Servers without any configured credentials raise a NoCredentialsError,
which is not a subclass of botocore.exceptions.ClientError, and hence
abort the password reset attempt.
Check for if we have any credentials at all before we attempt the API
call.
(cherry picked from commit 66c123dd43)
This adds a try-except block when running html2text when processing raw
messages from HTML to markdown.
convert_html_to_text is added mainly for testing convinience. We don't
have any sample of Mattermosts' problematic content that could trigger
this sort of error yet, so the test mocks convert_html_to_text to raise
error instead.
(cherry picked from commit 201a71b575)
User who did not have permission to create public channels
could create them by first creating a private or web-public
channel, if they had the permission to create them, and then
changing privacy of that stream to be a public stream.
Similarly user without permission to create private channels
could also create them.
This commit fixes both these bugs.
"first_message_id" field for subscription objects needs
to be updated when archiving a stream as we send a
notification message, but first_message_id will only
change if the stream did not have any messages previously.
This commit updates the code to update first_message_id
only when required.
(cherry picked from commit a6cc33f478)
Earlier, in `get_raw_unread_data` the ordering was applied inside
the CTE.
Once we leave the CTE scope and do a join, SQL makes no promise
about preserving the row order unless we re-specify ORDER BY in
the outer query.
Since, there was no ORDER BY clause in the outer query it was
resulting in a random ordering of the entries. This bug was caught
by `test_unreads_case_insensitive_topics` failing in a flaky way.
This commit fixes the bug.
(cherry picked from commit ccc82976dc)
Fetching a subscription and then checking if it exists was taking too
much space in a test and making it feel convoluted. We're planning to
check it more in future commits.
(cherry picked from commit 6baa106460)
In #34510 we fixed the underlying bug in the ldap integration, which
would cause users to end up with their system group memberships not
matching their .role value. However, users who may already be in that
state still need to be fixed through a migration. We implement that
here.
There are two things we fix here:
1. Group memberships. The user should have a direct group membership
for the specific system group implied by their .role.
2. We want to also add the missing RealmAuditLog entry.
This was broken, due the mechanism simply using our
is_guest/is_realm_admin/etc. role setters, but failing to adjust system
group memberships - resulting in corrupted database state.
We need to ensure that change_user_role is called for setting user role.
There are two relevant codepaths that run the sync based on
AUTH_LDAP_USER_FLAGS_BY_GROUP and thus need to get this right:
1. manage.py sync_ldap_user_data
2. Just-in-time user creation when a user without a Zulip account logs
in for the first using their ldap credentials. After
get_or_build_user returns, django-auth-ldap sees that the user
account has just been created, and proceeds to run ._populate_user().
Now that both user.save() and do_change_user_realm will be getting
called together, we need to ensure this always happens atomically.
This imposes the need to override _get_or_create_user to put it in a
transaction. The troublesome consequence is that this new
`atomic(savepoint=False)` causes the usual type of issue, where tests
testing error get their transaction rolled back and cannot continue
executing.
To get around that, we add a test helper
`artificial_transaction_savepoint` which allows these tests to wrap
their problematic blocks in an artificial transaction which provides a
savepoint, thus preventing the full test transaction rollback derailing
the rest of the test.
Zulip supports a configuration where account creation is limited solely
by being able to authenticate with a single-sign on authentication
backend, such as Google Authentication, SAML, or LDAP (i.e., the
organization places no restrictions on email address domains or
invitations being required to join, but has disabled the
EmailAuthBackend that is used for email/password authentication).
A bug in the Zulip server meant that Zulip allowed users to create an
account in such organizations by confirming their email address, without
having an account with the SSO authentication backend.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This works around tus/tusd#322, which in turn is caused by
aws/aws-sdk-go-v2#1816. This requires separate authentication via
service account key.
Fixes: #34186.
(cherry picked from commit e1aa8b1cb0)
The tusd protocol allows DELETE requests ("terminations") at any
point, including after a file has successfully been uploaded. This
can allow tusd to remove a file from the bucket, out from under Zulip.
We use the new-in-2.7.0 pre-terminate hook to look up the file which
the client is requesting to terminate, and reject the termination if
it is a file that the Zulip database is already aware of.
(cherry picked from commit cf51013bb7)
The type_id is the id of a UserProfile, Stream, or DirectMessageGroup,
not the id of a type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit ad31ef22f2)
We forgot to store the actual values in the audit log, making these logs
not very helpful in actually auditing a user's email change history.
(cherry picked from commit 5814ac559f)
Otherwise we get spammed with “Dropped breadcrumb because no client
bound” log messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit e8faa4a029)
In deploys `nginx_listen_port` set, tusd would fail to send its hook
requests, as it assumed that nginx would always be listening on
127.0.0.1:80.
Set the `nginx_listen_port` on the hook URL, if necessary.
(cherry picked from commit bee3c6eb59)
As explained in the comments, if in an export with consent there are no
consenting owners or in a public export there are no owners with email
visibility set to at least ADMINS, the exported data will, upon import,
create an organization without usable owner accounts.
Adds detailed tests for the work in the prior commits fixing the
treatment of private data in various tables in exports with consent and
public exports.
This is private information, as by inspecting the DirectMessageGroup
objects and their associated Subscription objects, you could determine
which users conversed with each other in a DM group.
This did *not* leak any actual message - only the fact that at least one
of the users in the group sent a group DM.
The prior significantly restricted what data gets exported from
non-consented users. The last thing we're missing is to fix the logic
to work correctly for public exports.
Prior commits focused on addressing exports with consent. This commit
adapts it to work with public exports.:
- Do not turn user accounts into mirror dummies in the public export - or
after export->import you'll end up with a realm with no functional
accounts; as every user is non-consented and the original logic added in
the prior commits will turn them into mirror dummies.
- Some of the custom fetch/process functions were changed without
considering public exports - now they work correctly, by setting
consenting_user_ids to an empty set.
The Subscription Config is constructed in a bit of a strange way, that's
not compatible with defining a custom_fetch function.
Instead we have to extend the system to support passing a custom
function for processing just the final list of rows right before it's
returned for writing to export files.
As explained in the comment, if we turn a non-consented deactivated user
into a mirror dummy, this will violate the rule that a deactivated user
cannot restore their account by themselves after an export->import
cycle.
As explained in the comment added to the function, in terms of privacy
concerns, it is fine to export all data for these accounts. And it is
important to do - so that exporting an organization which was originally
imported e.g. from Slack doesn't result in excessively limited data for
accounts that were mirror dummies and never "activated" themselves.
Now that we severely limited the way that non-consenting users get
exported, we need to start to consider bots as consenting when
appropriate - otherwise the exported bot accounts will be unusable after
importing.
