`deliver_scheduled_emails` tries to deliver the email synchronously,
and if it fails, it retries after 10 seconds. Since it does not track
retries, and always tries the earliest-scheduled-but-due message
first, the worker will not make forward progress if there is a
persistent failure with that message, and will retry indefinitely.
This can result in excessive network or email delivery charges from
the remote SMTP server.
Switch to delivering emails via a new queue worker. The
`deliver_scheduled_emails` job now serves only to pull deferred jobs
out of the table once they are due, insert them into RabbitMQ, and
then delete them. This limits the potential for head-of-queue
failures to failures inserting into RabbitMQ, which is more reasonable
than failures speaking to a complex external system we do not control.
Retries and any connections to the SMTP server are left to the
RabbitMQ consumer.
We build a new RabbitMQ queue, rather than use the existing
`email_senders` queue, because that queue is expected to be reasonably
low-latency, for things like missed message notifications. The
`send_future_email` codepath which inserts into ScheduledEmails is
also (ab)used to digest emails, which are extremely bursty in their
frequency -- and a large burst could significantly delay emails behind
it in the queue.
The new queue is explicitly only for messages which were not initiated
by user actions (e.g., invitation reminders, digests, new account
follow-ups) which are thus not latency-sensitive.
Fixes: #32463.
Fixes#33719
If the user is not subscribed to the new stream but is a subsriber
of the old stream, we were not sending any update events to the user.
This results in unexpected behaviour related to the message.
Passing the user group object in case of named user group is fine for
`do_change_stream_group_based_setting`. But for anonymous groups, if the
code path calling that function is not creating a new anonymous user
group, it has to modify the user group by itself before calling that
function. In that case, if `old_setting_api_value` is not provided,
`old_user_group` is calculated false, since the group id has not changed
for the stream, but the group membership has changed.
old_setting_api_value will be the same as new_setting_api_value in such
a case.
It is better to accept the new setting value as either an int or
UserGroupMembersDict, so that `do_change_stream_group_based_setting` can
decide what to do with that argument.
Earlier, the migration code was replacing the occurrence of "general
chat" as topic name in the database with `""`. This resulted in an
error which permanently broke `/near/` URL links to existing topics
named "general chat".
This commit updates the migration approach to instead move messages
from "general chat" to `""`. An edit history entry is added.
It results in the same expected behaviour and fixing that bug.
This is more consistent with how other URLs work in Zulip.
Replaces `/message_edit_typing` with `/messages/{message_id}/typing`.
The `message_id` parameter, previously passed in the request body,
is now included in the URL path.
Although, currently there are no scenarios where we are using
bulk_access_messages for edit. But we might do so in the future, and
it's better to have an explicit argument called is_modiying_message in
that case, so that the person making that change makes a conscious
decision of setting that property.
This is valuable so that one is forced to explicitly make a decision
on what is correct when adding new callers. Past experience tells us that
not having to explicitly show the decision leads to people introducing
security bugs in PRs that the maintainer has to catch in review, and our
goal for access control code should be that security bugs are hard to write.
Fixes#33688.
This is valuable so that one is forced to explicitly make a decision
on what is correct when adding new callers. Past experience tells us that
not having to explicitly show the decision leads to people introducing
security bugs in PRs that the maintainer has to catch in review, and our
goal for access control code should be that security bugs are hard to write.
Fixes part of #33688.
Tests were failing due to credit usage being capture in the next
month instead of current if the current time is last day of the
month. To fix this, we mock current time to not be the last day
of any month.
We do not fetch all the realm group settings using
select_related for register data now since it takes a
lot of time in planning phase. This commit updates
the code to fetch all the members and subgroups data
in user_groups_in_realm_serialized so that we do not
need to access each setting group individually.
user_groups_in_realm_serialized is updated to send the
required data accordingly.
Fixes#33656.
Following improvments are done in user_groups_in_realm_serialized-
- Members and subgroups are fetched using a single query using
union.
- Query to fetch groups now does not fetch setting fields since
we already have the members and subgroups of all the setting
groups and we can check if setting group was a named group
or not directly without checking named_user_group field of
UserGroup object as we have IDs of all named groups of the realm.
When a channel has both "general chat" and "" topic and a user
has visibility policy configured for both the topics, then
the migration to rename "general chat" to "" resulted in error
as it violated '(user_profile_id, stream_id, lower(topic_name)'
unique constraint.
This commit fixes the error.
For each row with topic="general chat", we create a new row
with topic="". On conflict we update the visibility policy
of the existing row -- we prefer the visibility policy of
'general chat' topic over "" as we're renaming 'general chat'
to "" so the configured policy of 'general chat' should be
preferred and any "" topic created wouldn't be intentional
or would be for testing purpose as it is not announced yet.
Finally, we delete the rows with "general chat" topic.
This commit updates the GitHub webhook integration to include the
old issue number in the transfer message sent for the new repository,
alongside the old repository's full name.
We no longer archive private streams when they become vacant,
since user can still have permissions to subscribe to it.
And streams can anyways be archived manually if needed.
Fixes#33689.
Fixes#33567.
We have used the flag `is_modifying_message` since it's more generic
than an archived channel specific flag and helps us understand better
what is the condition where we do not want to allow archived channels.
We have not added tests for message edit since it has an existing test
for this.
Zulip now supports empty string as a valid topic name.
For clients predating this feature, such messages appear
in "general chat" topic. Messages sent to "general chat" are
stored in the database as having a "" topic.
This commit adds a migration to rename the existing
"general chat" topic in the database to "".
Fixes parts of #32996.
This commit implements the backend of migrating the
`allow_edit_history` setting to
`message_edit_history_visibility_policy`.
This allows organizations, to have an intermediate setting to
view only the "Moves" history of the messages.
We still pass `realm_allow_edit_history` in `/register` response
though for older clients with its value being set depending on the
value of `realm_message_edit_history_visibility_policy`. We set
`realm_allow_edit_history` to `False` if the
`realm_message_edit_history_visibility_policy` is "None", and
`True` for "Moves only" or "All" message edit history.
Fixes part of #21398.
Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This parameter is no longer restricted to realm administrators. Any
user can get the streams they have metadata access to by setting this
parameter to true.
In public_stream_user_ids function, which is used to get users
who can access public streams, there is no need to fetch members
of can_add_subscribers_group as we eventually exclude guests
from them and we have already included all non guest users of
the realm.
We were not actually using anything but the IDs here, so it was a
bunch of wasted work to fetch these.
This essentially reverts f48e87cd3c. At
the time, something like that was required, because we needed to check
if the channel was deactivated before exposing it to the API, but more
recent reworking of the system to change the setting if the channel is
deactivated, rather than masking it in fetch_initial_state_data, means
we can do this cleanup.
We keep around the old `include_all_active` parameter for backwards
compatibility.
Web frontend doesn't use this API and thus there were no changes needed
there.
When sending invites and password reminders, ensure that the email
address is not on the AWS SES suppression list. Addresses often
mistakenly end up on there and are never removed; automatically
removing them, if necessary, before we reach out to attempt a signup
reduces support overhead and perceived buggy behaviour.
