Otherwise, this fails on `match.group(1)` as there is no match group.
The server would ideally respond with a 521 or 556 code[^1] on initial
connection, but aiosmtpd does not provide that option.
[^1]: https://www.rfc-editor.org/rfc/rfc7504
This is done in a best-effort way -- we rely on the final ACL check
happening when the message is sent. However, by then we will be in
the `email_mirror` worker, with no way to convey to the original SMTP
client that the sending failed; giving the client the information will
allow them to provide it back to the end-user, instead of merely being
swallowed up in the worker.
Using postfix to handle the incoming email gateway complicates things
a great deal:
- It cannot verify that incoming email addresses exist in Zulip before
accepting them; it thus accepts mail at the `RCPT TO` stage which it
cannot handle, and thus must reject after the `DATA`.
- It is built to handle both incoming and outgoing email, which
results in subtle errors (1c17583ad5, 79931051bd, a53092687e,
#18600).
- Rate-limiting happens much too late to avoid denial of
service (#12501).
- Mis-configurations of the HTTP endpoint can break incoming
mail (#18105).
Provide a replacement SMTP server which accepts incoming email on port
25, verifies that Zulip can accept the address, and that no
rate-limits are being broken, and then adds it directly to the
relevant queue.
Removes an incorrect comment which implied that missed-message
addresses were only usable once. We leave rate-limiting to only
channel email addresses, since missed-message addresses are unlikely
to be placed into automated systems, as channel email addresses are.
Also simplifies #7814 somewhat.
This is a preparatory refactor for migrating the internal structure of
Recipient objects for group DMs to use the DirectMessageGroup type,
not the legacy PERSONAL type. This step has the message-sending code
path check if a DirectMessageGroupe exists and prefer it if available.
It should have no effect in production other than doing a useless
database query for each outgoing DM, since we do not at present ever
create such DirectMessageGroup objects. (It will not add a marginal
database query once the migration is complete, just during this
transition).
This commit updates code to not prefetch can_access_all_users_group
and can_access_all_users_group__named_user_group fields using
select_related. We can just use get_realm_system_groups_name_dict
function to check if setting is set to "Everyone" group when
needed and can avoid unnecessarily fetching groups for every user
query.
This commit updates code to not pre-fetch DM permission
group settings using select_related and instead just
fetch the required data from DB when checking permission.
This will increase one query but will help in pre-fetching
the settings for all users and for all type of messages.
Fixes part of #33677.
This commit adds a `sender_id` parameter to the
`GET /streams/{stream_id}/email_address` endpoint to specify the
ID of a user or bot which should appear as the sender when messages
are sent to a channel using the channel email address.
Earlier, Email gateway bot was always the sender.
Fixes part of #31566.
This commit extracts the logic for creating or retrieving a channel
email token into a dedicated `get_channel_email_token` function.
This improves code clarity by decoupling token generation from
the email encoding process.
This commit renames the 'queue_json_publish' function to
'queue_json_publish_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.
In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
Fixes part of #30489.
This commit removes the 'email_token' field from the
'Stream' model, introduces a new model 'ChannelEmailAddress',
and backfills it.
This is a prep work towards our plan to generate unique
channel emails for different users, which can be used
to check post permissions at message send time.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
Besides "RE:" and "FWD:", "AW:" (from German "Antwort" for answer) is
another common prefix in email subjects. Let email_mirror automatically
remove it as well.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.
It also renames variables and methods of type -
"huddle_message" to "group_direct_message".
This is a part of #28640
Creates a new "realm_deactivated" email that can be sent to realm
owners as part of `do_deactivate_realm`, via a boolean flag,
`email_owners`.
This flag is set to `False` when `do_deactivate_realm` is used for
realm exports or changing a realm's subdomain, so that the active
organization owners are not emailed in those cases.
This flag is optional for the `deactivate_realm` management command,
but as there is no active user passed in that case, then the email
is sent without referencing who deactivated the realm.
It is passed as `True` for the support analytics view, but the email
that is generated does not include information about the support
admin user who completed the request for organization deactivation.
When an active organization owner deactivates the organization, then
the flag is `True` and an email is sent to them as well as any other
active organization owners, with a slight variation in the email text
for those two cases.
Adds specific tests for when `email_owners` is passed as `True`. All
existing tests for other functionality of `do_deactivate_user` pass
the flag as `False`.
Adds `localize` from django.util.formats as a jinja env filter so
that the dates in these emails are internationlized for the owner's
default language setting in the "realm_deactivated" email templates.
Fixes#24685.
It's going to be helpful in the future to record the reason for realm
deactivation.
- For information tracking
- For making a distinction between cases where we can allow realm owners
to reactivate their realm via a self-serve flow (e.g.
"owner_request") vs where we can't (ToS abuse).
Updates translated JsonableError strings that relate to streams
to use channel instead of stream. Separated from other error string
updates as this is a dense area of changes for this rename.
Part of stream to channel rename project.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.
Fixes part of #28640.
Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
The get_display_recipient helper is a clumsy way to get
stream names, and it's not even representative of how
most of our code retrieves stream names.
The new helper also double-checks that the Stream
object has the correct recipient id.
This commit adds code to pass all the required arguments to
select_related call for MissedMessageEmailAddress such that
only the required related fields are fetched from the database.
Previously, we did not pass any arguments to select_related,
so all the directly and indirectly related fields were fetched
when many of them were actually not being used and made the
query unnecessarily complex.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The content of a message is truncated to `MAX_MESSAGE_LENGTH`, which
is 1000 characters. Since the email gateway places attachments at the
very end of the extracted body, that means that they are the first
thing to get truncated off.
That is, if an incoming email message contains 1000 `a`s and an image
attachment, the link that attaches the attachment to the message will
get truncated off, leaving it dangling in the database.
Truncate the message body content separately from the attachment links
which are included at the end of the body.
When the email mirror gateway is sending messages "as" a user (as
triggered by having access to the missed-message email address),
attachments were still created as the Email Gateway bot. Since the
sender (the end-user) was not the owner of those attachments (the
gateway bot), nor were they referenced yet anywhere, this resulted in
the attachments being "orphaned" and not allowed to be accessed by
anyone -- despite the attachment links being embedded in the message.
This was accompanied by the error:
```
WARN [] User 12345 tried to share upload 123/3LkSA4OcoG6OpAknS2I0SFAQ/example.jpf in message 123456, but lacks permission
INFO [zerver.lib.email_mirror] Successfully processed email from user 12345 to example-stream
```
We solve this by creating attachment objects as the users the message
will be sent from.
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.
Fixes#2755.
Fixes#19994.
Although our POST /messages handler accepts the ‘to’ parameter with or
without JSON encoding, there are two problems with passing it as an
unencoded string.
Firstly, you’d fail to send a message to a stream named ‘true’ or
‘false’ or ‘null’ or ‘2022’, as the JSON interpretation is prioritized
over the plain string interpretation.
Secondly, and more importantly for our tests, it violates our OpenAPI
schema, which requires the parameter to be JSON-encoded. This is
because OpenAPI has no concept of a parameter that’s “optionally
JSON-encoded”, nor should it: such a parameter cannot be unambiguously
decoded for the reason above.
Our version of openapi-core doesn’t currently detect this schema
violation, but after the next upgrade it will.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
