Adds a log for the end-to-end latency from when the worker decided
to send push notifications & received a success response from bouncer.
Fixes part of #35368.
(cherry picked from commit c604ecb902)
This avoids a potential unnecessary message.recipient fetch required by
is_stream_message(). is_stream_message() methods precedes the addition
of the denormalized is_channel_message column and is now unnecessary.
In practice, we usually fetch Message objects with `.recipient` already,
so I don't expect any notable performance impact here - but it's still a
useful change to make.
(cherry picked from commit 51cef01c29)
In `send_push_notifications`, we were manually asserting types of the
fields returned by the `remotes/push/e2ee/notify` endpoint at runtime
and constructed the `response_data` data structure for further use.
Pydantic is the perfect tool for this task.
This commit updates the concerned code to use Pydantic's `TypeAdapter`.
No functional change.
Fixes part of #35368.
(cherry picked from commit 5a37942c61)
This commit adds an endpoint `/mobile_push/e2ee/test_notification`
to send an end-to-end encrypted test push notification to the user's
selected mobile device or all of their mobile devices.
We can determine whether the request is meant to revoke an already
sent push notification using the "type" field of the payload.
Passing `is_removal` parameter explicitly to `send_push_notifications`
is not required.
This commit updates the data that gets encrypted to be
the same on both android and iOS.
The data and its format is almost the same as what we send
as FCM payload to android clients with no E2EE support,
changes are:
For send push notification payload:
* 'realm_id`, 'server', 'sender_email', and 'realm_uri' fields
don't exist in the new payload.
* 'event' field renamed to 'type'
* 'stream' and 'stream_id' fields renamed to 'channel_name'
and 'channel_id' respectively.
* The value of 'recipient_type' will be 'channel' & 'direct'
instead of 'stream' & 'private' respectively.
* 'zulip_message_id' field renamed to 'message_id'
For remove push notification payload:
* 'realm_id`, 'server', and 'realm_uri' fields don't exist
in the new payload.
* 'event' field renamed to 'type'
* 'zulip_message_ids' field renamed to 'message_ids' and it's
value will be a JSON array instead of a string.
In the existing iOS client, we have no code of our own involved
in constructing the notifications in the UI, and instead we
leave it to the iOS SDK to do so.
Since, for clients with E2EE support the data is going to be
interpreted by our own code, not by the iOS SDK - we are free
to keep the same data and format.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit removes the 'zulip_message_id' field from the FCM
payload for remove push notification.
Clients that use `zulip_message_ids` instead have been out for
years and we no longer support older client versions that don't.
This reverts 1862c3d9ab.
The "time" field was added in the APNs payload to log
"Remote queuing latency".
The client doesn't use this field. Ideally, we should have
removed the field from the payload before sending it to APNs.
We no longer log "Remote queuing latency", so it's safe to
remove it from the APNs payload.
Only FCM payload will have this field as it is used by android
clients.
The function was used to reformat apns payload sent
by servers older than 613d093d7.
Since we no longer need to worry about servers older
than 1.6.0, it's safe to remove that function.
This commit replaces the `PUSH_NOTIFICATION_REDACT_CONTENT` server
setting with `require_e2ee_push_notifications` realm setting.
If `require_e2ee_push_notifications` set to True:
* Older clients: Content redacted
* Updated clients: Encrypted content
If `require_e2ee_push_notifications` set to False:
* Older clients: Content NOT redacted
* Updated clients: Encrypted content
Note: Older clients refers to clients that don't support E2EE.
Fixes part of #35370.
This commit adds a test and updates a few existing tests to
cover more cases related to send push notifications.
* We no longer mock the 'send_push_notifications_legacy' function
while testing 'send_push_notifications' codepath and vice-versa.
This makes the tests more realistic as both functions gets called
in 'handle_push_notification'.
This covers the case when only old clients (which don't support
E2EE) exists for a user. Or only updated clients (which supports
E2EE) exist.
* Adds a test 'test_both_old_and_new_client_coexists' for the case
when a user has both type of clients at an instant i.e. they have
updated a few devices only.
In 'send_push_notifications_legacy', when a user has no registered
devices:
* `uses_notification_bouncer()`=True: we log "Skipping..." and return.
* `uses_notification_bouncer()`=False: we make some function calls,
which effectively does nothing.
It's better to have a common check (and log) early in the codepath.
This commit makes that change.
This commit removes the DEVELOPMENT guard from the new
'send_push_notifications' function in 'handle_push_notification'
and 'handle_remove_push_notification'.
Earlier, we were passing a map `device_id_to_encrypted_data`
and http headers as separate fields to bouncer.
The downside of that approach is it restricts the bouncer to
process only one type of notice i.e. either notification for
a new message or removal of sent notification, because it
used to receive a fixed priority and push_type for all the
entries in the map.
Also, using map restricts the bouncer to receive only one
request per device_id. Server can't send multiple notices
to a device in a single call to bouncer.
Currently, the server isn't modelled in a way to make a
single call to the bouncer with:
* Both send-notification & remove-notification request data.
* Multiple send-notification request data to the same device.
This commit replaces the old protocol of sending data with
a list of objects where each object has the required data
for bouncer to send it to FCM or APNs.
This makes things a lot flexible and opens possibility for
server to batch requests in a different way if we'd like to.
'get_apns_payload_data_to_encrypt' was added in commit
0ae34ddb65, in parallel
to 'get_message_payload_apns' - to use in E2EE codepath.
The intent was to avoid nesting in the payload returned
by 'get_message_payload_apns' function, just like FCM
payload returned by 'get_message_payload_gcm'.
Turned out, the nesting is helpful in APNs case for various
reasons. So, this commit reverts that function and we'll
continue to use the older structure returned by the function
'get_message_payload_apns'.
This commit updates 'handle_remove_push_notification' function
to use the new 'send_push_notifications' function.
It leads to encrypt the removal payload before sending it to bouncer.
Fixes part of #35368.
This commit adds support to let server configure:
* fcm_priority
* apns_priority
* apns_push_type
while sending E2EE push notifications.
The values of these fields will vary depending on whether the
send request is to send push notification for a message or
revoke an already sent notification.
Since, the bouncer receives encrypted data so it can't inspect
the payload to determine whether it is a removal request or not,
hence can't configure priority on its own.
The server needs to specify explicitly.
We're not simply sending a single 'is_removal' flag because
allowing the server to configure them separately will help in
future to support other types of notifications with a different
combination of priority and push_type, like whose aim is to notify
user about information other than a new message or removal request.
Fixes part of #35368.
This commit adds support to send encrypted push notifications
to devices registered to receive encrypted notifications.
URL: `POST /api/v1/remotes/push/e2ee/notify`
payload: `realm_uuid` and `device_id_to_encrypted_data`
The POST request needs to be authenticated with the server’s
API key.
Note: For Zulip Cloud, a background fact about the push bouncer is
that it runs on the same server and database as the main application;
it’s not a separate service. So, as an optimization we directly call
'send_e2ee_push_notifications' function and skip the HTTP request.
APNs apparently treats its tokens case-insensitively; FCM does not.
Adjust the `unique_together` to instead be separate partial
constraints, keyed on the `kind` of the PushDeviceToken.
APNs tokens are provided by the client in hex, and we store them in
hex. The existing code which attempts to "validate" them by parsing
them as base64 only works because base64 is a superset of hex.
Enforce that APNs tokens are hex, and remove all of the pieces of test
code which were incorrectly passing them in as base64 strings.
This commit adds a `push_devices` dictionary to
`POST /register` response, keyed with push account ID,
where each entry describes the user's push device's
registration status and error code (if registration failed).
This commit adds a zilencer endpoint to let self-hosted
servers register push devices to whom mobile push notifications
will be sent.
POST "/api/v1/remotes/push/e2ee/register"
Payload: realm_uuid, push_account_id, encrypted_push_registration,
bouncer_public_key
The post request needs to be authenticated with the server’s API key.
Note: For Zulip Cloud, a background fact about the push bouncer is
that it runs on the same server and database as the main application;
it’s not a separate service.
So, as an optimization, we plan to directly call the
`do_register_remote_push_device` function and skip the HTTP request.
Fixeszulip/zulip-flutter#1617.
It turns out that an APNs token (which is a hex string) is equally
valid in lower or upper case. The old app would send the server
the lower-case form of the token, but the new app sends the
upper-case form.
Because we've been treating tokens case-sensitively, if the user
upgrades from the old app to the new, that results in the server
and bouncer each having two copies of the token (one lower-case and
one upper-case), and therefore sending that device two copies of
each notification: zulip/zulip-flutter#1617.
To fix that immediately, have the bouncer drop duplicate tokens
before sending the notifications to APNs.
Work is also in progress on fixing this in a better-structured way,
by having the database correctly treat tokens as the same when they
differ only in case.
To maintain API compatibility during migration to DirectMessageGroup
for 1:1 DMs, generate notification payloads for such messages in the
same format as those sent to a Personal recipient.
Fixes: part of issue #25713.
In aioapns 4.0, which we upgraded to in acd7353538, the `key`
parameter is treated as the ASCII-armored string contents of the key,
not the path to the file with that content.
Read the file ourselves, and pass the string it expects.
This is valuable so that one is forced to explicitly make a decision
on what is correct when adding new callers. Past experience tells us that
not having to explicitly show the decision leads to people introducing
security bugs in PRs that the maintainer has to catch in review, and our
goal for access control code should be that security bugs are hard to write.
Fixes#33688.
Now that we introduced an URL for serving permalinks redirecting to
docs in #33444, the docs_url mechanism is no longer needed, as we can
have a URL that's safe to hard-code in register_server.py.
The HostnameAlreadyInUseBouncerError.docs_url has been merged in main
briefly enough, that this should be safe to remove.
This commit adds support to display `Message.EMPTY_TOPIC_FALLBACK_NAME`
value (translated) in the push notifications for topics having the
actual value of empty string.
Fixes part of #32996.
This means that the URL is only hard-coded on the bouncer side. That's
useful, because now we'll be able to change the URL and only need a
bouncer deployment for users to get the new URL when they encounter
HostnameAlreadyInUseBouncerError. As opposed to self-hosted servers
being stuck with an outdated docs link hardcoded in their
register_server.py.
An even better way than the current json error message recommending the
--registration-transfer option is to return an appropriate error code
and have that get picked up by the register_server command.
The register_server command can then display a more comprehensive,
better formatted error message with proper whitespaces and a pointer to
the documentation.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.
It also renames variables and methods of type -
"huddle_message" to "group_direct_message".
This is a part of #28640
