For get and filter queries of NamedUserGroup, realm_for_sharding
field is used instead of realm field, as directly using
realm_for_sharding field on NamedUserGroup makes the query faster
than using realm present on the base UserGroup table.
Until now we were not documenting bouncer's REST API endpoints.
We plan to document the newly introduced "remotes/push/e2ee/register"
and "remotes/push/e2ee/notify" endpoints.
This commit does the prep work for documenting bouncer endpoints:
* mark the older endpoints related to sending non-E2EE push
notifications as "intentionally_undocumented" - we'll remove
them in future.
* the remaining endpoints are marked pending-to-document with
helpful comments.
Refactoring, no functional change.
This commit refactors `send_e2ee_push_notification_apple`
and `send_e2ee_push_notification_android` to return a
`SentPushNotificationResult` dataclass.
It's a cleaner protocol than passing a mutable data structure
`delete_device_ids` as argument and updating it within
functions.
Fixes part of #35368.
We enforce at registration time that hostnames with deactivated=False
must be unique, though we do not enforce it in the database because we
have historical data for which that is not true.
However, we should perform all hostname lookups with a
`deactivated=False` filter, as that will resolve the majority of
`RemoteZulipServer.MultipleObjectsReturned` errors. Once the
production data is cleaned up, this can be enforced via a partial
unique index.
Adds a visible example to the PATCH /channel_folders endpoint for
reordering an organization's channel folders. So that the example
works in a normal dev environment, we also add a second channel
folder when populating the dev database.
Earlier, we were passing a map `device_id_to_encrypted_data`
and http headers as separate fields to bouncer.
The downside of that approach is it restricts the bouncer to
process only one type of notice i.e. either notification for
a new message or removal of sent notification, because it
used to receive a fixed priority and push_type for all the
entries in the map.
Also, using map restricts the bouncer to receive only one
request per device_id. Server can't send multiple notices
to a device in a single call to bouncer.
Currently, the server isn't modelled in a way to make a
single call to the bouncer with:
* Both send-notification & remove-notification request data.
* Multiple send-notification request data to the same device.
This commit replaces the old protocol of sending data with
a list of objects where each object has the required data
for bouncer to send it to FCM or APNs.
This makes things a lot flexible and opens possibility for
server to batch requests in a different way if we'd like to.
This commit adds support to let server configure:
* fcm_priority
* apns_priority
* apns_push_type
while sending E2EE push notifications.
The values of these fields will vary depending on whether the
send request is to send push notification for a message or
revoke an already sent notification.
Since, the bouncer receives encrypted data so it can't inspect
the payload to determine whether it is a removal request or not,
hence can't configure priority on its own.
The server needs to specify explicitly.
We're not simply sending a single 'is_removal' flag because
allowing the server to configure them separately will help in
future to support other types of notifications with a different
combination of priority and push_type, like whose aim is to notify
user about information other than a new message or removal request.
Fixes part of #35368.
This commit adds support to send encrypted push notifications
to devices registered to receive encrypted notifications.
URL: `POST /api/v1/remotes/push/e2ee/notify`
payload: `realm_uuid` and `device_id_to_encrypted_data`
The POST request needs to be authenticated with the server’s
API key.
Note: For Zulip Cloud, a background fact about the push bouncer is
that it runs on the same server and database as the main application;
it’s not a separate service. So, as an optimization we directly call
'send_e2ee_push_notifications' function and skip the HTTP request.
APNs apparently treats its tokens case-insensitively; FCM does not.
Adjust the `unique_together` to instead be separate partial
constraints, keyed on the `kind` of the PushDeviceToken.
APNs tokens are provided by the client in hex, and we store them in
hex. The existing code which attempts to "validate" them by parsing
them as base64 only works because base64 is a superset of hex.
Enforce that APNs tokens are hex, and remove all of the pieces of test
code which were incorrectly passing them in as base64 strings.
This commit adds a `PushDevice` model where each row
corresponds to an account on an install of the app
that has attempted to register with the bouncer to
receive mobile push notifications.
This is the core server table storing registrations
that are potentially registered with the mobile push
notifications bouncer service.
This commit adds a zilencer endpoint to let self-hosted
servers register push devices to whom mobile push notifications
will be sent.
POST "/api/v1/remotes/push/e2ee/register"
Payload: realm_uuid, push_account_id, encrypted_push_registration,
bouncer_public_key
The post request needs to be authenticated with the server’s API key.
Note: For Zulip Cloud, a background fact about the push bouncer is
that it runs on the same server and database as the main application;
it’s not a separate service.
So, as an optimization, we plan to directly call the
`do_register_remote_push_device` function and skip the HTTP request.
The 'user_uuid' parameter of 'get_remote_realm_helper' was only
used for logging when realm lookup fails, but we've never made
use of that detail in practice.
There is no strong reason to keep that.
This commit adds a `RemotePushDevice` model where each row
corresponds to an account on an install of the app registered
to receive mobile push notifications.
The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.
The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.
- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
This commit updates code to use "\x07" as value for
"subject" field of Message objects for DMs and group
DMs, so that we have a unique value for DMs and group
DMs which cannot be used for channel messages.
This helps in avoiding having an empty string value as
topic for DMs, which is also used for "general chat"
channel messages, as large number of DMs in the realm
resulted in PostgreSQL query planner thinking that there
are too many "general chat" messages and thus generated
bad query plans for operations like fetching
"general chat" messages in a stream or moving messages
to and from "general chat" topic.
This change as done for ArchivedMessage and
ScheduledMessage objects as well.
Note that the clients still get "subject" value as
an empty string "".
This commit also adds tests for checking that "\x07"
cannot be used as topic for channel messages.
Fixes#34360.
This commit:
* Creates a migration to rename any existing Client with
name="populate_db" to "ZulipDataImport".
* Updates populate_db.py to use ZulipDataImport for new
message creation
These changes should make code to identify imported messages
considerably more readable.
Fixes#33909.
This adds a new API endpoint that enables users to report messages for
review by admins or moderators. Reports will be sent to the
`moderate_request_channel`, so it must be configured for this feature to
be enabled.
Fixes part of #20047.
Co-authored-by: Adam Sah <140002+asah@users.noreply.github.com>
We introduced `is_channel_message` flag in 90d76b692b.
While populating messages in dev environment, we were not setting
it to False for DMs / group DMs.
This commit fixes that bug.
This commit updates populate_db to mark all onboarding steps as seen
for existing users to avoid unnecessary popups during development.
Developers should create a new user in development environment
to test the onboarding steps.
Email clients tend to sort emails by the "Date" header, which is not
when the email was received -- emails can be arbitrarily delayed
during relaying. Messages without a Date header (as all Zulip
messages previously) have one inserted by the first mailserver they
encounter. As there are now multiple email-sending queues, we would
like the view of the database, as presented by the emails that are
sent out, to be consistent based on the Date header, which may not be
the same as when the client gets the email in their inbox.
Insert a Date header of when the Zulip system inserted the data into
the local queue, as that encodes when the full information was pulled
from the database. This also opens the door to multiple workers
servicing the email_senders queue, to limit backlogging during large
notifications, without having to worry about inconsistent delivery
order between those two workers.
Messages which are sent synchronously via `send_email()` get a Date
header of when we attempt to send the message; this is, in practice,
no different from Django's default behaviour of doing so, but makes
the behaviour slightly more consistent.
An even better way than the current json error message recommending the
--registration-transfer option is to return an appropriate error code
and have that get picked up by the register_server command.
The register_server command can then display a more comprehensive,
better formatted error message with proper whitespaces and a pointer to
the documentation.
This is the final naming that we want, compared to the naming we merged
in #32399.
Includes renaming the API endpoints, but that should be fine as the
original PR was just merged and this isn't deployed anywhere.
Users most likely to run into this will be the ones who are moving to a
new server, but keeping their original domain and thus just need to
transfer the registration.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.
The flow is the following:
1. The host sends a POST request to
/api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
sends a POST to /remotes/server/register/verify_challenge endpoint of
the bouncer.
4. Upon receiving the POST request, the bouncer GETS
https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
and thus regains its push notifications registration.
Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
Renames migrate_customer_to_legacy_plan to
create_complimentary_access_plan for how this
function is currently used.
Prep for adding a complimentary access plan
for Zulip Cloud.
As we can now add a complimentary access plan (formerly legacy plan)
to a remote server or remote realm via the support view, we no
longer use or need this management command for our billing admin.
If we move a paid plan from a remote server to a remote realm, and
the plan has automated license management, then we create an updated
license ledger entry when we move the plan for the remote realm
billing data so that we have an accurate user count for licenses
when the plan is next invoiced.
