zulip/docs/production/mobile-push-notifications.md

# Mobile push notification service

Zulip's iOS and Android [mobile apps](https://zulip.com/apps/) support
receiving push notifications from Zulip servers to notify users when
new messages have arrived. This is an important feature for having a
great mobile app experience.

Google's and Apple's security model for mobile push notifications does not allow
self-hosted Zulip servers to directly send mobile notifications to the Zulip
mobile apps. The Zulip Mobile Push Notification Service solves this problem by
forwarding mobile push notifications generated by your server to the Zulip
mobile apps.

## Signing up

:::{important}

The Zulip Server 10.0+ [installer](install.md#step-2-install-zulip)
includes a `--push-notifications` flag that automates this
registration process.

These instructions apply to Zulip 9.0+. If you are running an older
version of Zulip ([check](https://zulip.com/help/view-zulip-version)
if you are unsure), see the [Zulip 8.x
documentation](https://zulip.readthedocs.io/en/8.4/production/mobile-push-notifications.html).
:::

You can enable the mobile push notification service for your Zulip
server as follows:

1. Make sure your server has outgoing HTTPS access to the public Internet. If
   that is restricted by a proxy, you will need to [configure Zulip to use your
   outgoing HTTP proxy](deployment.md#customizing-the-outgoing-http-proxy)
   first.

1. Set `ZULIP_SERVICE_PUSH_NOTIFICATIONS = True` in your
   `/etc/zulip/settings.py` file. Simply uncomment the appropriate line [in
   settings.py][update-settings-docs] by deleting the initial `# `.

1. Decide whether to share usage statistics with the Zulip team.

   By default, Zulip installations using the Mobile Push Notification Service
   submit additional usage statistics that help Zulip's maintainers allocate
   resources towards supporting self-hosted installations
   ([details](#uploading-usage-statistics)). You can disable submitting usage
   statistics now or at any time by setting
   `ZULIP_SERVICE_SUBMIT_USAGE_STATISTICS=False` in `/etc/zulip/settings.py`
   (uncomment the appropriate line).

   Note that all systems using the service upload [basic
   metadata](#uploading-basic-metadata) about the organizations hosted
   by the installation.

   [update-settings-docs]: ../production/upgrade.md#updating-settingspy-inline-documentation

1. [Restart your Zulip server](settings.md#changing-server-settings) so that
   your configuration changes take effect.

1. Run the registration command. If you installed Zulip directly on the server
   (without Docker), run as root:

   ```
   su zulip -c '/home/zulip/deployments/current/manage.py register_server'
   ```

   Or if you're using Docker, run:

   ```
   docker exec -it -u zulip <container_name> /home/zulip/deployments/current/manage.py register_server
   ```

   This command will print the registration data it would send to the Mobile
   Push Notification Service, ask you to accept the terms of service, and if
   you accept, register your server. If you have trouble, [contact Zulip
   support](https://zulip.com/help/contact-support) with the output of this
   command.

1. Organizations with more than 10 users must upgrade their
   [plan](https://zulip.com/plans/) in order to access the Mobile Push
   Notification Service. See [plan management](#plan-management) for details.

1. If you or your users have already set up the Zulip mobile app, you'll each
   need to log out of the mobile app, and log back in again in order to start
   getting push notifications.

Congratulations! You've successfully set up the service. You can now test mobile
push notifications by following [these
instructions](https://zulip.com/help/mobile-notifications#testing-mobile-notifications).

## Plan management

To access the Mobile Push Notification Service, organizations with more than 10
users must upgrade to a paid [plan](https://zulip.com/plans/#self-hosted), or
the free Community plan (if
[eligible](https://zulip.com/help/self-hosted-billing#free-community-plan)).
While upgrading your Zulip server to version 8.0+ makes it more convenient to
manage your plan, the same plans are offered for all Zulip versions.

### Plan management for a Zulip organization

On a self-hosted Zulip server running Zulip 8.0+, [organization
owners](https://zulip.com/help/roles-and-permissions) and billing administrators
can conveniently access plan management from the Zulip app. See [help center
documentation](https://zulip.com/help/self-hosted-billing) for detailed
instructions.

You can add billing administrators using the `change_user_role` [management
command][management-commands], passing [the organization's
`string_id`][accessing-string-id], and the email address of the Zulip user who
should be added as a billing administrator.

```
/home/zulip/deployments/current/manage.py change_user_role -r '' username@example.com is_billing_admin
```

You can remove a user's billing administrator permissions with the `--revoke`
option:

```
/home/zulip/deployments/current/manage.py change_user_role --revoke -r '' username@example.com is_billing_admin
```

[management-commands]: ../production/management-commands.md
[accessing-string-id]: https://zulip.readthedocs.io/en/stable/production/management-commands.html#accessing-an-organization-s-string-id

### Plan management for an entire Zulip server

Servers running Zulip releases older than Zulip 8.0 can start the plan
management log in process at
<https://selfhosting.zulip.com/serverlogin/>. This option is also
available for Zulip 8.0+ servers, and makes it possible to use a
single plan for multiple organizations on one installation. See [help
center documentation](https://zulip.com/help/self-hosted-billing) for
detailed log in instructions.

You will use your server's `zulip_org_id` and `zulip_org_key` as the username
and password to access plan management. You can obtain these from
`/etc/zulip/zulip-secrets.conf` on your Zulip server, or via the following
commands:

```
/home/zulip/deployments/current/scripts/get-django-setting ZULIP_ORG_ID
/home/zulip/deployments/current/scripts/get-django-setting ZULIP_ORG_KEY
```

## Why a push notification service is necessary

Both Google's and Apple's push notification services have a security
model that does not support mutually untrusted self-hosted servers
sending push notifications to the same app. In particular, when an
app is published to their respective app stores, one must compile into
the app a secret corresponding to the server that will be able to
publish push notifications for the app. This means that it is
impossible for a single app in their stores to receive push
notifications from multiple, mutually untrusted, servers.

Zulip's solution to this problem is to provide a central push
notification forwarding service, which allows registered Zulip servers
to send push notifications to the Zulip app indirectly (through the
forwarding service).

## Security and privacy

Use of the push notification bouncer is subject to the Zulip Cloud [Terms of
Service](https://zulip.com/policies/terms), [Privacy
Policy](https://zulip.com/policies/privacy) and [Rules of
Use](https://zulip.com/policies/rules). By using push notifications, you agree
to these terms.

We've designed this push notification bouncer service with security
and privacy in mind:

- A central design goal of the Push Notification Service is to
  avoid any message content being stored or logged by the service,
  even in error cases.
- The Push Notification Service only stores the necessary metadata for
  delivering the notifications to the appropriate devices and
  otherwise operating the service:
  - The APNS/FCM tokens needed to securely send mobile push
    notifications to iOS and Android devices, one per device
    registered to be notified by your Zulip server.
  - User ID numbers generated by your Zulip server, needed to route
    a given notification to the appropriate set of mobile devices.
    These user ID numbers are opaque to the Push Notification
    Service and Kandra Labs.
  - [Basic organization metadata](#uploading-basic-metadata),
    [optional usage statistics](#uploading-usage-statistics), and
    aggregate statistics about how many push notifications are sent by
    each customer.
- The Push Notification Service receives (but does not store) the
  contents of individual mobile push notifications:

  - The numeric message ID generated by your Zulip server.
  - Metadata on the message's sender (name and avatar URL).
  - Metadata on the message's recipient (channel name + ID, topic,
    direct message recipients, etc.).
  - A timestamp.
  - The message's content.

  There's a `PUSH_NOTIFICATION_REDACT_CONTENT` setting available to
  disable any message content being sent via the push notification
  bouncer (i.e., message content will be replaced with
  `New message`). Note that this setting makes push notifications
  significantly less usable.

  We plan to
  [replace that setting with end-to-end encryption](https://github.com/zulip/zulip/issues/6954)
  which would eliminate that usability tradeoff and additionally allow
  us to not have any access to the other details mentioned in this
  section.

- All of the network requests (both from Zulip servers to the Push
  Notification Service and from the Push Notification Service to the
  relevant Google and Apple services) are encrypted over the wire with
  SSL/TLS.
- The code for the push notification forwarding service is 100% open
  source and available as part of the
  [Zulip server project on GitHub](https://github.com/zulip/zulip)
  (specifically, [here](https://github.com/zulip/zulip/tree/main/zilencer)).
- The push notification forwarding servers are professionally managed
  by a small team of security-sensitive engineers.

If you have any questions about the security model, [contact Zulip
support](https://zulip.com/help/contact-support).

### Uploading basic metadata

All Zulip installations running Zulip 8.0 or greater that are
registered for the Mobile Push Notification Service regularly upload
to the service basic metadata about the organizations hosted by the
installation. (Older Zulip servers upload these metadata only if
[uploading usage statistics](#uploading-usage-statistics) is enabled).

Uploaded metadata consists of, for each organization hosted by the
installation:

- A subset of the basic metadata returned by the unauthenticated [`GET
/server_settings` API
  endpoint](https://zulip.com/api/get-server-settings).

  The purpose of that API endpoint is to serve the minimal data
  needed by the Zulip mobile apps in order to:

  - Verify that a given URL is indeed a valid Zulip server URL
  - Present a correct login form, offering only the supported features
    and authentication methods for that organization and Zulip server
    version.

  Most of the metadata it returns is necessarily displayed to anyone
  with network access to the Zulip server on the login and signup
  pages for your Zulip organization as well.

  (Some fields returned by this endpoint, like the organization icon
  and description, are not included in uploaded metadata.)

- The [organization type](https://zulip.com/help/organization-type)
  and creation date.
- The number of user accounts with each role.

Our use of uploaded metadata is governed by the same [Terms of
Service](https://zulip.com/policies/terms) and [Privacy
Policy](https://zulip.com/policies/privacy) that covers the Mobile
Push Notification Service itself.

### Uploading usage statistics

By default, Zulip installations that register for the Mobile Push
Notification Service upload the following usage statistics. You can
disable these uploads any time by setting
`ZULIP_SERVICE_SUBMIT_USAGE_STATISTICS=False` in `/etc/zulip/settings.py`.

- Totals for messages sent and read with subtotals for various
  combinations of clients and integrations.
- Totals for active users under a few definitions (1day, 7day, 15day)
  and related statistics.

Some of the graphs on your server's [usage statistics
page](https://zulip.com/help/analytics) can be generated from these
statistics.

When enabled, usage statistics are submitted via an hourly cron
job. If you'd like to access plan management immediately after
enabling `SUBMIT_USAGE_STATISTICS=True` (the legacy form of this setting)
on a pre-8.0 Zulip server, you can run the analytics job manually via:

```
/home/zulip/deployments/current/manage.py update_analytics_counts
```

Our use of uploaded usage statistics is governed by the same [Terms of
Service](https://zulip.com/policies/terms) and [Privacy
Policy](https://zulip.com/policies/privacy) that covers the Mobile
Push Notification Service itself.

## Rate limits

The Mobile Push Notification Service API has a very high default rate
limit of 1000 requests per minute. A Zulip server makes requests to
this API every time it sends a push notification, which is fairly
frequent, but we believe it to be unlikely that a self-hosted
installation will hit this limit.

This limit is primarily intended to protect the service against DoS
attacks (intentional or otherwise). If you hit this limit or you
anticipate that your server will require sending more push
notifications than the limit permits, please [contact
support](https://zulip.com/help/contact-support).

## Updating your server's registration

Your server's registration includes the server's hostname and contact
email address (from `EXTERNAL_HOST` and `ZULIP_ADMINISTRATOR` in
`/etc/zulip/settings.py`, aka the `--hostname` and `--email` options
in the installer). You can update your server's registration data by
running `manage.py register_server` again.

If you'd like to rotate your server's API key for this service
(`zulip_org_key`), you need to use
`manage.py register_server --rotate-key` option; it will automatically
generate a new `zulip_org_key` and store that new key in
`/etc/zulip/zulip-secrets.conf`.

## Moving your registration to a new server

When migrating your Zulip deployment to a new machine, you will likely want to
retain your original registration and successfully transfer it. This is
especially important if you have an active plan for the Mobile Push
Notification Service.

The best way to preserve your registration when moving to a new server is to
copy over the credentials from the old server. These credentials are stored in
the `/etc/zulip/zulip-secrets.conf` file, specifically in the `zulip_org_id`
and `zulip_org_key` fields. After installing Zulip on the new machine, ensure
that `zulip_org_id` and `zulip_org_key` are set to the same values as on the
old server.

If you used the [official backup tool](export-and-import.md#backups)
to restore your Zulip deployment on the new machine, it will have
automatically transferred all secrets, including the registration
credentials, correctly.

### Transferring your registration if you lost the original credentials

If you have lost your original credentials, you can still transfer your Zulip
registration to a new server by following these steps:

1. Ensure Zulip is installed and accessible:

   - Install Zulip on the new machine and ensure it is fully operational.
   - The server must be accessible on the hostname associated with the original
     registration, with properly configured SSL certificates.
   - This process **will not work** if your Zulip server is on a local network
     or otherwise unreachable from the internet by our Mobile Push Notification
     Service. If that’s the case, contact
     [support@zulip.com](mailto:support@zulip.com) for assistance.

1. Run the below command to transfer your registration to the new server. This will
   execute a verification flow to prove to our Mobile Push Notification Service that
   you control the hostname and upon success, re-generate the credentials for
   using the registration and write them to the `/etc/zulip/zulip-secrets.conf` file.

   ```bash
   /home/zulip/deployments/current/manage.py register_server --registration-transfer
   ```

   Note that the `zulip_org_key` value changes in the process, and therefore if you
   still have an old server running using the service, it will lose access upon
   execution of this command.

1. Apply the changes by restarting the server:

   ```bash
   /home/zulip/deployments/current/scripts/restart-server
   ```

   Finally, [verify][verify-push-notifications] that push
   notifications are working correctly. If you encounter further
   issues, contact [support@zulip.com](mailto:support@zulip.com).

1. If you store `/etc/zulip/zulip-secrets.conf` secrets externally in
   an external configuration management tool (Ansible, etc.), or
   [backups](export-and-import.md#backups), this is a good time to
   update that configuration.

[verify-push-notifications]: https://zulip.com/help/mobile-notifications#testing-mobile-notifications

## Deactivating your server's registration

If you are deleting your Zulip server or otherwise no longer want to
use the Mobile Push Notification Service, you can deactivate your server's
registration.

1. [Cancel any paid
   plans](https://zulip.com/help/self-hosted-billing#cancel-paid-plan)
   associated with your server.

1. Run the deregistration command. If you installed Zulip directly on
   the server (without Docker), run as root:

   ```
   su zulip -c '/home/zulip/deployments/current/manage.py register_server --deactivate'
   ```

   Or if you're using Docker, run:

   ```
   docker exec -it -u zulip <container_name> /home/zulip/deployments/current/manage.py register_server --deactivate
   ```

1. Comment out the
   `ZULIP_SERVICE_PUSH_NOTIFICATIONS = True` line
   in your `/etc/zulip/settings.py` file (i.e., add `# ` at the
   start of the line), and [restart your Zulip
   server](settings.md#changing-server-settings).

If you ever need to reactivate your server's registration, [contact Zulip
support](https://zulip.com/help/contact-support).

### Pausing use of the Mobile Push Notification Service

You can temporarily stop using the Mobile Push Notification Service. Comment out
the `PUSH_NOTIFICATION_BOUNCER_URL = 'https://push.zulipchat.com'` line in your
`/etc/zulip/settings.py` file (i.e., add `# ` at the start of the line), and
[restart your Zulip server](settings.md#changing-server-settings). This approach makes it
easy to start using the service again by uncommenting the same line.

## Sending push notifications directly from your server

This section documents an alternative way to send push notifications
that does not involve using the Mobile Push Notification Service at
the cost of needing to compile and distribute modified versions of the
Zulip mobile apps.

We don't recommend this path -- patching and shipping a production
mobile app can take dozens of hours to set up even for an experienced
developer, and even more time to maintain. And it doesn't provide
material privacy benefits -- your organization's push notification
data would still go through Apple/Google's servers, just not Kandra
Labs'. But in the interest of transparency, we document in this
section roughly what's involved in doing so.

As [discussed above](#why-a-push-notification-service-is-necessary),
it is impossible for a single app in the Google or Apple
store to receive push notifications from multiple, mutually
untrusted, servers. The Mobile Push Notification Service is one of
the possible solutions to this problem.

The other possible solution is for an individual Zulip server's administrators
to build and distribute their own copy of the Zulip mobile apps, hardcoding a
key that they possess. This solution is possible with Zulip, but it requires the
server administrators to publish their own copies of the Zulip mobile apps.
There's nothing the Zulip team can do to eliminate this onerous requirement.

The main work is thus distributing your own copies of the Zulip mobile apps
configured to use APNS/FCM keys that you generate. This is not for
the faint of heart! If you haven't done this before, be warned that
one can easily spend hundreds of dollars (on things like a DUNS number
registration) and a week struggling through the hoops Apple requires
to build and distribute an app through the Apple app store, even if
you're making no code modifications to an app already present in the
store (as would be the case here). The Zulip mobile app also gets
frequent updates that you will have to either forgo or republish to
the app stores yourself.

If you've done that work, the Zulip server configuration for sending
push notifications through the new app is quite straightforward:

- Create an
  [FCM push notifications](https://firebase.google.com/docs/cloud-messaging)
  key in the Google Developer console and set `android_gcm_api_key` in
  `/etc/zulip/zulip-secrets.conf` to that key.

- In Apple's developer console, register a [token][apple-doc-token] or
  [certificate][apple-doc-cert] for sending push notifications.
  Then in `/etc/zulip/settings.py`, set `APNS_SANDBOX=False`, and:

  - If using APNs [certificate-based authentication][apple-doc-cert],
    set `APNS_CERT_FILE` to the path of your APNs certificate file.

  - If using APNs [token-based authentication][apple-doc-token],
    set `APNS_TOKEN_KEY_FILE` to the path of your APNs token key file,
    `APNS_TOKEN_KEY_ID` to the corresponding 10-character key ID, and
    `APNS_TEAM_ID` to your 10-character Apple team ID.

- Restart the Zulip server.

[apple-doc-cert]: https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/establishing_a_certificate-based_connection_to_apns
[apple-doc-token]: https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/establishing_a_token-based_connection_to_apns