When these exceptions are thrown from the request-to-bouncer functions
inside of manage.py register_server/update_analytics_counts, they
shouldn't be silenced, merely calling maybe_mark_pushes_disabled in the
background.
This results in the occurrence of the error not being shown to the
user. Failure to upload analytics data when running these commands
should result in a loud, obvious error.
Failure of running register_server before this change:
```
./manage.py register_server
This command registers your server for the Mobile Push Notifications Service.
Doing so will share basic metadata with the service's maintainers:
* This server's configured hostname: zulipdev.com:9991
* This server's configured contact email address: desdemona+admin@zulip.com
* Metadata about each organization hosted by the server; see:
<https://zulip.com/doc-permalinks/basic-metadata>
Use of this service is governed by the Zulip Terms of Service:
<https://zulip.com/policies/terms>
Do you want to agree to the Zulip Terms of Service and proceed? [Y/n]
Mobile Push Notification Service registration successfully updated!
```
The occurrence of the error is not revealed to the user. Same concern
applies to the update_analytics_counts command.
After this change:
```
./manage.py register_server
This command registers your server for the Mobile Push Notifications Service.
Doing so will share basic metadata with the service's maintainers:
<...>
Do you want to agree to the Zulip Terms of Service and proceed? [Y/n]
Traceback (most recent call last):
File "/srv/zulip/./manage.py", line 150, in <module>
execute_from_command_line(sys.argv)
File "/srv/zulip/./manage.py", line 115, in execute_from_command_line
utility.execute()
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/zulip/zerver/lib/management.py", line 97, in execute
super().execute(*args, **options)
File "/srv/zulip-venv-cache/bb36fc1fcb6d8c70a9a0bcb7bac45d78623a9ff4/zulip-py3-venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
output = self.handle(*args, **options)
File "/srv/zulip/zerver/management/commands/register_server.py", line 137, in handle
send_server_data_to_push_bouncer(consider_usage_statistics=False, raise_on_error=True)
File "/srv/zulip/zerver/lib/remote_server.py", line 453, in send_server_data_to_push_bouncer
response = send_to_push_bouncer(
File "/srv/zulip/zerver/lib/remote_server.py", line 233, in send_to_push_bouncer
raise JsonableError(msg)
zerver.lib.exceptions.JsonableError: Duplicate registration detected.
```
This is the final naming that we want, compared to the naming we merged
in #32399.
Includes renaming the API endpoints, but that should be fine as the
original PR was just merged and this isn't deployed anywhere.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.
The flow is the following:
1. The host sends a POST request to
/api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
sends a POST to /remotes/server/register/verify_challenge endpoint of
the bouncer.
4. Upon receiving the POST request, the bouncer GETS
https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
and thus regains its push notifications registration.
Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.
This commit changes just the model property, which has no API
compatibility concerns.
Fixes#28403
Uses redis to remember the last time push notifications were experienced
working. This needs to work across processes, so can't be done just in
memory.
As this is transient data that's fairly harmless to lose and thus
doesn't require the persistence benefits of the database, and we're
keeping a single "row", so don't need an entire new db table, we settle
on using redis instead of postgres. This is also consistent with how we
store other kinds of such transient data.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.
As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
The problem was that earlier this was just an uncaught JsonableError,
leading to a full traceback getting spammed to the admins.
The prior commit introduced a clear .code for this error on the bouncer
side, meaning the self-hosted server can now detect that and handle it
nicely, by just logging.error about it and also take the opportunity to
adjust the realm.push_notifications_... flags.
This fixes the exception case on the initial
`/api/v1/remotes/server/analytics/status` case. Other exceptions from
`send_to_push_bouncer` are allowed to escape.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
This is a rename of the previous
enqueue_register_realm_with_push_bouncer_if_needed but is clearer
about the fact that this will also upload audit logs if available.
Given that most of the use cases for realms-only code path would
really like to upload audit logs too, and the others would likely
produce a better user experience if they upoaded audit logs, we
should just have a single main code path here i.e.
'send_analytics_to_push_bouncer'.
We still only upload usage statistics according to documented
option, and only from the analytics cron job.
The error handling takes place in 'send_analytics_to_push_bouncer'
itself.
When a self-hosted Zulip server does a data export and then import
process into a different hosting environment (i.e. not sharing the
RemoteZulipServer with the original, we'll have various things that
fail where we look up the RemoteRealm by UUID and find it but the
RemoteZulipServer it is associated with is the wrong one.
Right now, we ask user to contact support via an error page but
might develop UI to help user do the migration directly.
This is a prep commit to return, for each remote realm, the 'uuid',
'can_push', and 'expected_end_timestamp'.
This data will be used in 'initialize_push_notifications'.
These metadata are essentially all publicily available anyway, and
making uploading them unconditional will simplify some things.
The documentation is not quite accurate in that it claims the server
will upload some metadata that is not actually uploaded yet (but will
by soon). This seems harmless.
This makes it possible for a self-hosted realm administrator to
directly access a logged-page on the push notifications bouncer
service, enabling billing, support contacts, and other administrator
for enterprise customers to be managed without manual setup.
This reduces the giant load spike at 5 minute past the hour, when all
remote servers currently attempt to submit their records.
We do not wish to slew over a full hour, because we want to ensure
that we do not hold the lock when the next hour's analytics runs. It
is also not necessary to have that much variation; 10 minutes is
picked as an arbitrary "long enough" time to spread requests over.
This is a useful helper using the same API as
send_analytics_to_push_bouncer(), but uploading only realms info. This
is useful to upload realms info without the risk of taking a long time
to process the request due to too much of the *Count analytics data.
Add the new model for recording basic information about Realms on remote
server, to go with the other analytics data. Also adds necessary changes
to the bouncer endpoint and the send_analytics_to_push_bouncer()
function to submit such Realm information.
The comment has drifted away from where it should be placed within the
code and also talks about RealmCounts specifically, while we have other
object types that this equally applies to.
The former name is kind of misleading - this function is for the remote
server to send analytics to the push bouncer. Under our usual
terminology, a "remote server" is a self-hosted Zulip server. So data is
sent FROM not TO a remote server.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Note that we use the DjangoJSONEncoder so that we have builtin support
for parsing Decimal and datetime.
During this intermediate state, the migration that creates
extra_data_json field has been run. We prepare for running the backfilling
migration that populates extra_data_json from extra_data.
This change implements double-write, which is important to keep the
state of extra data consistent. For most extra_data usage, this is
handled by the overriden `save` method on `AbstractRealmAuditLog`, where
we either generates extra_data_json using orjson.loads or
ast.literal_eval.
While backfilling ensures that old realm audit log entries have
extra_data_json populated, double-write ensures that any new entries
generated will also have extra_data_json set. So that we can then safely
rename extra_data_json to extra_data while ensuring the non-nullable
invariant.
For completeness, we additionally set RealmAuditLog.NEW_VALUE for
the USER_FULL_NAME_CHANGED event. This cannot be handled with the
overridden `save`.
This addresses: https://github.com/zulip/zulip/pull/23116#discussion_r1040277795
Note that extra_data_json at this point is not used yet. So the test
cases do not need to switch to testing extra_data_json. This is later
done after we rename extra_data_json to extra_data.
Double-write for the remote server audit logs is special, because we only
get the dumped bytes from an external source. Luckily, none of the
payload carries extra_data that is not generated using orjson.dumps for
audit logs of event types in SYNC_BILLING_EVENTS. This can be verified
by looking at:
`git grep -A 6 -E "event_type=.*(USER_CREATED|USER_ACTIVATED|USER_DEACTIVATED|USER_REACTIVATED|USER_ROLE_CHANGED|REALM_DEACTIVATED|REALM_REACTIVATED)"`
Therefore, we just need to populate extra_data_json doing an
orjson.loads call after a None-check.
Co-authored-by: Zixuan James Li <p359101898@gmail.com>
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name> while trying to connect to push notification bouncer."
This adds the X-Smokescreen-Role header to proxy connections, to track
usage from various codepaths, and enforces a timeout. Timeouts were
kept consistent with their previous values, or set to 5s if they had
none previously.
