This commit adds a zilencer endpoint to let self-hosted
servers register push devices to whom mobile push notifications
will be sent.
POST "/api/v1/remotes/push/e2ee/register"
Payload: realm_uuid, push_account_id, encrypted_push_registration,
bouncer_public_key
The post request needs to be authenticated with the server’s API key.
Note: For Zulip Cloud, a background fact about the push bouncer is
that it runs on the same server and database as the main application;
it’s not a separate service.
So, as an optimization, we plan to directly call the
`do_register_remote_push_device` function and skip the HTTP request.
The 'user_uuid' parameter of 'get_remote_realm_helper' was only
used for logging when realm lookup fails, but we've never made
use of that detail in practice.
There is no strong reason to keep that.
This commit adds a `RemotePushDevice` model where each row
corresponds to an account on an install of the app registered
to receive mobile push notifications.
The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.
The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.
- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
This commit updates code to use "\x07" as value for
"subject" field of Message objects for DMs and group
DMs, so that we have a unique value for DMs and group
DMs which cannot be used for channel messages.
This helps in avoiding having an empty string value as
topic for DMs, which is also used for "general chat"
channel messages, as large number of DMs in the realm
resulted in PostgreSQL query planner thinking that there
are too many "general chat" messages and thus generated
bad query plans for operations like fetching
"general chat" messages in a stream or moving messages
to and from "general chat" topic.
This change as done for ArchivedMessage and
ScheduledMessage objects as well.
Note that the clients still get "subject" value as
an empty string "".
This commit also adds tests for checking that "\x07"
cannot be used as topic for channel messages.
Fixes#34360.
This commit:
* Creates a migration to rename any existing Client with
name="populate_db" to "ZulipDataImport".
* Updates populate_db.py to use ZulipDataImport for new
message creation
These changes should make code to identify imported messages
considerably more readable.
Fixes#33909.
This adds a new API endpoint that enables users to report messages for
review by admins or moderators. Reports will be sent to the
`moderate_request_channel`, so it must be configured for this feature to
be enabled.
Fixes part of #20047.
Co-authored-by: Adam Sah <140002+asah@users.noreply.github.com>
We introduced `is_channel_message` flag in 90d76b692b.
While populating messages in dev environment, we were not setting
it to False for DMs / group DMs.
This commit fixes that bug.
This commit updates populate_db to mark all onboarding steps as seen
for existing users to avoid unnecessary popups during development.
Developers should create a new user in development environment
to test the onboarding steps.
Email clients tend to sort emails by the "Date" header, which is not
when the email was received -- emails can be arbitrarily delayed
during relaying. Messages without a Date header (as all Zulip
messages previously) have one inserted by the first mailserver they
encounter. As there are now multiple email-sending queues, we would
like the view of the database, as presented by the emails that are
sent out, to be consistent based on the Date header, which may not be
the same as when the client gets the email in their inbox.
Insert a Date header of when the Zulip system inserted the data into
the local queue, as that encodes when the full information was pulled
from the database. This also opens the door to multiple workers
servicing the email_senders queue, to limit backlogging during large
notifications, without having to worry about inconsistent delivery
order between those two workers.
Messages which are sent synchronously via `send_email()` get a Date
header of when we attempt to send the message; this is, in practice,
no different from Django's default behaviour of doing so, but makes
the behaviour slightly more consistent.
An even better way than the current json error message recommending the
--registration-transfer option is to return an appropriate error code
and have that get picked up by the register_server command.
The register_server command can then display a more comprehensive,
better formatted error message with proper whitespaces and a pointer to
the documentation.
This is the final naming that we want, compared to the naming we merged
in #32399.
Includes renaming the API endpoints, but that should be fine as the
original PR was just merged and this isn't deployed anywhere.
Users most likely to run into this will be the ones who are moving to a
new server, but keeping their original domain and thus just need to
transfer the registration.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.
The flow is the following:
1. The host sends a POST request to
/api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
sends a POST to /remotes/server/register/verify_challenge endpoint of
the bouncer.
4. Upon receiving the POST request, the bouncer GETS
https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
and thus regains its push notifications registration.
Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
Renames migrate_customer_to_legacy_plan to
create_complimentary_access_plan for how this
function is currently used.
Prep for adding a complimentary access plan
for Zulip Cloud.
As we can now add a complimentary access plan (formerly legacy plan)
to a remote server or remote realm via the support view, we no
longer use or need this management command for our billing admin.
If we move a paid plan from a remote server to a remote realm, and
the plan has automated license management, then we create an updated
license ledger entry when we move the plan for the remote realm
billing data so that we have an accurate user count for licenses
when the plan is next invoiced.
In commit ea863bab5b, handle_customer_migration_from_server_to_realm
was updated to only move a remote server's plan in the case that there
is only one remote realm on the server.
This commit renames the 'queue_json_publish' function to
'queue_json_publish_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.
In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
Fixes part of #30489.
Python parameter defaults are only evaluated once at the function
definition site, not at each call site. So these defaults were
incorrectly evaluating to the server’s startup time rather than the
current time.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds `durable=True` to the outermost db transactions
created in the following:
* confirm_email_change
* handle_upload_pre_finish_hook
* deliver_scheduled_emails
* restore_data_from_archive
* do_change_realm_subdomain
* do_create_realm
* do_deactivate_realm
* do_reactivate_realm
* do_delete_user
* do_delete_user_preserving_messages
* create_stripe_customer
* process_initial_upgrade
* do_update_plan
* request_sponsorship
* upload_message_attachment
* register_remote_server
* do_soft_deactivate_users
* maybe_send_batched_emails
It helps to avoid creating unintended savepoints in the future.
This is as a part of our plan to explicitly mark all the
transaction.atomic calls with either 'savepoint=False' or
'durable=True' as required.
* 'savepoint=True' is used in special cases.
This commit adds 'durable=True' to the outermost transaction
in 'remote_server_post_analytics'.
It also adds 'savepoint=False' to inner transaction.atomic
decorator to avoid creating savepoint.
This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.
* 'savepoint=True' is used in special cases.
This commit adds 'durable=True' to the outermost transactions
of the following functions:
* do_create_multiuse_invite_link
* do_revoke_user_invite
* do_revoke_multi_use_invite
* sync_ldap_user_data
* do_reactivate_remote_server
* do_deactivate_remote_server
* bulk_handle_digest_email
* handle_customer_migration_from_server_to_realm
* add_reaction
* remove_reaction
* deactivate_user_group
It helps to avoid creating unintended savepoints in the future.
This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.
* 'savepoint=True' is used in special cases.
We create an unnamed user group with just the group creator as it's
member when trying to set the default. The pattern I've followed across
most of the acting_user additions is to just put the user declared
somewhere before the check_add_user_group and see if the test passes.
If it does not, then I'll look at what kind of user it needs to be set
to `acting_user`.
Reorders audit log string methods to have the following pattern:
"event_type event_time (id): modified_object". And the event type
is the name for the AuditLogEventType enum.
Created using manage.py squashmigrations with a couple changes:
- Patched Django to optimize AddConstraints/RemoveConstraints properly.
- Used the StateOperations from the
migrations.SeparateDatabaseAndState section in migration 0060, so
that the constraint changes could be optimized properly.
- Removed dependencies on zerver, since this project does not actually
have any dependencies on zerver migrations.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.
The field is also used to narrow a new user in DM with
welcome bot on the first load.
This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
