Just like with signup confirmation links, we shouldn't trigger email
change based on a GET to the confirmation URL - POST should be required.
So upon GET of the confirmation link, we serve a form which will
immediately be POSTed by JS code to finalize the email change.
The `push_registration_encryption_keys` map stores the
assymetric key pair generated on bouncer.
The public key will be used by the client to encrypt
registration data and the bouncer will use the corresponding
private key to decrypt.
- Updated the `generate_secrets.py` script to generate the map
during installation in dev environment.
- Added a management command to add / remove key i.e. use it
for key rotation while retaining the older key-pair for a period
of time.
When a user is added to a channel, we send
the user that was added a Notification Bot
DMs to let them know about it.
In this commit, we add an option for whether or not
this message is sent.
If more than 100 users are added at once, we
do not send notification bot DMs since it would
be a performance-costly operation.
We also send this threshold value of 100 in the
initial state data to the clients.
Fixes part of #31189
This pulls in changes from the latest django-jinja[^1]
`makemessages.py` monkey-patching. Specifically, it adds support for
`trimmed`, `notrimmed`, and the `ext.i18n.trimmed` policy. We enable
that, which removes unsightly and unnecessary whitespace inside of
`{% trans %}` blocks.
[^1]: aac828ca63/django_jinja/management/commands/makemessages.py
Server can now send partial data to the client to help in
developement. We don't want this to be widely used right now,
hence no documentation changes have been made.
This will likely be a check on client capability later.
Using postfix to handle the incoming email gateway complicates things
a great deal:
- It cannot verify that incoming email addresses exist in Zulip before
accepting them; it thus accepts mail at the `RCPT TO` stage which it
cannot handle, and thus must reject after the `DATA`.
- It is built to handle both incoming and outgoing email, which
results in subtle errors (1c17583ad5, 79931051bd, a53092687e,
#18600).
- Rate-limiting happens much too late to avoid denial of
service (#12501).
- Mis-configurations of the HTTP endpoint can break incoming
mail (#18105).
Provide a replacement SMTP server which accepts incoming email on port
25, verifies that Zulip can accept the address, and that no
rate-limits are being broken, and then adds it directly to the
relevant queue.
Removes an incorrect comment which implied that missed-message
addresses were only usable once. We leave rate-limiting to only
channel email addresses, since missed-message addresses are unlikely
to be placed into automated systems, as channel email addresses are.
Also simplifies #7814 somewhat.
This extracts the core general group-syncing logic, which is independent
of any LDAP-specific concerns, to a separate function. This allows the
use of this core logic for group sync in other authentication backends.
The code also gets a bit cleaned up in the process to make it more
readable (with some readability tweaks to the log strings as well).
This adds a new API endpoint that enables users to report messages for
review by admins or moderators. Reports will be sent to the
`moderate_request_channel`, so it must be configured for this feature to
be enabled.
Fixes part of #20047.
Co-authored-by: Adam Sah <140002+asah@users.noreply.github.com>
The `push.{EXTERNAL_HOST}` formula effectively assumes that the host
in EXTERNAL_HOST is zulipdev.com -- it's relying on the fact that
push.zulipdev.com is in DNS as an alias of zulipdev.com. That's a
special fact that wouldn't be true of most hostnames.
It especially isn't true of IP addresses. If one has set
EXTERNAL_HOST to a value like `192.168.0.2:9991` -- for example in
order to reach the dev server from a separate machine, such as a
physical mobile device:
https://github.com/zulip/zulip-mobile/blob/main/docs/howto/dev-server.md
then setting ZULIP_SERVICES_URL to `http://push.192.168.0.2:9991`
is definitely not going to work. For example:
https://chat.zulip.org/#narrow/channel/243-mobile-team/topic/notifications.20from.20dev.20server/near/2159863
In the dev server, the point with ZULIP_SERVICES_URL is that it
should point back to the same dev server. So use a formula that
says that more directly and so more reliably.
Tests are a different matter: we want a distinct URL there, because
we'll be inspecting the URLs in requests. But in tests there's also
no requirement that the two hosts have anything to do with each
other; so the existing formula is fine there, and keep it in place.
(In tests it'd probably be better to use proper RFC 6761 test
domains, like `chat.example.com` and `push.example.net`. But
that's an independent question.)
This was broken, due the mechanism simply using our
is_guest/is_realm_admin/etc. role setters, but failing to adjust system
group memberships - resulting in corrupted database state.
We need to ensure that change_user_role is called for setting user role.
There are two relevant codepaths that run the sync based on
AUTH_LDAP_USER_FLAGS_BY_GROUP and thus need to get this right:
1. manage.py sync_ldap_user_data
2. Just-in-time user creation when a user without a Zulip account logs
in for the first using their ldap credentials. After
get_or_build_user returns, django-auth-ldap sees that the user
account has just been created, and proceeds to run ._populate_user().
Now that both user.save() and do_change_user_realm will be getting
called together, we need to ensure this always happens atomically.
This imposes the need to override _get_or_create_user to put it in a
transaction. The troublesome consequence is that this new
`atomic(savepoint=False)` causes the usual type of issue, where tests
testing error get their transaction rolled back and cannot continue
executing.
To get around that, we add a test helper
`artificial_transaction_savepoint` which allows these tests to wrap
their problematic blocks in an artificial transaction which provides a
savepoint, thus preventing the full test transaction rollback derailing
the rest of the test.
Without these overrides, we cannot test the functionality in DEVELOPMENT
and TESTING.
There are two codepaths that we're covering here:
1. The sync which happens via `sync_ldap_user_data`.
2. The sync which happens during just-in-time user creation upon first
login via ldap.
Both codepaths end up triggering ldap_user._get_or_create_user().
Sending emails synchronously is useful because it reports
configuration errors -- but it also means that occasional failures can
result in ugly 500's, since those don't retry.
Add a setting which forces all email to go through the `emil_senders`
queue, so it can be retried as needed.
We should not key off of `S3_KEY`/`S3_SECRET_KEY`, since those are
optional if the host is in EC2 and using instance profiles. Instead,
check if `LOCAL_UPLOADS_DIR` is None1, which is the authoritative
source for if the S3 backend is in use.
Legacy settings contained type "Dict" which were removed in zulip 9.0,
so this type was wrong, but it also serves no purpose.
(The non-commented types are checked in the development environment).
To zulip/python-zulip-api, to keep them closer to their source code.
- Renamed the generate_zulip_bots_static_files to
generate_bots_integrations_static_files to accomodate the new function.
- Added a new function to
tools/setup/generate_bots_integrations_static_files to copy the
integration docs into static/generated/integrations.
- Updated integrations.py and computed_settings.py to use the new doc
paths.
- Deleted the affected integration docs.
- Updated the dependency URL.
Apparently, while we set our own maximum password length of 100
characters, zxcvbn had a hardcoded maximum length of 72 characters,
and threw an exception if that was exceeded.
The fact that we're discovering this now would suggest that nobody has
previously attempted a password between 72 and 100 characters in
length.
This commit adds a one-time modal to display navigation tour
video to new users.
Includes an `NAVIGATION_TOUR_VIDEO_URL` server-setting to specify
the video's URL. When set to None, the modal is not displayed.
Fixes#29304.
Email clients tend to sort emails by the "Date" header, which is not
when the email was received -- emails can be arbitrarily delayed
during relaying. Messages without a Date header (as all Zulip
messages previously) have one inserted by the first mailserver they
encounter. As there are now multiple email-sending queues, we would
like the view of the database, as presented by the emails that are
sent out, to be consistent based on the Date header, which may not be
the same as when the client gets the email in their inbox.
Insert a Date header of when the Zulip system inserted the data into
the local queue, as that encodes when the full information was pulled
from the database. This also opens the door to multiple workers
servicing the email_senders queue, to limit backlogging during large
notifications, without having to worry about inconsistent delivery
order between those two workers.
Messages which are sent synchronously via `send_email()` get a Date
header of when we attempt to send the message; this is, in practice,
no different from Django's default behaviour of doing so, but makes
the behaviour slightly more consistent.
This is more consistent with how other URLs work in Zulip.
Replaces `/message_edit_typing` with `/messages/{message_id}/typing`.
The `message_id` parameter, previously passed in the request body,
is now included in the URL path.
Adds a second Zoom integration that uses the Zoom Server to Server
OAuth app process. Only one of the two Zoom integrations can be
configured on a Zulip server.
Adds a cache for the access token from the Zoom server so that it
can be used by the server to create meetings for the approximate
duration of the access token
In the web-app compose box, if the user's delivery email does not
match a user on the configured Zoom account for the server to server
integration, then a compose box error banner will be shown when the
error response is received after clicking/selecting the video or
audio call button.
Also updates the production documentation for the both types of Zoom
integration apps (Server to Server and General). The General app
process for Zoom now requires unlisted apps to go through their
review process, which we now have documented.
Fixes#33117.
These two endpoints make up ~85% of requests to Zulip servers; since
presence is also a performance-critical endpoint, having them share
the same cache increases how hot it stays in memcached, in addition to
making the presence endpoint faster.
This comes at the very slightly cost of one extra field. Checks
for the `is_bot` column are switched to the equivalent `bot_type`
check, since the columns are slightly duplicative, and we can get away
with only checking bot_type.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them. Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.
In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user. However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each. This leads to a query which is over 12k of text long, and
results in 319 columns. In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.
We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry. That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.
Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object. The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
This commit adds typing indicators for message editing in stream
as well as in dm, if the send typing notification
for corresponding is enabled.
Based on earlier work in #28585.
Co-authored-by: Rohan Gudimetla <rohan.gudimetla07@gmail.com>
Fixes#25719.
