Server can now send partial data to the client to help in
developement. We don't want this to be widely used right now,
hence no documentation changes have been made.
This will likely be a check on client capability later.
Using postfix to handle the incoming email gateway complicates things
a great deal:
- It cannot verify that incoming email addresses exist in Zulip before
accepting them; it thus accepts mail at the `RCPT TO` stage which it
cannot handle, and thus must reject after the `DATA`.
- It is built to handle both incoming and outgoing email, which
results in subtle errors (1c17583ad5, 79931051bd, a53092687e,
#18600).
- Rate-limiting happens much too late to avoid denial of
service (#12501).
- Mis-configurations of the HTTP endpoint can break incoming
mail (#18105).
Provide a replacement SMTP server which accepts incoming email on port
25, verifies that Zulip can accept the address, and that no
rate-limits are being broken, and then adds it directly to the
relevant queue.
Removes an incorrect comment which implied that missed-message
addresses were only usable once. We leave rate-limiting to only
channel email addresses, since missed-message addresses are unlikely
to be placed into automated systems, as channel email addresses are.
Also simplifies #7814 somewhat.
This extracts the core general group-syncing logic, which is independent
of any LDAP-specific concerns, to a separate function. This allows the
use of this core logic for group sync in other authentication backends.
The code also gets a bit cleaned up in the process to make it more
readable (with some readability tweaks to the log strings as well).
This adds a new API endpoint that enables users to report messages for
review by admins or moderators. Reports will be sent to the
`moderate_request_channel`, so it must be configured for this feature to
be enabled.
Fixes part of #20047.
Co-authored-by: Adam Sah <140002+asah@users.noreply.github.com>
The `push.{EXTERNAL_HOST}` formula effectively assumes that the host
in EXTERNAL_HOST is zulipdev.com -- it's relying on the fact that
push.zulipdev.com is in DNS as an alias of zulipdev.com. That's a
special fact that wouldn't be true of most hostnames.
It especially isn't true of IP addresses. If one has set
EXTERNAL_HOST to a value like `192.168.0.2:9991` -- for example in
order to reach the dev server from a separate machine, such as a
physical mobile device:
https://github.com/zulip/zulip-mobile/blob/main/docs/howto/dev-server.md
then setting ZULIP_SERVICES_URL to `http://push.192.168.0.2:9991`
is definitely not going to work. For example:
https://chat.zulip.org/#narrow/channel/243-mobile-team/topic/notifications.20from.20dev.20server/near/2159863
In the dev server, the point with ZULIP_SERVICES_URL is that it
should point back to the same dev server. So use a formula that
says that more directly and so more reliably.
Tests are a different matter: we want a distinct URL there, because
we'll be inspecting the URLs in requests. But in tests there's also
no requirement that the two hosts have anything to do with each
other; so the existing formula is fine there, and keep it in place.
(In tests it'd probably be better to use proper RFC 6761 test
domains, like `chat.example.com` and `push.example.net`. But
that's an independent question.)
This was broken, due the mechanism simply using our
is_guest/is_realm_admin/etc. role setters, but failing to adjust system
group memberships - resulting in corrupted database state.
We need to ensure that change_user_role is called for setting user role.
There are two relevant codepaths that run the sync based on
AUTH_LDAP_USER_FLAGS_BY_GROUP and thus need to get this right:
1. manage.py sync_ldap_user_data
2. Just-in-time user creation when a user without a Zulip account logs
in for the first using their ldap credentials. After
get_or_build_user returns, django-auth-ldap sees that the user
account has just been created, and proceeds to run ._populate_user().
Now that both user.save() and do_change_user_realm will be getting
called together, we need to ensure this always happens atomically.
This imposes the need to override _get_or_create_user to put it in a
transaction. The troublesome consequence is that this new
`atomic(savepoint=False)` causes the usual type of issue, where tests
testing error get their transaction rolled back and cannot continue
executing.
To get around that, we add a test helper
`artificial_transaction_savepoint` which allows these tests to wrap
their problematic blocks in an artificial transaction which provides a
savepoint, thus preventing the full test transaction rollback derailing
the rest of the test.
Without these overrides, we cannot test the functionality in DEVELOPMENT
and TESTING.
There are two codepaths that we're covering here:
1. The sync which happens via `sync_ldap_user_data`.
2. The sync which happens during just-in-time user creation upon first
login via ldap.
Both codepaths end up triggering ldap_user._get_or_create_user().
Sending emails synchronously is useful because it reports
configuration errors -- but it also means that occasional failures can
result in ugly 500's, since those don't retry.
Add a setting which forces all email to go through the `emil_senders`
queue, so it can be retried as needed.
We should not key off of `S3_KEY`/`S3_SECRET_KEY`, since those are
optional if the host is in EC2 and using instance profiles. Instead,
check if `LOCAL_UPLOADS_DIR` is None1, which is the authoritative
source for if the S3 backend is in use.
Legacy settings contained type "Dict" which were removed in zulip 9.0,
so this type was wrong, but it also serves no purpose.
(The non-commented types are checked in the development environment).
To zulip/python-zulip-api, to keep them closer to their source code.
- Renamed the generate_zulip_bots_static_files to
generate_bots_integrations_static_files to accomodate the new function.
- Added a new function to
tools/setup/generate_bots_integrations_static_files to copy the
integration docs into static/generated/integrations.
- Updated integrations.py and computed_settings.py to use the new doc
paths.
- Deleted the affected integration docs.
- Updated the dependency URL.
Apparently, while we set our own maximum password length of 100
characters, zxcvbn had a hardcoded maximum length of 72 characters,
and threw an exception if that was exceeded.
The fact that we're discovering this now would suggest that nobody has
previously attempted a password between 72 and 100 characters in
length.
This commit adds a one-time modal to display navigation tour
video to new users.
Includes an `NAVIGATION_TOUR_VIDEO_URL` server-setting to specify
the video's URL. When set to None, the modal is not displayed.
Fixes#29304.
Email clients tend to sort emails by the "Date" header, which is not
when the email was received -- emails can be arbitrarily delayed
during relaying. Messages without a Date header (as all Zulip
messages previously) have one inserted by the first mailserver they
encounter. As there are now multiple email-sending queues, we would
like the view of the database, as presented by the emails that are
sent out, to be consistent based on the Date header, which may not be
the same as when the client gets the email in their inbox.
Insert a Date header of when the Zulip system inserted the data into
the local queue, as that encodes when the full information was pulled
from the database. This also opens the door to multiple workers
servicing the email_senders queue, to limit backlogging during large
notifications, without having to worry about inconsistent delivery
order between those two workers.
Messages which are sent synchronously via `send_email()` get a Date
header of when we attempt to send the message; this is, in practice,
no different from Django's default behaviour of doing so, but makes
the behaviour slightly more consistent.
This is more consistent with how other URLs work in Zulip.
Replaces `/message_edit_typing` with `/messages/{message_id}/typing`.
The `message_id` parameter, previously passed in the request body,
is now included in the URL path.
Adds a second Zoom integration that uses the Zoom Server to Server
OAuth app process. Only one of the two Zoom integrations can be
configured on a Zulip server.
Adds a cache for the access token from the Zoom server so that it
can be used by the server to create meetings for the approximate
duration of the access token
In the web-app compose box, if the user's delivery email does not
match a user on the configured Zoom account for the server to server
integration, then a compose box error banner will be shown when the
error response is received after clicking/selecting the video or
audio call button.
Also updates the production documentation for the both types of Zoom
integration apps (Server to Server and General). The General app
process for Zoom now requires unlisted apps to go through their
review process, which we now have documented.
Fixes#33117.
These two endpoints make up ~85% of requests to Zulip servers; since
presence is also a performance-critical endpoint, having them share
the same cache increases how hot it stays in memcached, in addition to
making the presence endpoint faster.
This comes at the very slightly cost of one extra field. Checks
for the `is_bot` column are switched to the equivalent `bot_type`
check, since the columns are slightly duplicative, and we can get away
with only checking bot_type.
Tornado requests try hard to not make SQL queries -- and if they're
necessary, to minimize the number of them. Specifically, both session
objects and user objects are cached in memcached, and we expect that
both of them will have been filled there by Django before any requests
are made to Tornado.
In the event that memcached is flushed, or data is otherwise evicted,
we perform two database queries -- one for the session, and one for
the user. However, the *width* of the latter query has grown
significantly over time, as the Realm object grew more fields, and
recently with the addition of role groups, which require multiple
joins each. This leads to a query which is over 12k of text long, and
results in 319 columns. In the event of a memcached flush, this can
result in a *significant* amount of SQL traffic, as nearly every
active Tornado request will make that query.
We do not wish to narrow the default query for Django; we instead tag
the request in the REST wrapper, and use that to use a much narrower
user cache entry. That narrower cache entry is filled before the
queue is created in Django; we also use it to explicitly set the log
data, so the second "half" of the continued Tornado request does not
need to fetch any user data either when writing its log line.
Because they use different cache keys, this only affects the
session-based `/json/events` endpoint, which caches by user-id; the
`/api/v1/events` endpoint, which uses an API-key cache, keeps its wide
user object. The former is 50% of the total request volume, whereas
the latter is only 2%, so adding an additional cache for it is
unnecessary complexity.
This commit adds typing indicators for message editing in stream
as well as in dm, if the send typing notification
for corresponding is enabled.
Based on earlier work in #28585.
Co-authored-by: Rohan Gudimetla <rohan.gudimetla07@gmail.com>
Fixes#25719.
This will gives us the flexibility to edit our documentation, including
section names, without worrying about breaking links to docs hard-coded
in older releases / deployed self-hosted servers.
The functionality of gravatar can break anonymity if
the user has had a gravatar account set up previously.
This option allows specifically cloud instances to
have gravatar disabled selectively.
This is the final naming that we want, compared to the naming we merged
in #32399.
Includes renaming the API endpoints, but that should be fine as the
original PR was just merged and this isn't deployed anywhere.
If the server controls the registration's hostname, it can reclaim its
registration credentials. This is useful, because self-hosted admins
frequently lose the credentials when moving their Zulip server to a
different machine / deployment method.
The flow is the following:
1. The host sends a POST request to
/api/v1/remotes/server/register/takeover.
2. The bouncer responds with a signed token.
3. The host prepares to serve this token at /api/v1/zulip-services/verify and
sends a POST to /remotes/server/register/verify_challenge endpoint of
the bouncer.
4. Upon receiving the POST request, the bouncer GETS
https://{hostname}/api/v1/zulip-services/verify, verifies the secret and
responds to the original POST with the registration credentials.
5. The host can now save these credentials to it zulip-secrets.conf file
and thus regains its push notifications registration.
Includes a global rate limit on the usage of the /verify_challenge
endpoint, as it causes us to make outgoing requests.
Fixes#32199
We only need a log in button since that will take users to
"/accounts/go" if we are on a non-realm specific URL.
"/accounts/go" already has link to go to "Find accounts" page.
