Commit Graph

7703 Commits

Author SHA1 Message Date
Greg Price
528a76a419 push_notifs: Set APNs topic individually on each notification.
This makes it possible to send notifications to more than one app ID
from the same server: for example, the main Zulip mobile app and the
new Flutter-based app, which has a separate app ID for use through its
beta period so that it can be installed alongside the existing app.
2023-11-16 12:03:30 -08:00
Sahil Batra
dc98136346 streams: Send stream deletion events on unsubscribing users.
This commit adds code to send stream deletion events when
unsubscribing non-admin users from private streams and
when unsubscribing guests from public streams since
non-admins cannot access unsubscribed private streams
and guests cannot access unsubscribed public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra
d4fb244d2d CVE-2023-47642: Invalid metadata access for formerly subscribed streams.
It was discovered by the Zulip development team that active users who
had previously been subscribed to a stream incorrectly continued being
able to use the Zulip API to access metadata for that stream. As a
result, users who had been removed from a stream, but still had an
account in the organization, could still view metadata for that
stream (including the stream name, description, settings, and an email
address used to send emails into the stream via the incoming email
integration). This potentially allowed users to see changes to a
stream’s metadata after they had lost access to the stream.

This bug was present in all Zulip releases prior to today's Zulip
Server 7.5.
2023-11-16 13:25:33 -05:00
Sahil Batra
3c8701ee36 streams: Add API endpoint to get stream email.
This commit adds new API endpoint to get stream email which is
used by the web-app as well to get the email when a user tries
to open the stream email modal.

The stream email is returned only to the users who have access
to it. Specifically for private streams only subscribed users
have access to its email. And for public streams, all non-guest
users and only subscribed guests have access to its email.
All users can access email of web-public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra
432001656e streams: Remove "email_address" field from Subscription objects.
This commit removes "email_address" field from Subscription objects
and we would instead a new endpoint in next commit to get email
address for stream with proper access check.

This change also fixes the bug where we would include email address
for the unsubscribed private stream as well when user did not have
permission to send message to the stream, and having email allowed
the unsubscribed user to send message to the stream.

Note that the unsubscribed user can still send message to the stream
if the user had noted down the email before being unsubscribed
and the stream token is not changed after unsubscribing the user.
2023-11-16 13:25:33 -05:00
Mateusz Mandera
1819b85b85 management: Allow changing is_billing_admin using change_user_role. 2023-11-15 18:36:07 -08:00
Anders Kaseorg
cb9a04d3e3 test_create_video_call: Add missing not None assertions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Tim Abbott
2e2997bd7d typing: Limit typing notifications in large streams. 2023-11-15 09:42:25 -08:00
Alex Vandiver
70b20e9d2b markdown: Use \p{White_Space} equivalent for linkifier boundaries.
We do not use \p{White_Space} itself because re2 does not support it.
2023-11-14 20:43:39 -08:00
Sahil Batra
8f79cec51a users: Pass bogus data for inaccessible users.
We now pass bogus data for inaccessible users when sending
the users data in "realm_users" field of "register" response
or when using endpoints like "GET /users" to get data of
all the users in realm.

We would add a client capability field in future commits
such that new clients would receive data only for accessible
users and they can form the bogus data by themselves.
2023-11-13 08:04:45 -08:00
Sahil Batra
c82bb3ec76 models: Add can_access_all_users_group setting.
This commit adds new setting for controlling who can access
all users in the realm which would have "Everyone" and
"Members only" option.

Fixes part of #10970.
2023-11-13 08:04:45 -08:00
Mateusz Mandera
48db4bf854 counts: Add new mobile_pushes RemoteRealmCount stats.
This requires a bit of complexity to avoid a name collision in
COUNT_STATS with the RemoteInstallationCount stats with the same name.
2023-11-10 16:09:11 -08:00
Mateusz Mandera
8a6d5b4997 counts: Add new Add new mobile_pushes_sent::day LoggingCountStat.
This is a CountStat for tracking how many mobile notifications the
server requested.
1. On a self-hosted server, that means requesting from the push bouncer.
2. On a server that's its own push bouncer, that's just the number
   directly sent.

This number has room for inaccuracy due to incrementing by the number of
user devices on a self-hosted server, as it doesn't account for errors
that may occur in the GCM/APNs low-level sending codepaths on the bouncer.

Also tests that a server that's its own push bouncer correctly
increments its mobile_pushes_sent::day CountStat, by basing it on the
values returned from the send_apple/android_push_notification functions
which tell us the actual number of successfully sent notifications.

Since the return values of send_..._push_notification are now
used in those codepaths, we need to tweak our mocks in some unrelated
tests to set up some return value to avoid errors.
2023-11-10 16:09:11 -08:00
Prakhar Pratyush
c597de6a1d topic_mentions: Rename wildcard_mentioned to stream_wildcard_mentioned.
Rename the existing 'wildcard_mentioned' flag to
'stream_wildcard_mentioned'.

The 'wildcard_mentioned' flag is deprecated and exists for
backwards compatibility.

We have two separate flags for stream and topic wildcard mentions,
i.e., 'stream_wildcard_mentioned' and 'topic_wildcard_mentioned',
respectively.

* stream wildcard mentions: `@all`, `@everyone`, and `@stream`
* topic wildcard mentions: `@topic`

The `wildcard_mentioned` flag is included in the events and
API response if either `stream_wildcard_mentioned` or
`topic_wildcard_mentioned` is set.
2023-11-10 11:06:26 -08:00
Prakhar Pratyush
24fa361f40 migration: Clear old data for unused usermessage flags.
In c37871ac3a, we renamed the
two unused and historical bits of the 'flags' bitfield of
the 'UserMessage' table:

* 'summarize_in_home' to 'topic_wildcard_mentioned'
* 'summarize_in_stream' to 'group_mentioned'

This commit clears out the old data for those bits.

Additionally, we are clearing 'force_expand' and 'force_collapse'
unused flags to save future work.
2023-11-10 11:06:26 -08:00
Mateusz Mandera
1312c7ccd7 zilencer: Add mechanism to update RemoteRealm when Realm is changed.
This requires a migration to allow RemoteRealmAuditLog.remote_id to be
NULL, and to add a RemoteRealmAuditLog.remote_realm.
2023-11-08 15:54:22 -08:00
Mateusz Mandera
76e0511481 zilencer: Add new model RemoteRealm and send the data to the bouncer.
Add the new model for recording basic information about Realms on remote
server, to go with the other analytics data. Also adds necessary changes
to the bouncer endpoint and the send_analytics_to_push_bouncer()
function to submit such Realm information.
2023-11-08 15:54:22 -08:00
roanster007
dc492867af user_mention: Fix mentions of deactivated users.
Previously, when a deactivated user was mentioned, he wasn't
rendered as a Pill. This is because the dataset for validating mentions
only included active users, which is fixed by removing that filter.

To allow only silent mentions of them, an extra is_active property
added to FullNameInfo class, which is populated from the query,
which tells if user is deactivated. This is used to convert any
mentions of them to silent mentions in the backend markdown.

Fixes #26857
2023-11-08 09:48:31 -08:00
Sahil Batra
deb8431a47 users: Rename get_raw_user_data to get_users_for_api.
This is a prep commit for adding feature of restricting
user access to guests such that we can keep the code
easy to read and understand when that feature is added.
2023-11-08 09:22:26 -08:00
Greg Price
7a0ebc0d18 push_notifs: Log an error if ios_app_id missing on any APNs token. 2023-11-07 16:19:42 -08:00
Greg Price
f109e3b598 push_notifs: Backfill ios_app_id on bouncer. 2023-11-07 16:19:42 -08:00
Greg Price
be2a9a03d0 push_notifs: Check app ID has a plausible shape at bouncer. 2023-11-07 16:19:42 -08:00
Greg Price
1b2178f558 push_notifs: Require ios_app_id on register at bouncer, too.
We'll need this information in order to properly direct APNs
notifications.  Happily, the Zulip server always sends it when
registering an APNs token; and it appears it always has done so
since the commit:

cddee49e7 Add support infrastructure for push notification bouncer service.

back in 2016.  So there's no compatibility issue from requiring it.
2023-11-07 16:19:42 -08:00
Greg Price
9c2d53bcef push_notifs: Stop dropping ios_app_id on floor at bouncer.
This missing `REQ` call has meant we just drop this parameter:
even though the remote Zulip server passes it (for all APNs tokens),
we never notice and never store it.  Fix that.
2023-11-07 16:19:42 -08:00
Greg Price
9072ddda05 push_notifs: Remove remaining references to ZULIP_IOS_APP_ID setting. 2023-11-07 16:19:42 -08:00
Greg Price
a9796ec503 push_notifs: Check app ID has a plausible shape. 2023-11-07 16:19:42 -08:00
Greg Price
fb3af7fbcb push_notifs: Make appid required on add_apns_device_token.
We're going to need to use this information, so we shouldn't just
assume a value; the client should tell us the actual value.

Conveniently, the Zulip mobile app does already pass this parameter
and has since forever.  So we can just start requiring it, with no
compatibility constraint.
2023-11-07 16:19:42 -08:00
Greg Price
5d0d35d3f5 push_notifs: Always supply appid to add_apns_device_token in tests.
We already always pass this parameter from the mobile client,
so this makes the tests more realistic already.  And we'll shortly
be making this parameter required.
2023-11-07 16:19:42 -08:00
Greg Price
3893a70888 push_notifs: Cut spurious token_kind arguments in some test data.
These endpoints distinguish the token kinds by having different
URL paths, and don't take any argument by this name.
2023-11-07 16:19:42 -08:00
Prakhar Pratyush
94679d590f email_notification: Include prior message context only when mentioned.
Earlier, email message notifications included prior messages sent
to the same topic for context. This is more confusing than helpful
for messages that the user is likely to have received notifications
for all the prior messages in the conversation already (or read them
in the Zulip UI).

Now, we include prior context only when the user is mentioned via
personal, group, stream or topic wildcard mention.

Fixes #27479.
2023-11-07 11:44:38 -08:00
Prakhar Pratyush
c0f445294c test_message_notif_email: Verify email has multiple stream messages.
This commit improves the test to explicitly verify that multiple
messages that were sent in quick succession to a topic are included
in the email body when we have email notifications enabled for a
given stream.

Earlier, the test was only verifying the email subject and the fact
that only one email was sent.

It is important to verify the fact that all the messages sent to a
topic in quick succession should be included in the email body.
2023-11-07 11:44:38 -08:00
Prakhar Pratyush
e6e156709a typing_notifications: Don't notify long_term_idle subscribers.
The event for stream typing notifications is no longer sent
to the long_term_idle subscribers of the stream.

This helps to reduce the tornado's work of parsing super-long
JSON-encoded lists of user IDs in large streams. Now the lists
are shorter.
2023-11-07 09:30:27 -08:00
Aman Agrawal
be6f467f42 home: Add sponsorship_pending page_param.
This will be used in gear menu to inform admin of their
sponsorship application status.

This includes some additional tweaks for when to show
billing and plans to users.
2023-11-06 14:55:14 -08:00
Aman Agrawal
00df9c6c18 urls: Show sponsorship page on a new URL.
Fixes #27377

Replaced #sponsorship links with sponsorship/ links and in other
places liking gear menu.
2023-11-06 14:55:14 -08:00
Aman Agrawal
ddab0e6cd2 test_user_groups: Fix flaky tests.
There order of group ids doesn't matter here and thus the
compared values can have the ids in different order and test
should still pass. So, using `set` for comparing unordered
lists seems like the right fix here.
2023-11-04 12:46:06 -07:00
Prakhar Pratyush
b0ef76bf27 topic_mentions: Set 'topic_wildcard_mentioned' flag on @topic mention.
Earlier, the 'wildcard_mentioned' flag was set for both the
stream and topic wildcard mentions.

Now, the 'topic_wildcard_mentioned' flag is set for topic
wildcard mentions, and the 'wildcard_mentioned' flag is set for
stream wildcard mentions.

We will rename the 'wildcard_mentioned' flag to
'stream_wildcard_mentioned' in a later commit.
2023-11-02 09:25:51 -07:00
Prakhar Pratyush
c37871ac3a user_message: Rename unused flags and create an index.
This commit renames the two unused and historical bits of the
'fields' bitfield of the 'UserMessage' and 'ArchivedUserMessage'
tables.

* 'summarize_in_home' to 'topic_wildcard_mentioned'
* 'summarize_in_stream' to 'group_mentioned'

The 'group_mentioned' flag doesn't affect the feature,
but completing the work here helps to save future migration
and indexing efforts on the UserMessage table, as we plan to
use this flag in the future for group mentions.

The unused bits may have old data; we'll clear that in
a separate commit.

It creates the 'zerver_usermessage_any_mentioned_message_id'
index concurrently.
2023-11-02 09:25:51 -07:00
Sahil Batra
bb15b2d708 users: Send "update" events when deactivating or reactivating users.
We now send "realm_user/update" (and "realm_bot/update" for bots)
events with "is_active" field when deactivating and reactivating
users, including bots.

We would want to use "remove" event for a user losing access
to another user for #10970, so it is better to use "update"
event for deactivation as we only update "is_active" field
in the user objects and the clients still have the data for
deactivated users.

Previously, we used to send "add" event for reactivation along
with complete user objects, but clients should have the data
for deactivated users as well, so an "update" event is enough
like we do when deactivating users.
2023-11-02 08:31:12 -07:00
Mateusz Mandera
6819ecee92 zilencer: Add new LoggingCountStat mobile_pushes_forwarded.
This one counts actual successful deliveries.
2023-11-01 17:26:10 -07:00
Mateusz Mandera
b7117d51b2 zilencer: Don't allow syncing mobile_pushes_received::day count. 2023-11-01 17:26:10 -07:00
Alex Vandiver
54605e54b6 tests: Switch test_i18n fetch helper to use client_foo methods.
38f2a2f475 updated the comment but not the code.  Using
`self.client.post` instead of `self.client_post` means that we do not
set the host headers correctly.
2023-11-01 11:24:12 -07:00
Sahil Batra
47c8e369cf register: Pass the configuration objects for group settings in response.
This commit adds code to pass configuration objects for group
permission settings in register response to clients such that
we do need to duplicate that data in clients and can avoid
future bugs due to inconsistency.

The "server_supported_permission_settings" field is included
in the response if "realm" is present in "fetch_event_types",
as this is what we do for other server-related fields.
2023-11-01 10:42:56 -07:00
Sahil Batra
e458b73a01 user_groups: Move constants for system group names to a new class.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
2023-11-01 10:42:56 -07:00
Sahil Batra
08d41f9eb3 settings: Rename permissions_configuration variable.
This commit renames permissions_configuration variable to
permission_configuration since the object contains config for
a single permission setting and thus permission_configuration
seems like a better name.
2023-11-01 10:42:56 -07:00
Sahil Batra
9a6cf82adc streams: Fix sending stream-related events to guests.
Previous behavior-
- Guest did not receive stream creation events for new
web-public streams.
- Guest did not receive peer_add and peer_remove events
for web-public and subscribed public streams.

This commit fixes the behavior to be -
- Guests now receive stream creation events for new
web-public streams.
- Guest now receive peer_add and peer_remove events for
web-public and subscribed public streams.
2023-10-31 10:54:21 -07:00
Sahil Batra
71b8f49614 streams: Return early if there is no change in subscriptions.
This commit updates code in bulk_remove_subscriptions and
bulk_add_subscriptions to return early if there are no
subscribers to remove or add to the streams.

This change helps us in avoiding unnecessary queries like the
one used to get subscribers list of streams, which is then used
to send events but we would not send any events if no subscribers
are added or removed and some more similar queries.
2023-10-31 10:54:21 -07:00
Sahil Batra
cdd15b4a69 test_events: Fix typo in comments. 2023-10-31 10:54:21 -07:00
Hemant Umre
d27a50852e onboarding: Internationalize the introductory messages in streams.
We use `Realm.default_language` value, which is set by selecting
the 'Organization language', to internationalize the introductory
messages of the initial streams.

Fixes #25729.
2023-10-26 16:27:35 -07:00
Hemant Umre
ac1f711fef registration: Set the organization language at creation time.
In this commit, we add a new dropdown 'Organization language' on
the `/new` and `/realm/register` pages. This dropdown allows setting
the language of the organization during its creation. This allows
messages from Welcome Bot and introductory messages in streams to be
internationalized.

Fixes a part of #25729.
2023-10-26 16:27:35 -07:00
Sahil Batra
c9fccf476a settings: Rename default_view and escape_navigates_to_default_view.
This commit renames default_view and escape_navigates_to_default_view
settings to web_home_view and web_escape_navigates_to_home_view in
database and API to match with our recent renaming of user facing
strings related to this.

We also rename the variables, functions, comments in code and class
names and IDs for elements related to this.
2023-10-26 11:00:12 -07:00