Commit Graph

3762 Commits

Author SHA1 Message Date
Tim Abbott
e9f889d57e portico: Fix totally broken ToS re-accept form styling.
This form isn't actively used, which is how it ended up broken, but it
basically didn't display its content properly at all.

Convert it to use our standard white-box framework.

This still doesn't look great in various ways, but it's at least not
obviously totally busted now.
2018-12-17 07:28:29 -08:00
Tim Abbott
8a90441d2f slack import: Import long-inactive users as long-term idle.
This avoids creating UserMessage rows for long-inactive users in
organizations with many thousands of users.
2018-12-16 18:52:20 -08:00
Tim Abbott
142d9cb63f report: Allow error-reporting views from unauthed users.
This should make it possible for blueslip error reports to be sent on
our logged-out portico pages, which should in turn make it possible to
debug any such issues as they occur.
2018-12-16 15:44:48 -08:00
ishanrai05
4105fb683b notifications: Optimize push notifications code path in tests.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.

This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.

This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.

Tweaked by tabbott to fix a few small issues.

Fixes #10895.
2018-12-15 11:12:43 -08:00
Tim Abbott
a63eae48cc test_push_notifications: Fix leak that can leak to test flakes.
While reviewing #11012, I discovered a nondeterministic result for
test_signup, which I tracked down to specifically this triple of tests
failing when run in this order:

test-backend GCMSuccessTest \
  zerver.tests.test_push_notifications.TestAPNs.test_get_apns_client \
  zerver.tests.test_signup.LoginTest.test_register

with a query count mismatch like this:

expected length: 73
actual length: 79

Comparing the list of queries, it's clear that test_register was
seeing `push_notifications_enabled()` returning True in this test order.

It's not clear why GCMSuccessTest was required here (it was!), but
further debugging determined the problem was that
`test_get_apns_client` left the _apns_client initialization system in
a state where get_apns_client would return a non-None value, resulting
in push_notifications_enabled() returning True for future tests.

The immediate fix is to just reset the `_apns_client` and
`_apns_client_initializedstate` state properly after the test runs;
but arguably we should do a larger refactor to make this less
fragile.
2018-12-15 11:12:43 -08:00
Rishi Gupta
111eda604b portico: Add /atlassian for Zulip users migrating from HipChat. 2018-12-14 23:42:47 -08:00
Tim Abbott
8c30c36006 realm filters: Expand set of characters allowed in prefixes.
Our list of allowed characters in realm filter patterns has long been
too string; fix this by extending the pattern.

Also, extend the tests to have examples of actual strings one would
use with the patterns, for clarity.

Fixes #10953, fixes #6835.
2018-12-14 11:33:13 -08:00
Tim Abbott
626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott
0a5221a819 ldap: Extract dev_ldap_directory.py.
This gets what is fundamentally unit testing code out of backends.py.
2018-12-13 16:24:15 -08:00
Tim Abbott
5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Rishi Gupta
8a95526ced billing: Always transition to Realm.LIMITED via do_change_plan_type.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.

Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
2018-12-13 13:26:24 -08:00
Rishi Gupta
b245c661da billing: Change do_change_plan_type to take a realm instead of a user.
More often than not, changes in plan type are not directly due to user
action.
2018-12-13 13:26:24 -08:00
Vishnu Ks
8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott
ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Tim Abbott
8a11c94a2d fakeldap: Add thumbnailPhoto/jpegPhoto for testing avatar syncing. 2018-12-12 11:23:33 -08:00
Roman Godov
9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott
15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Max Nussenbaum
536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Jack Zhang
be9b6a6dee compatibility: Add a compatibility check to api_get_server_settings.
This should make it convenient for the mobile app to present errors of
the form "Your Zulip app is not new enough for this Zulip server".
2018-12-10 15:06:09 -08:00
Jack Zhang
8711fe7bae compatibility: Respond with HTTP 400 when no User-Agent header.
Prior to this commit, a request to /compatibility without a User-Agent
header would result in the server 500ing.
2018-12-10 15:04:34 -08:00
Tim Abbott
a1d99adce7 auth: Handle SSO_APPEND_DOMAIN in remote_user SSO for mobile.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.

Fixes #11005.
2018-12-10 10:36:15 -08:00
Marco Burstein
ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht
a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
Joshua Pan
441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott
5c2e7a169f test_events: Use a better assertion when comparing event count. 2018-12-06 17:43:26 -08:00
Tim Abbott
e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott
9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Tim Abbott
9da4b58c8e docs: Fix /help/ sidebar being accidentally missing.
Apparently, we didn't have a test for this, and it was broken in a
recent refactoring.
2018-12-06 10:13:51 -08:00
Vishnu Ks
18f90754da portico: Change wording of finding organization in /accounts/go. 2018-12-06 09:24:22 -08:00
Tim Abbott
9e06ab45bf webhooks: Fix HttpResponse with notify_bot_owner_on_invalid_json.
Apparently, there was a bug in notify_bot_owner_on_invalid_json, where
we didn't reraise the JsonableError.

We fix this with a refactoring that makes the exception layering
clearer as well.
2018-12-05 15:17:31 -08:00
Greg Price
32c6b81b3a compatibility: Add test data from real server logs.
In a quick scan of today's nginx logs on chat.zulip.org, there
were 20 distinct user-agents that begin with 'ZulipMobile/'.
Here's a representative sampling of them, such that the rest
were all boringly similar to one of these.

First, to make room for these without an excess of copy-paste and
overlong lines, convert this test to a data-oriented style.  The
existing, synthetic cases appear in the new data followed by the
seen-in-the-wild cases.

Happily, the code being tested passes all these new cases unchanged.
2018-12-05 14:58:13 -08:00
Greg Price
fb7bfbe9ab compatibility: Require version 16.2.96+ for our Android app.
This release is from 2018-08-22, a little over 100 days ago.

It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null".  We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.

The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
2018-12-05 14:58:13 -08:00
Greg Price
0fa4fdcff9 compatibility: Rewrite verbose test for compact legibility.
This will facilitate adding a bunch more test cases shortly.
2018-12-05 14:58:13 -08:00
Greg Price
2bac6de6c6 compatibility: Add and test a helper find_mobile_os. 2018-12-05 14:58:13 -08:00
Greg Price
1859faccb7 compatibility: Treat empty version as unknown, not old. 2018-12-05 14:58:13 -08:00
Greg Price
ebfa4b746d compatibility: Make 1.2 < 1.2.3; rewrite version_lt internals. 2018-12-05 14:58:13 -08:00
Greg Price
557aca2aa7 compatibility: Implement a version comparator. 2018-12-05 14:58:13 -08:00
rht
e59ff6e6db slack import: Eliminate need to load all messages into memory.
This works by yielding messages sorted based on timestamp.  Because
the Slack exports are broken into files by date, it's convenient to do
a 2-layer sorting process, where we open all the files for a given
day, and then sort their messages by timestamp before yielding them.

Fixes #10930.
2018-12-05 12:20:50 -08:00
Tim Abbott
380231af9d push_notifications: Add tests for BrokenPipeError case.
This was missing in d723dbfef7.
2018-12-05 10:44:25 -08:00
Tim Abbott
b47535d8bb push notifications: Fix exception when handling deleted messages.
If a user deletes message between when it triggered a potential push
notification for another user, and when that notification was actually
sent, we'd end up with a situation where the `Message` table didn't
have an entry for the requested message ID.

The clean fix for this is to still throw an exception in the event
that the message doesn't exist at all, but if it exists in
ArchivedMessage, don't throw a user-facing exception.
2018-12-05 10:38:37 -08:00
Tim Abbott
196ec78261 logging_handlers: Fix unnecessary broken import. 2018-12-04 18:18:07 -08:00
Anders Kaseorg
9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Vishnu Ks
fc4b6d8f5e auth: Redirect login to realm_redirect page instead of find_accounts. 2018-12-04 09:35:35 -08:00
Vishnu Ks
788b98d041 portico: Add page for redirecting to a realm subdomain. 2018-12-04 09:35:35 -08:00
Raymond Akornor
92dc3637df send_email: Add support for multiple recipients.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.

We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.

Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.

Fixes #10896.
2018-12-03 15:12:11 -08:00
Tim Abbott
0fddf9a610 docs: Fix 500s with super long fake /help/ article URLs.
While this would never happen for a real article, this prevents a 500
in this case for a situation which is definitely user error and should
be a 40x (in this case, 404).

As part of this, we refactor the main view code to do validation in a
single code path, since the semi-duplicated-in-3-places logic was
getting pretty buggy.
2018-12-03 14:43:45 -08:00
Vishnu Ks
f3033207b9 emails: Cover all LDAP cases when emailing login details.
This provides a nice user experience for folks where we do know what
their LDAP credentials are.

Though we need to fix #10917 before the content in the email with be
correct.
2018-11-29 12:14:22 -08:00
Rohitt Vashishtha
c4e50a34d3 bugdown: Refactor get_user to get_user_by_name.
Also adds a warning against the use of this function.
2018-11-29 10:19:08 -08:00
Hemanth V. Alluri
6b1d724f5c zerver: Add bugdown rendering for text custom profile fields.
This is the first step of letting users use Zulip markdown in their
SHORT_TEXT and LONG_TEXT custom profile fields, so that they can
include emphasis, links, etc.

This doesn't include any frontend logic yet, however.
2018-11-29 09:47:32 -08:00
Rohitt Vashishtha
681368b937 bugdown: Refactor get_possible_mentions_info and related functions.
This commit changes the return type of get_possible_mentions_info  to a
list instead of a dict, thus disposing off the hacky logic of storing
users with duplicate full names with name|id keys that made the code
obfuscated.

The other functions continue to use the dicts as before, however, there
are minor variable changes where needed in accordance with the updated
definition of get_possible_mentions_info.
2018-11-28 14:07:52 -08:00