Commit Graph

6096 Commits

Author SHA1 Message Date
Tim Abbott
642086ff65 subdomains: Pass whether root domain is available to registration.
This data is necessary to determine whether to offer the root domain
in the realm creation form.
2017-10-18 22:53:10 -07:00
Tim Abbott
0884588a5e subdomains: Add a variable for how root domain is represented.
We use Realm.SUBDOMAIN_FOR_ROOT_DOMAIN as the special name for how the
root domain is referred to as a subdomain in the code.
2017-10-18 22:47:31 -07:00
Vishnu Ks
07438b2f2c forms: Save realm_creation setting on RegistrationForm.
This will be useful for making the checking behavior depend on the
status of this form.
2017-10-18 22:40:20 -07:00
Tim Abbott
4ddb812354 check_subdomain: Remove case for user_subdomain being None.
With REALMS_HAVE_SUBDOMAINS always on, user_subdomain is never None,
so this was just dead code.
2017-10-18 22:35:07 -07:00
Tim Abbott
b850875840 mypy: Change realm.subdomain to no longer be Optional.
It hasn't been optional since we removed REALMS_HAVE_SUBDOMAINS.
2017-10-18 22:35:05 -07:00
Tim Abbott
1ab2ca5986 subdomains: Extract zerver.lib.subdomains library.
These never really belonged with the rest of zerver.lib.utils.py, and
having a separate library makes it easier to enforce full test
coverage.
2017-10-18 22:27:48 -07:00
Tim Abbott
e98ca0714b notifications: Simplify how triggers are passed around.
This removes the utterly unnecessary `triggers` dict (which always was
a dict with exactly one value True) in favor of a single field,
'trigger'.

Inspired by Kunal Gupta's work in #6659.
2017-10-18 21:42:05 -07:00
Tim Abbott
735b49e505 zilencer: Fix uniqueness rules for RemotePushDeviceToken.
Since a user could use the same installation of the Zulip mobile app
with multiple Zulip servers, correct behavior is to allow reusing the
same token with multiple Zulip servers in the RemotePushDeviceToken
model.
2017-10-18 21:24:06 -07:00
Tim Abbott
c61b6d06e5 email_mirror: Strip content before checking for empty emails.
This may fix an exception we were getting of the form:

"Error queueing internal message by emailgateway@zulip.com: Message
must not be empty".
2017-10-18 21:13:03 -07:00
Tim Abbott
31afdd2ff9 internal_prep_message: Include traceback with sending errors.
This should make it easier to debug problems.
2017-10-18 21:08:41 -07:00
derAnfaenger
02f50680cf test_messages.py: Switch misordered send_message parameters. 2017-10-18 11:55:11 -07:00
derAnfaenger
cfadb43b93 codebase: Remove multiple whitespaces after comma. 2017-10-18 10:04:23 -07:00
derAnfaenger
51c5fec0e4 embedded bots: Add BotUserStateData model. 2017-10-18 09:15:53 -07:00
Tim Abbott
f998550fb4 reactions: Add missing mypy import for Message. 2017-10-18 08:59:29 -07:00
rht
691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht
b2ad8fd747 py3: Remove all from __future__ import unicode_literals.
This was mostly used in migrations, so it's a pretty safe change.
2017-10-17 23:07:42 -07:00
Tim Abbott
64cd0c2c5a notifications: Add failing test for push notifications on muted topics.
This is some pre-work as part of #7059.
2017-10-17 22:49:33 -07:00
Tim Abbott
024c27dd3e notifications: Fix sending push notifications on muted streams.
Apparently, the stream muting feature did not take priority over the
stream_push_notify feature.  This fixes that gap.

Fixes zulip/zulip-mobile#1314.
2017-10-17 22:38:54 -07:00
Harshit Bansal
31ce96ff2a views/reactions.py: Extract create_historical_message() function. 2017-10-17 22:24:34 -07:00
Tim Abbott
d37820f39f tests: Fix test_message_edit_notifications.
This was broken in a minor way by my fix to #7031.
2017-10-17 22:12:32 -07:00
Alena Volkova
5515a075ec urls: Move the report endpoints to be API-style routes. 2017-10-17 22:05:56 -07:00
Tim Abbott
513b6d624f notifications: Fix double-sending in missedmessage_hook.
While the missedmessage_hook logic originally did a reasonably good
job of avoiding double-sending notifications, there was a corner case
it didn't handle, namely a user who had been presence-idle when a
message was sent and became also event-queue-idle as well within the
next 10 minutes.  For those users, they got a notification at message
send time, and the missedmessage_hook would deliver it a second time.

We fix this by just checking the conveniently available push_notified
and email_notified variables that indicate whether the message already
had a notification triggered.

Fixes #7031.
2017-10-17 21:54:03 -07:00
rht
1da3c400e3 realm import: Convert the authentication_methods from list to bitfield.
This properly reflects how this is stored in the DB.

Tweaked by tabbott to use a proper function.
2017-10-17 21:32:20 -07:00
Tim Abbott
3983ebce9a test_fixtures: Move test_db_status state under UUID_VAR_PATH.
This should mean that maintaining two Zulip development environments
using the same Git checkout no longer has caching problems keeping
track of the migration status.
2017-10-17 21:15:58 -07:00
Tim Abbott
b7c3b4df0c migration_status: Switch to passing the path in Python.
This is an important prerequisite to changing these paths used in
provisioning.
2017-10-17 21:15:58 -07:00
Tim Abbott
c69c38b14e export: Fix importing/exporting of user avatars.
We apparently failed to update the export code for handling what
directories avatar files should live in during the earlier process.

Fixes #7052.
2017-10-17 21:15:58 -07:00
Tim Abbott
15c0380a33 export: Fix exporting with an output file at a relative path.
Fixes #7051.
2017-10-17 20:47:29 -07:00
Eeshan Garg
46132dade2 integrations: Distinguish between static/ URLs and logo file paths.
Previously, to check whether a logo file existed, we simply took
the static/ URL for the logo and treated it as a file path. This
led to problems when static/* was not the correct parent directory
for our static files (for example, when settings.PRODUCTION = True).

Now, we treat URLs and file paths differently and the logo file
path is constructed by joining settings.STATIC_ROOT and the
relative path to the logo file.

Fixes #7018.
2017-10-17 16:26:19 -07:00
Steve Howell
6fbaf7e80f Remove sender-related fields from message cache.
This change makes the cache entries smaller for message
dictionaries.  It also ensures we get valid data put into
message dictionaries if, for example, the sender's avatar
changes.

After this change, all of the attributes for a message
sender are only fetched during post-processing with two
exceptions:

    * We get sender_id for "free" from the message,
      and it's the primary key that we need to figure
      out which data to fetch in post-processing.

    * We need sender_realm_id to be able to cache topic
      links, and a sender's realm id will never change,
      so it's not a concern for invalidating cache rows.

All the other attributes are either likely to change (e.g.
sender avatar_version) and/or impact the size of cache
entries more severely than the two small id fields above.

This change should improve our overall system performance
by reducing the amount of memory used by every N message
rows we cache, and typically N will be in the thousands or
so on a large realm.

The other major implication of this change is that when
a user changes their avatar, and then later messages that
the user sent are fetched, all of the fields that go into
computing the avatar url will be pulled from the database,
not from cache.
2017-10-16 23:37:10 -07:00
Steve Howell
33338f78e9 tests: Make test_bulk_message_fetching() more thorough.
We include the post-processing step when counting queries now.
2017-10-16 23:37:10 -07:00
Steve Howell
d909355dc2 refactor: Move methods from models.py -> lib/messages.py.
Message.get_raw_db_rows is moved to MessageDict, since its
implementation details are highly coupled to other methods
in MessageDict.

And then sew_messages_and_reactions comes along for the
ride.

We eventually want to move Reaction.get_raw_db_rows to there
as well.
2017-10-16 23:37:10 -07:00
Steve Howell
4919eb4abd Extract MessageDict.set_sender_avatar().
We now populate the avatar url as part of the post
processing step of building message dictionaries,
so that the avatar url is no longer in cache.

This change makes the cache slimmer, because instead
of caching the avatar url (which often includes a long
hash), we just cache the smaller fields that are used
to compute the url.

Note that this commit still has the problem that we're
essentially computing the avatar url from cached fields
that can be invalid.  We will address that a few commits
later.

An immediate benefit of this change is that how we compute
avatar urls (or whether we compute them all) is now decoupled
from caching concerns.  We will address this later as
well.  (Some clients will be capable of computing their
own gravatar urls, for example.)
2017-10-16 23:37:10 -07:00
Steve Howell
3c6cc3d454 Defer deleting intermediate values in message dictionaries.
We're about to have multiple post-processing stages for building
message dictionaries.  Rather than having individual "hydration"
methods remove intermediate values, we just wait until the end.

This decouples the hyrdration steps.  The potentional problem
here is that we may have a field like sender_is_mirror_dummy
that isn't part of the final payload, but we need it for
calculating display recipients and avatars.  We don't want to
delete it too early from the objects.
2017-10-16 23:37:10 -07:00
derAnfaenger
87468f46ae queue processors: Test flow through UserActivityIntervalWorker.consume(). 2017-10-16 23:20:13 -07:00
derAnfaenger
ad0407578d queue processors: Test flow through UserPresenceWorker.consume(). 2017-10-16 23:20:13 -07:00
derAnfaenger
af699500b7 tests: Add option to call queue processor consumer.
This makes tests of queue processors more realistic,
by adding a parameter to `queue_json_publish` that
calls a queue's consumer function if accessed in a test.

Fixes part of #6542.
2017-10-16 23:20:13 -07:00
Eeshan Garg
fb17481a7f webhooks/gci: Support claim, comment, submit, approve event types. 2017-10-16 22:01:19 -07:00
Tim Abbott
b5c107ed27 push_notifications: Remove unnecessary check for no devices.
This should have been checked by the caller anyway.
2017-10-13 17:30:20 -07:00
Tim Abbott
27a450b58d push_notifications: Improve error message for GCM sending issues.
This addresses one of the sources of confusion in #6993.
2017-10-13 17:30:11 -07:00
Umair Khan
a48a86237d ldap: Change logging level to warning.
Fixes #6960.
2017-10-13 17:13:18 -07:00
Harshit Bansal
7d5bcf5534 notifications: Use lxml instead of hacky regexes to scrub inline images. 2017-10-13 16:13:58 +00:00
Harshit Bansal
8d42f42ef2 notifications: Correctly convert relative narrow links to absolute URLs. 2017-10-13 15:44:47 +00:00
Greg Price
b095463f9b logging: Clear out the Django default config on logger django.
By default, Django sets up two handlers on this logger, one of them
its AdminEmailHandler.  We have our own handler for sending email on
error, and we want to stick to that -- we like the format somewhat
better, and crucially we've given it some rate-limiting through
ZulipLimiter.

Since we cleaned out our logging config in e0a5e6fad, though, we've
been sending error emails through both paths.  The config we'd had
before that for `django` was redundant with the config on the root --
but having *a* config there was essential for causing
`logging.config.dictConfig`, when Django passes it our LOGGING dict,
to clear out that logger's previous config.  So, give it an empty
config.

Django by default configures two loggers: `django` and
`django.server`.  We have our own settings for `django.server`
anyway, so this is the only one we need to add.

The stdlib `logging` and `logging.config` docs aren't 100% clear, and
while the source of `logging` is admirably straightforward the source
of `logging.config` is a little twisty, so it's not easy to become
totally confident that this has the right effect just by reading.
Fortunately we can put some of that source-diving to work in writing
a test for it.
2017-10-12 22:45:14 -07:00
Tim Abbott
1d314c3bf4 bugdown: Remove ERROR_BOT markdown rendering notices.
Nobody has used this feature in years, and it causes certain types of
markdown issues in development to completely DoS the development
environment by making it possible for the "Bugdown timeout" exception
handler to timeout in bugdown.

Since we already send an email to the server administrators, there's
no need to replace this feature with anything.
2017-10-12 17:45:33 -07:00
Tim Abbott
a466bf8a47 report: Fix handling of reports with no more_info. 2017-10-12 17:38:28 -07:00
Steve Howell
e0bc1b114e Make sure mentions refer only to active users.
An active user can share the same full name as a deactivated
user.  We now only allow mention syntax to find users who are
activated.

Fixed #6978
2017-10-12 17:11:36 -07:00
Steve Howell
a9d25f8719 refactor: Simplify avatar_url.
This function is now a thin wrapper around get_avatar_field.
2017-10-12 14:00:41 -07:00
Steve Howell
b0e844c676 refactor: Use get_avatar_field in message.py.
This is part of deprecating avatar_url_from_dict and
eventually supporting the client_gravatar field in
message-related requests from clients.
2017-10-12 14:00:41 -07:00
Steve Howell
a6c3621f55 Support client_gravatar in /json/users endpoint. 2017-10-12 14:00:41 -07:00
Steve Howell
1fc6a5febc Add get_avatar_field() function.
This function is designed to replace avatar_url() and
avatar_url_from_dict() over time.

There are a few things new about it:

    * We make the parameters more explicit, rather than
      passing in an opaque dictionary or requiring a
      UserProfile object.  (A lot of our callers want
      to use `values()` for efficiency sake, since we
      are often doing bulk user operations.)

    * We start to support the client_gravatar option.
2017-10-12 14:00:41 -07:00