Commit Graph

4337 Commits

Author SHA1 Message Date
Steve Howell
4ae117ed14 Use bulk_remove_subscriptions() in do_deactivate_stream().
We are deprecating do_remove_subscription().
2016-10-20 22:05:19 -07:00
Steve Howell
44bbece5a6 Remove do_remove_subscription() calls in tests.py. 2016-10-20 22:05:19 -07:00
Steve Howell
b2d06fad42 Extract get_peer_user_ids_for_stream_change(). 2016-10-20 22:05:19 -07:00
Steve Howell
5eee77fe81 Remove do_add_subcription(). 2016-10-20 22:05:19 -07:00
Steve Howell
78b73a0c91 Replace do_add_subscription() in test_helpers. 2016-10-20 22:05:19 -07:00
Umair Khan
45166cb39c is_template_database_current: Check populate_db and postgres-init-test-db.
Compare the hash of 'zilencer/management/commands/populate_db' with
'var/populate_db_hash' and 'tools/setup/postgres-init-test-db' with
'var/postgres_init_test_db_hash' and if any comparison fails rebuild
the database.

With fixes from tabbott.
2016-10-20 22:04:47 -07:00
K.Kanakhin
7bc0c7f0d5 bugdown: Fix patterns ordering in Markdown extension.
- Put 'modal_link' and 'link' inline patterns after avatar inline pattern
  to correct applying them to parsed text.

Fixes:  #1654
2016-10-20 11:18:41 -07:00
Tim Abbott
50356cc325 event_queue: Eliminate buggy need_timeout heartbeat code.
As best I can tell, this option was completely confused in two ways:
* The name is confusing; it actually controls whether we _clear_ the
  timeout associated with the current handler
* It's not clear why one would not want it to be unconditionally true.

From reading the history, I'm pretty sure I had just misread the code
when I created this.

And this caused a real bug; a later refactoring caused us to basically
never cancel the timeouts, which in turn resulted in 90% of all events
traffic being hearbeats with a much lower frequency (~5s) than the
intended 45s.  Removing this code fixes that nasty bug.
2016-10-19 14:32:49 -07:00
Tim Abbott
a6e779a56c events: Add event types to logging.
This should make it a lot easier to analyze our events traffic and
debug potential issues with it.
2016-10-19 14:32:16 -07:00
Rishi Gupta
83f902953a newsletter signup: Update merge_vars.
merge_vars is the user meta data we store in mailchimp. This commit changes
what we store from realm.domain to realm.id, since realm.domain is being
deprecated, and changes the OPTIN_TIME to a nicer format.
2016-10-18 19:29:12 -07:00
Brock Whittaker
77233c7e3b bugdown: Distinguish between YouTube Videos and Image Previews.
This distinguishes between YouTube Videos and Image Previews by adding
a particular “youtube-video” class to the preview along with changing
the title to the video ID rather than the link. This serves to allow
the lightbox to ID when a lightbox preview should be treated like a
YouTube video rather than an image preview.

This also modifies the tests in bug down to expect a youtube-video class
along with the title to just be the video ID on YouTube rather than the
entire URL link.
2016-10-18 14:45:47 -07:00
Tim Abbott
56d0cc69e9 socket: Remove unused djsession_engine. 2016-10-17 23:19:28 -07:00
Mohsen Ibrahim
bfe5787197 Fix Django.utils.importlib deprecation warnings.
This changes `from django.utils.importlib import import_module` to
`from importlib import import_module`, as `django.utils.importlib` will
be removed in django version 1.9.
2016-10-17 23:19:28 -07:00
Mohsen Ibrahim
e9899e4dc9 bugdown: Fix some Python-Markdown deprecation warnings.
Python-Markdown changed how one is supposed to declare extensions
being used.
2016-10-17 23:13:41 -07:00
Tim Abbott
63a49be09e event_queue: Import tornado.ioloop correctly.
We actually use tornado.ioloop, not the root tornado module, and while
the current code does work, it causes mypy errors with the incremental
cache.
2016-10-17 17:57:46 -07:00
Tim Abbott
4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott
e37b6488b8 Bugdown: Remove checked-in CodeHilite extension.
The changes that required us to fork this extension had been merged
into upstream CodeHilite, so we can remove it and switch to using the
version that comes with python-markdown.
2016-10-16 00:40:08 -07:00
umkay
c028815385 bugdown: Update python-markdown to version 2.6.7.
This updates Bugdown to reflect the changes in the updated
markdown. In particular, we now pass a default config object in the
__init__ for the Bugdown extension, update the make_md_engine function
to take kwargs as opposed to a config list, and have UListProcessor
inherit from ulist as opposed to olist (which no longer works).

We update the (forked from upstream) fenced_code extension's
makeExtension to take args and kwargs, and update
FencedBlockPreprocessor __init__ method with updated Codehilite
arguments.

We update the (forked from upstream) Codehilite extension to
mirror the logic with the latest upstream Codehilite:
	Add parse_hl_lines function
	update makeExtension to take args and kwarfs instead of config
	list
	Add regex for highlight lines
	use linenums instead of linenos
	use get_formatter_by_name instead of HtmlFormatter
	user get_lexer_by_name instead of TextLexer
	add hl_lines and use_pygments arguments to the codehlite
	constructor
2016-10-15 23:38:32 -07:00
Tomasz Kolek
2941c1f517 Move converting commits data to text to separate function.
Create get_commits_content function for converting
commits list to text.
2016-10-14 16:54:13 -07:00
Tomasz Kolek
d2cedd3667 Rename consts in lib/webhooks/git.py to make it more general.
Rename:
   PUSH_COMMITS_LIMIT to COMMITS_LIMIT
   PUSH_COMMIT_ROW_TEMPLATE to COMMIT_ROW_TEMPLATE
   PUSH_COMMITS_MORE_THAN_LIMIT_TEMPLATE to COMMITS_MORE_THAN_LIMIT_TEMPLATE
2016-10-14 16:54:13 -07:00
Tomasz Kolek
ee70cc5141 Add git integration method to build message for pull requests events. 2016-10-14 16:54:13 -07:00
umkay
278574a6ec Fix cursor() arguments for connection classes.
In order to use the latest version of psycopg2 with the latest version
of sqlalchemy we must integrate the changes made in the 2.4.6 release of
psycopg2. See
c86ca7687f
and the release notes for psycopg2 2.4.6.
2016-10-14 13:35:06 -07:00
Rishi Gupta
82b814a1cd analytics: Simplify frequency and measurement interval options.
Change the CountStat object to take an is_gauge variable instead of a
smallest_interval variable. Previously, (smallest_interval, frequency)
could be any of (hour, hour), (hour, day), (hour, gauge), (day, hour),
(day, day), or (day, gauge).
The current change is equivalent to excluding (hour, day) and (day, hour)
from the list above.

This change, along with other recent changes, allows us to simplify how we
handle time intervals. This commit also removes the TimeInterval object.
2016-10-14 10:18:37 -07:00
Tim Abbott
0f1468e64f do_update_message_flags: Use access_message for check. 2016-10-11 17:20:15 -07:00
Tim Abbott
f1a399a4e1 message: Create new access_message library function.
With reactions and other upcoming features, we'll be adding several
places where we need to check whether a particular user can access a
particular message.  It's best to just have a single helper function
for this purpose that we can use everywhere.
2016-10-11 17:17:19 -07:00
Eklavya Sharma
41587edd2e bugdown: Change parameter name.
Change the parameter name of some functions from 'md' to 'content',
since the name 'md' seems to be the reason why this parameter was
wrongly annotated.
2016-10-11 11:31:01 -07:00
Eklavya Sharma
f35d0f1c75 bugdown: Fix type annotations.
In some functions the first parameter was wrongly annotated as
`markdown.Markdown`. Change that to `text_type`.
2016-10-11 11:31:01 -07:00
Eklavya Sharma
ccce3ec0be zerver/lib/digest.py: Fix a type annotation. 2016-10-11 11:27:35 -07:00
Tomasz Kolek
312905007e Add EMPTY_SHA const to zerver/lib/webhooks/git.py. 2016-10-10 17:39:05 -07:00
Tomasz Kolek
b69b5e55ce Add git integration method to build message for remove branch events.
Add get_remove_branch_event_message method to zerver/lib/webhooks/git.py
for normalizing remove branch message in all git integrations.
2016-10-10 17:39:05 -07:00
Tomasz Kolek
5f7c1519d1 Add git integration method to build message for force push events.
Add get_force_push_commits_event_message method to zerver/lib/webhooks/git.py
for normalizing force push message in all git integrations.
2016-10-10 17:39:05 -07:00
Tomasz Kolek
32934a02e9 Add template for subject that contains repo name and branch name. 2016-10-10 17:39:05 -07:00
Tomasz Kolek
8697a1e5a2 Add common git integration method to build message for push events.
This adds get_push_commits_event_message method to
zerver/lib/webhooks/git.py for normalizing push messages in all git
integrations.
2016-10-10 17:32:41 -07:00
sonali0901
5623fa9994 request docs: Fix typo in spelling of "parameter". 2016-10-09 11:13:49 -07:00
Rishi Gupta
ea050d5fd8 test_runner: Fix error message for failed import.
Previously, we suggested running
`python -c import zerver.tests.test_mytest`
when importing test_mytest failed, which doesn't work.

This commit adds the missing quotes, making it
`python -c 'import zerver.tests.test_mytest'`
2016-10-08 11:51:53 -07:00
Rishi Gupta
777fcaa6a0 Add new organization type field to Realm objects.
Adds a new field org_type to Realm.  Defaults for restricted_to_domain
and invite_required are now controlled by org_type at time of realm
creation (see zerver.lib.actions.do_create_realm), rather than at the
database level.  Note that the backend defaults are all
org_type=corporate, since that matches the current assumptions in the
codebase, whereas the frontend default is org_type=community, since if
a user isn't sure they probably want community.

Since we will likely in the future enable/disable various
administrative features based on whether an organization is corporate
or community, we discuss those issues in the realm creation form.
Before we actually implement any such features, we'll want to make
sure users understand what type of organization they are a member of.

Choice of org_type (via radio button) has been added to the realm
creation flow and the realm creation management command, and the
open-realm option removed.

The database defaults have not been changed, which allows our testing code
to work unchanged.

[includes some HTML/CSS work by Brock Whittaker to make it look nice]
2016-10-05 17:01:46 -07:00
Tomasz Kolek
dbeab6aa6f Optimize checks of test database state by moving into Python.
Previously, the generate-fixtures shell script by called into Django
multiple times in order to check whether the database was in a
reasonable state.  Since there's a lot of overhead to starting up
Django, this resulted in `test-backend` and `test-js-with-casper`
being quite slow to run a single small test (2.8s or so) even on my
very fast laptop.

We fix this is by moving the checks into a new Python library, so that
we can avoid paying the Django startup overhead 3 times unnecessarily.
The result saves about 1.2s (~40%) from the time required to run a
single backend test.

Fixes #1221.
2016-10-05 10:40:19 -07:00
umkay
d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Steve Howell
b2f84f0fa4 Move render_markdown() to lib/message.py.
This removes a bugdown circular dependency.
2016-10-04 11:34:53 -07:00
Steve Howell
7fb992dba3 Simplify and "fix" render_old_messages management command.
The command to render old messages now looks for all messages
not matching the bugdown version, and it no longer directly calls
into model code.  We should still be extremely cautious about
using this code.
2016-10-04 11:31:20 -07:00
Steve Howell
6b71f5bd5f Inline most calls to set_rendered_content().
This is part of breaking the circular dependency on
bugdown in models.py.  A subsequent commit will fully
kill off set_rendered_content().
2016-10-04 11:31:20 -07:00
Steve Howell
583a6bbadd Extract zerver/lib/message.py.
This pulls message-related code from models.py into a new
module called message.py, and it starts to break some bugdown
dependencies.  All the methods here are basically related to
serializing Message objects as dictionaries for caches and
events.

    extract_message_dict
    stringify_message_dict
    message_to_dict
    message_to_dict_json
    MessageDict.to_dict_uncached
    MessageDict.to_dict_uncached_helper
    MessageDict.build_dict_from_raw_db_row
    MessageDict.build_message_dict

This fix also removes a circular dependency related
to get_avatar_url.

Also, there was kind of a latent bug in Message.need_to_render_content
where it was depending on other calls to Message to import bugdown
and set it globally in the namespace.  We really need to just
eliminate the function, since it's so small and used by code that
may be doing very sketchy things, but for now I just fix it.  (The
bug would possibly be exposed by moving build_message_dict out to the
library.)
2016-10-04 11:31:20 -07:00
Steve Howell
ac994fdd51 Move three functions from models.py to lib/cache.py.
I move these three functions to lib/cache.py:

    to_dict_cache_key_id
    to_dict_cache_key
    flush_message

This will prepare us for a more significant refactoring that
eventually breaks down some circular dependencies with
Message and bugdown.
2016-10-04 11:31:20 -07:00
Tim Abbott
ac2007dd9b resize_avatar: Add a size option. 2016-10-02 21:29:04 -07:00
Tim Abbott
87b9017845 upload: Extract get_avatar_url and add to UploadBackend. 2016-10-02 21:20:43 -07:00
Tim Abbott
22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
Tim Abbott
a4e5450ace subdomains: Add support for aliases of the root subdomain.
We default to counting "www" as such an alias.
2016-09-27 23:25:07 -07:00
hackerkid
ea39fb2556 Add option for hosting each realm on its own subdomain.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.

This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
  for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
  enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
  (but does not restrict the API; that will be tightened in a future commit).

Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).

[substantially modified by tabbott as part of merging]
2016-09-27 23:24:14 -07:00
Tim Abbott
bbab3cdc30 test_helpers: Add HostRequestMock helper class. 2016-09-27 21:18:29 -07:00
Tim Abbott
8442e9249c Reflow annotation for generic_bulk_cached_fetch.
This is a test of mypy's new support for annotating functions that
take lots of arguments.
2016-09-27 20:36:56 -07:00