Commit Graph

1908 Commits

Author SHA1 Message Date
umkay
d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Tim Abbott
279f805448 run-dev: Automate passing --interface='' for vagrant environment.
Fixes #877.
2016-10-04 10:48:26 -07:00
Umair Khan
b6ad24b11b Fix names of sprite emojis. 2016-09-30 10:42:52 -07:00
Tim Abbott
5ed37beb02 test-backend: Simplify argument rewriting logic. 2016-09-28 11:52:05 -07:00
sonali0901
4869e1b0b2 test-backend: Add convenience arguments to run subsets of the tests.
This optimizes the process of running individual or small groups of
backend tests (./tools/test-backend
zerver.tests.test_bugdown.FencedBlockPreprocessorTest.test_simple_quoting)
to allow the following syntaxes:

    ./tools/test-backend zerver/tests/test_bugdown.py
    ./tools/test-backend zerver.tests.test_bugdown.py
    ./tools/test-backend zerver/tests/test_bugdown
    ./tools/test-backend zerver.tests.test_bugdown
    ./tools/test-backend test_bugdown.py
    ./tools/test-backend test_bugdown
    ./tools/test-backend FencedBlockPreprocessorTest
    ./tools/test-backend FencedBlockPreprocessorTest.test_simple_quoting

Fixes #1670.
2016-09-28 11:52:05 -07:00
Tim Abbott
cd5b38f5d8 provision: Fix node_modules being owned by root.
A bug in the node_cache.py code resulted in the node_modules symlink
in Zulip development environments being incorrectly owned by root.
This causes that bug to be fixed the next time a user provisions.
2016-09-28 00:36:48 -07:00
Tim Abbott
a17a6a21c0 Upgrade Django to 1.8.15 with Zulip patches. 2016-09-27 20:29:08 -07:00
Tim Abbott
d25bbdf574 lint: Fix missing cast causing mypy errors. 2016-09-27 18:38:32 -07:00
Tim Abbott
aaa221233a lint: Ban 2-space indentation in our CSS.
Probably most properly we should check for any number of spaces that
isn't 4, but that's a bit more work to do with our linter framework,
and in practice basically every CSS whitespace error we see is 2-space.
2016-09-27 18:06:44 -07:00
Brock Whittaker
87afe61860 Add overlay lightbox for displaying inline image previews.
This adds an event listener (by way of delegation) to the
.message_inline_image elements that pops up the overlay and hides it
when the overlay exit is clicked.

Fixes #654.
2016-09-27 17:50:51 -07:00
Sahil Dua
058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
umkay
798e6faa9e provision: Use NVM to install node and npm.
NVM takes a specific node version and installs the node package and
a corresponding compatible npm package.

We use it in a somewhat hackish way to install node/npm globally with
a pinned version, since that's how we actually want to consume node in
our development environment.

Other details:
- Travis CI now is configured to use the version of node installed by
provision; the easiest way to do this was to sabotage the existing node
installation.
- jsdom is upgraded to a current version, which both requires recent
node and also is required for the tests to pass with recent node.
This fixes running the node tests on Xenial.

Fixes #1498.

[tweaked by tabbott]
2016-09-23 14:34:44 -07:00
Steve Howell
28bb9c883a css linter: Report empty declarations more clearly.
Raise a CssParserException when declarations are empty.
2016-09-23 09:13:43 -07:00
Umair Khan
4301d11b4a Use npm caching in tools/update-prod-static. 2016-09-16 12:40:48 -07:00
Umair Khan
a82990f63c Use npm caching in tools/provision.py. 2016-09-16 12:40:48 -07:00
Tim Abbott
5fcc8146c1 emoji_dump: Clean up spammy sprite creation output.
Glue apparently prints 1 line per file being processed.
2016-09-14 11:49:55 -07:00
Steve Howell
4c5eb3d06e Fix transaction behavior for update_subscriptions_backend().
This commit extracts compose_views() from update_subscriptions_backend(),
and it implements the correct behavior for forcing transactions to roll
back, which is to raise an exception.

There were really three steps in this commit:

- Extract buggy code to compose_views().
- Add tests on compose_views().
- Fix bugs exposed by the new tests by converting errors to exceptions.
2016-09-12 22:03:39 -07:00
Tim Abbott
5c88fbdc6f mypy: Require new Python functions to have type annotations.
We're now at the point where 100% of functions checked by mypy is
fully annotated; to avoid regressions, we're enforcing the requirement
that it stay this way.  We still have a moderate amount of code that
is neither checked by mypy nor annotated, but it seems reasonable to
annotate that code at the same time as we get a chance to fix the mypy
issues in it.

This is implemented by using the --disallow-untyped-defs option in
mypy by default.
2016-09-12 14:56:08 -07:00
Gordon P. Hemsley
0f4148920a Switch tools/webpack from optparse to argparse. 2016-09-12 12:00:16 -07:00
Gordon P. Hemsley
aace1c5bc9 Switch tools/update-prod-static from optparse to argparse. 2016-09-12 12:00:16 -07:00
Gordon P. Hemsley
9755f3f302 Switch tools/test-tools from optparse to argparse. 2016-09-12 12:00:16 -07:00
Gordon P. Hemsley
9b47f4ab3a Switch tools/minify-js from optparse to argparse. 2016-09-12 12:00:16 -07:00
Gordon P. Hemsley
1e9a4b2bab Switch tools/check-templates from optparse to argparse. 2016-09-12 12:00:16 -07:00
Gordon P. Hemsley
fc6f72174b Switch tools/find-add-class from optparse to argparse. 2016-09-12 12:00:16 -07:00
Tim Abbott
9b867be075 mypy: Exclude api/zulip/__init__.py so we can start enforcing. 2016-09-12 09:03:50 -07:00
Tim Abbott
1eb16d03a2 Finish annotating template/css parsers and tests. 2016-09-12 09:03:50 -07:00
Steve Howell
b2ae2dc7cc Exclude contrib_bots from run-mypy. 2016-09-12 08:21:46 -07:00
Umair Khan
6c21d7a2c8 Annotate tools/setup/emoji_dump/emoji_dump.py. 2016-09-12 08:10:49 -07:00
Steve Howell
98ac51d07d Clean up imports in html_branches.py. 2016-09-11 15:30:10 -07:00
Steve Howell
071c957f14 tools: Simplify html_tag_tree().
Because of some recent changes to the tokenizer, we no longer
need to call is_special_html_tag() to filter out special tags.

I also tried to make the start/end logic for pushing/popping
the stack more obvious.
2016-09-11 13:36:55 -07:00
Gordon P. Hemsley
331617efab Factor out HtmlTreeBranch and related code from template parser.
This code is not directly related to the template parser, so it
can safely live in its own file.

The only significant change to the code is to the signature of
`html_branches` so that it can be called without requiring a file.
Since it's only used in html_grep, that has been updated to reflect
this change.

Fixes: #1774.
2016-09-11 14:57:17 -04:00
Gordon P. Hemsley
4d3350bd7b Improve branch test coverage for html_tag_tree() in template parser. 2016-09-11 09:59:13 -07:00
Gordon P. Hemsley
87b87621fe Reach 100% branch coverage for CSS parser. 2016-09-11 09:59:13 -07:00
Gordon P. Hemsley
911634d1d4 Analyze branch coverage in test-tools. 2016-09-11 09:59:13 -07:00
Tim Abbott
d32e691fe9 Annotate check-mirroring. 2016-09-10 11:57:01 -07:00
Tim Abbott
317e2a63b3 Remove tools/deprecated/review and associated files.
This hasn't been used since before Zulip was open source, and isn't
super reusable, so we can remove it.  It'll always be there in the
history if someone ends up wanting it.

While we're at it, we remove the GitPython dependency (only used for
this tool) and the example MSMTP config for the review tool.
2016-09-10 10:53:02 -07:00
Tim Abbott
24906f7678 emoji_dump: Update SIZE to 136x136 to reflect new emoji set.
I'm not entirely sure that this is correct, but it suffices to make
the spritesheet emoji have the right size.
2016-09-08 15:18:04 -07:00
Tim Abbott
a4536cca1e emoji_dump: Generate a sprite sheet for the Zulip emoji. 2016-09-08 15:16:58 -07:00
Tim Abbott
abf85e7fbd emoji_dump: Add warning for fallback to black/white.
I don't think the black/white fallback code is actually used by
default, and thus when it is used, it's usually a sign that something
is broken.  We should throw an error, but at the very least it makes
sense to print a warning.
2016-09-08 15:05:38 -07:00
Tim Abbott
aca6636729 emoji_dump: Fix buggy path management.
In d583710f7c, I apparently broke the
color emoji handling, which was masked (for test purposes) by the fact
that we catch an expection if color doesn't work and in that case fall
back to black and white emoji.
2016-09-08 15:04:09 -07:00
Tim Abbott
a6fffe5120 emoji_dump: Move size constants to top of file. 2016-09-08 14:30:20 -07:00
Tim Abbott
d583710f7c emoji_dump: Refactor path code to be a bit cleaner. 2016-09-08 14:27:47 -07:00
Tim Abbott
3c34264559 run-dev: Automatically fixup missing handlebars binary. 2016-09-07 19:35:41 -07:00
Tim Abbott
c9f9fb265e Update provision.py and Vagrantfile to support VMWare.
This adds support for using VMWare Fusion as the Vagrant provider,
which has better performance than Virtualbox at the price of being
nonfree (in all senses of the term).

We haven't done solid benchmarking as to how much faster it is than
the Virtualbox provider.
2016-09-07 19:30:48 -07:00
Gordon P. Hemsley
3ffc6b5a35 Clean up how missing end tags are handled in template parsing.
The null token was an artificial placeholder that wasn't making clear what
the problem was. Throwing an exception is bolder.
2016-08-30 19:24:17 -04:00
Gordon P. Hemsley
2e865f03bf Protect against nameless HTML tags in tools/lib/template_parser.py.
Fixed IndexError when there is only zero or more whitespace characters
between < and >. (str.split() will return an empty list in this case, which
means there is no index 0.)
2016-08-30 19:00:13 -04:00
Gordon P. Hemsley
c9d1a4247f Improve test coverage for tools.
* Replace generic Exception with TemplateParserException.
* Add tests to cover many of the uncovered lines in
  tools/lib/template_parser.py.
* Add an exclusion line to the naïve pattern for checking for missing
  tuples in format strings, to keep the linter happy.
2016-08-30 18:51:40 -04:00
Gordon P. Hemsley
7070e1ce7c Remove tools/tests/* from test-tools test coverage.
Testing the tests seems redundant.
2016-08-30 18:27:46 -04:00
Kouhei Sutou
683f49aa99 Support full text search for all languages using pgroonga.
This adds support for using PGroonga to back the Zulip full-text
search feature.  Because built-in PostgreSQL full text search doesn't
support languages that don't put space between terms such as Japanese,
Chinese and so on. PGroonga supports all languages including Japanese
and Chinese.

Developers will need to re-provision when rebasing past this patch for
the tests to pass, since provision is what installs the PGroonga
package and extension.

PGroonga is enabled by default in development but not in production;
the hope is that after the PGroonga support is tested further, we can
enable it by default.

Fixes #615.

[docs and tests tweaked by tabbott]
2016-08-26 21:04:03 -07:00
Tim Abbott
b200c6cdc0 Move assets/ to live under static/.
This decreases the clutter in the root directory of the Zulip
repository.
2016-08-26 10:23:38 -07:00