Commit Graph

403 Commits

Author SHA1 Message Date
rht
c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Tim Abbott
1cd017288d views: Fix imports of REQ/has_request_variables from the wrong place.
These were never in zerver/decorator.py, and so it makes sense to
import them zerver/lib/request.py, mostly for ease of finding things.
2017-10-27 15:07:31 -07:00
rht
691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht
2f3ae84e5a py3: Remove all __future__ import division. 2017-10-17 23:09:12 -07:00
rht
b2ad8fd747 py3: Remove all from __future__ import unicode_literals.
This was mostly used in migrations, so it's a pretty safe change.
2017-10-17 23:07:42 -07:00
rht
a603a4f9f5 Remove from __future__ import absolute_import.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
  compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
  script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py

For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
2017-10-17 22:59:42 -07:00
Rishi Gupta
e31758c257 analytics: Do not run update_analytics_counts if there are no realms.
Having no realms was not possible before, but will be once system bots are
no longer on a special system realm.
2017-10-05 11:22:06 -07:00
Rishi Gupta
c7bdabbda8 analytics: Disallow non-UTC fill times in process_count_stat.
No change in behavior, but we aren't supporting non-UTC times in analytics
as a whole any more, so might as well change this check as well.
2017-10-05 11:22:06 -07:00
Rishi Gupta
0596c4a810 analytics: Enforce various datetime arguments are in UTC.
Sort of a hacky hammer, but
* The original design of the analytics system mistakenly attempted to play
  nicely with non-UTC datetimes.
* Timezone errors are really hard to find and debug, and don't jump out that
  easily when reading code.

I don't know of any outstanding errors, but putting a few "assert this
timezone is in UTC" around will hopefully reduce the chance that there are
any current or future timezone errors.

Note that none of these functions are called outside of the analytics code
(and tests). This commit also doesn't change any current behavior, assuming
a database where all datetimes have been being stored in UTC.
2017-10-05 11:22:06 -07:00
Rishi Gupta
0c2b4d22a7 analytics: Convert datetimes coming from the API into UTC.
Previously, entering a non-UTC end time for a daily stat would give you
incorrect results. This is because:
* All daily stats are collected at and have end_times in the database in
  midnight UTC.
* For daily stats, time_range returns a list of datetimes at midnight in the
  timezone of its end argument. These datetimes are the only ones we look
  for when looking for rows corresponding to the stat in the database.
* Previously, we passed on the end argument from the API to time_range,
  without modification.
2017-10-05 11:22:06 -07:00
Rishi Gupta
0f31cddf49 analytics: Add management command to clear single stat. 2017-10-05 11:22:06 -07:00
Tim Abbott
575bd0e255 analytics: Update name for old Android app. 2017-10-03 11:59:41 -07:00
rht
dc5136ed96 analytics: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
rht
4494112862 analytics: Remove absolute_import. 2017-09-27 20:20:07 -07:00
rht
74f8a527e4 analytics: Remove print_function. 2017-09-27 18:05:45 -07:00
Aditya Bansal
d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Umair Khan
6d4ba50ceb result.json: Upgrade test_views. 2017-08-17 09:03:35 -07:00
Greg Price
61666a9262 zulip_ops: Delete the long-disused stats1.zulip.net config and its dependencies.
This consists of the `zulip_ops::stats` Puppet class, which has apparently
not been used since 2014, and a number of files that I believe were
only used for that.  Also a couple of tiny loose ends in other files.
2017-08-15 17:30:31 -07:00
Tim Abbott
e3e2d9093c analytics: Remove references to UserProfile.objects.get. 2017-08-15 12:48:33 -07:00
neiljp (Neil Pilgrim)
b782db48e1 mypy: Remove superfluous older 'type: ignore' annotations. 2017-08-08 11:27:51 -07:00
Greg Price
c127630dcf Delete some obsolete usage-stats tools.
These are no longer useful, with our spiffy new analytics framework,
and we haven't in fact been using them for some time, while the
`active-user-stats` cron job does cause regular mail from cron.
Just delete them.
2017-07-31 17:06:15 -07:00
Vishnu Ks
a99c60ce07 analytics: Add translation tags to stats.html. 2017-07-16 16:16:43 -07:00
Vishnu Ks
b0e4cfd480 analytics: Replace get_user_profile_by email in client_activity. 2017-07-14 13:35:43 -07:00
Vishnu Ks
e5c5960faa analytics: Remove unused get_user_profile_by_email import. 2017-07-14 13:35:43 -07:00
Vishnu Ks
0a67e00702 analytics: Use example_user in test_views.py. 2017-07-14 13:35:43 -07:00
Tim Abbott
0b60f11cf8 analytics: Display Zulip Electron app as the main desktop app.
This should make the /stats data significantly clearer.
2017-07-07 18:31:47 -07:00
Tim Abbott
31caab4229 analytics: Rename 'new iOS app' to 'Mobile app'. 2017-07-07 18:31:13 -07:00
Umair Khan
c74f125b7c analytics: Add on_delete in foreign keys.
on_delete will be a required arg for ForeignKey in Django 2.0. Set it
to models.CASCADE on models and in existing migrations if you want to
maintain the current default behavior.
See https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.ForeignKey.on_delete
2017-06-13 15:13:49 -07:00
Aditya Bansal
42b0680ab2 pep8: Add compliance with rule E261 populate_analytics_db.py. 2017-05-31 17:07:15 -07:00
Aditya Bansal
30dfb54c65 pep8: Add compliance with rule E261 to analytics/views.py. 2017-05-31 17:07:15 -07:00
Aditya Bansal
061cb9ae44 pep8: Add compliance with rule E261 to analytics/tests/test_views.py. 2017-05-31 17:07:15 -07:00
umkay
d9b23b39d3 mypy: Fix strict-optional in analytics. 2017-05-26 15:39:39 -07:00
Christian Hudon
c80e6edb4e mypy: Declare models with null=True Optional. 2017-05-23 14:36:40 -07:00
Rishi Gupta
ee16abaf3b analytics: Fix sort function in views.sort_client_labels.
sort_client_labels sorts first by total, and then to ensure deterministic
outcomes, sorts (reverse) alphabetically by label.

Fixes regression introduced in 0c0e539.
2017-05-09 11:32:35 -07:00
Aditya Bansal
3f0b22ce31 pep8: Add compliance with rule E261 to test_counts.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal
2ca1f60ac5 pep8: Add compliance with rule E261 to analytics/models.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal
13d9b98c39 pep8: Add compliance with rule E261 to analyze_mit.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal
9e11185fe2 pep8: Add compliance with rule E261 to active_user_stats.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal
27b87943af pep8: Add compliance with rule E261 to counts.py. 2017-05-07 23:21:50 -07:00
Rishi Gupta
92978d6fb2 analytics: Fix --utc argument in update_analytics_counts.py. 2017-04-28 16:15:07 -07:00
Rishi Gupta
73ae2abd4e analytics: Add --verbose option to update_analytics_counts.py. 2017-04-28 16:15:07 -07:00
Rishi Gupta
61bf445da4 analytics: Restrict fill_to_time to hour boundaries in process_count_stat. 2017-04-28 16:15:07 -07:00
Rishi Gupta
dfbeab73b5 analytics: Change update_analytics_counts to only use hour boundaries.
Fixes a recent regression where analytics were not being run on hour
boundaries.

Includes a migration that dumps all the analytics data.
2017-04-28 16:15:07 -07:00
Rishi Gupta
f595f4f7f2 analytics: Change Number of Users chart to use realm_active_humans::day.
Previously we showed the total number of users with an active account. This
changes it to show only the number of users that have logged in in the past
two weeks.
2017-04-25 18:35:13 -07:00
Maxim Averin
73a1dd63d5 analytics: Refactor legacy 'zulip_internal' decorator.
Rename 'zulip_internal' decorator to 'require_server_admin', add
documentation for 'server_admin', explaining how to give permission
for ./activity page.

Fixes: #1463.
2017-04-22 11:42:02 -07:00
Rishi Gupta
5e49da9285 analytics: Only update daily stats on day boundaries.
Previously we would update FillState for daily stats on hourly boundaries as
well. This would create two extra queries on the FillState table every hour
(for each CountStat), which adds roughly 50ms of extra processing for each
CountStat each day, as well as two extra lines each hour in the analytics
log. This can be a minor annoyance when backfilling stats.
2017-04-18 11:02:51 -07:00
Rishi Gupta
c5f1398052 analytics: Add section comments in counts.count_stats_.
Also reorders the stats a bit.
2017-04-18 11:02:51 -07:00
Rishi Gupta
b335ad2794 models: Add MIN_INTERVAL_LENGTH to UserActivityInterval.
Was previously a floating magic number appearing in both
zerver/lib/actions.py and analytics/lib/counts.py.
2017-04-18 11:02:51 -07:00
hackerkid
5c8f011d66 Remove unused timezone import. 2017-04-16 12:28:56 -07:00
hackerkid
b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00