Add changelog for Zulip 1.3.13 release.

If a user's session cookie expired, the next REST API request their
browser did would go into the json_unauthorized code path.  This
returned a response with a WWW-Authenticate tag for HTTP Basic Auth
(since that's what the REST API uses), even for /json requests which
should only be authenticated using session auth.

We fix this by explicitly passing the desired WWW-Authenticate state.

Fixes: #800.

.coveralls.yml

@@ -0,0 +1,2 @@
+service_name: travis-pro
+repo_token: hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG

.gitattributes

@@ -14,7 +14,8 @@
 /zproject/local_settings.py export-ignore
 /zproject/test_settings.py export-ignore
 /zerver/fixtures export-ignore
-/zerver/tests.py export-ignore
 /zerver/tests export-ignore
+/frontend_tests export-ignore
 /node_modules export-ignore
 /humbug export-ignore
+/locale export-ignore

.gitignore

@@ -2,8 +2,18 @@
 *~
 /all_messages_log.*
 /event_log/*
-/server.log
+/digest.log*
+/errors.log*
+/manage.log*
+/server.log*
+/workers.log*
+/email-deliverer.log
+/email-mirror.log
+/sync_ldap_user_data.log
 /update-prod-static.log
+frontend_tests/casper_tests/server.log
+frontend_tests/casper_lib/test_credentials.js
+memcached_prefix
 /prod-static
 /errors/*
 *.sw[po]
@@ -13,24 +23,26 @@ stats/
 zerver/fixtures/available-migrations
 zerver/fixtures/migration-status
 zerver/fixtures/test_data1.json
-zerver/tests/frontend/test_credentials.js
 .kdev4
 zulip.kdev4
-memcached_prefix
+remote_cache_prefix
 coverage/
 /queue_error
-/workers.log
 .test-js-with-node.html
-digest.log
-errors.log
-manage.log
 .kateproject.d/
 .kateproject
 *.kate-swp
 event_queues.json
 .vagrant
 /zproject/dev-secrets.conf
+static/js/bundle.js
 static/third/gemoji/
 static/third/zxcvbn/
-tools/emoji_dump/bitmaps/
-tools/emoji_dump/*.ttx
+tools/setup/emoji_dump/bitmaps/
+tools/setup/emoji_dump/*.ttx
+tools/phantomjs
+node_modules
+npm-debug.log
+uploads/
+test_uploads/
+*.mo

.travis.yml

@@ -1,15 +1,42 @@
+before_install:
+   - nvm install 0.10
 install:
-  - pip install pbs
-  - python provision.py --travis
-cache: apt
+  - pip install coveralls
+  - tools/travis/setup-$TEST_SUITE
+  - tools/clean-venv-cache --travis
+cache:
+  - apt: false
+  - directories:
+    - $HOME/phantomjs
+    - $HOME/zulip-venv-cache
+    - node_modules
+    - $HOME/node
+env:
+  global:
+    - COVERALLS_PARALLEL=true
+    - COVERALLS_SERVICE_NAME=travis-pro
+    - COVERALLS_REPO_TOKEN=hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG
+  matrix:
+    - TEST_SUITE=frontend
+    - TEST_SUITE=backend
+    - TEST_SUITE=production
+    - TEST_SUITE=py3k
 language: python
 python:
   - "2.7"
+matrix:
+  include:
+    - python: "3.4"
+      env: TEST_SUITE=mypy
 # command to run tests
 script:
-  - source /srv/zulip-venv/bin/activate && env PATH=$PATH:/srv/zulip-venv/bin ./tools/test-all
+  - ./tools/travis/$TEST_SUITE
 sudo: required
 services:
 - docker
 addons:
   postgresql: "9.3"
+after_success:
+  coveralls
+notifications:
+  webhooks: https://coveralls.io/webhook?repo_token=$COVERALLS_REPO_TOKEN

.tx/config

@@ -0,0 +1,17 @@
+[main]
+host = https://www.transifex.com
+
+[zulip.djangopo]
+source_file = locale/en/LC_MESSAGES/django.po
+source_lang = en
+type = PO
+file_filter = locale/<lang>/LC_MESSAGES/django.po
+lang_map = zh-Hans: zh_CN
+
+[zulip.translationsjson]
+source_file = static/locale/en/translations.json
+source_lang = en
+type = KEYVALUEJSON
+file_filter = static/locale/<lang>/translations.json
+lang_map = zh-Hans: zh-CN
+

Dockerfile

@@ -0,0 +1,15 @@
+FROM ubuntu:trusty
+
+EXPOSE 9991
+
+RUN apt-get update && apt-get install -y \
+  python-pbs \
+  wget
+
+RUN useradd -d /home/zulip -m zulip && echo 'zulip ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
+
+USER zulip
+
+RUN ln -nsf /srv/zulip ~/zulip
+
+WORKDIR /srv/zulip

README.md

@@ -1,5 +1,11 @@
-Zulip
-=====
+**[Zulip overview](#zulip-overview)** |
+**[Installing for dev](#installing-the-zulip-development-environment)** |
+**[Installing for production](#running-zulip-in-production)** |
+**[Ways to contribute](#ways-to-contribute)** |
+**[How to get involved](#how-to-get-involved-with-contributing-to-zulip)** |
+**[License](#license)**
+
+# Zulip overview
 
 Zulip is a powerful, open source group chat application. Written in
 Python and using the Django framework, Zulip supports both private
@@ -10,214 +16,166 @@ previews, group private messages, audible notifications,
 missed-message emails, desktop apps, and much more.
 
 Further information on the Zulip project and its features can be found
-at https://www.zulip.org
+at https://www.zulip.org.
 
-Contributing to Zulip
-=====================
+[![Build Status](https://travis-ci.org/zulip/zulip.svg?branch=master)](https://travis-ci.org/zulip/zulip) [![Coverage Status](https://coveralls.io/repos/github/zulip/zulip/badge.svg?branch=master)](https://coveralls.io/github/zulip/zulip?branch=master)
 
-Zulip welcomes all forms of contributions!
+## Installing the Zulip Development environment
 
-Before a pull request can be merged, you need to to sign the [Dropbox
-Contributor License Agreement](https://opensource.dropbox.com/cla/).
+The Zulip development environment is the recommended option for folks
+interested in trying out Zulip.  This is documented in
+[README.dev.md](https://github.com/zulip/zulip/blob/master/README.dev.md).
 
-Please run the tests (tools/test-all) before submitting your pull
-request and read our [commit message style
-guidelines](http://zulip.readthedocs.org/en/latest/code-style.html#commit-messages).
+## Running Zulip in production
 
-Zulip has a growing collection of developer documentation including
-detailed documentation on coding style available on [Read The
-Docs](https://zulip.readthedocs.org/).
+Zulip in production only supports Ubuntu 14.04 right now, but work is
+ongoing on adding support for additional platforms. The installation
+process is documented at https://zulip.org/server.html and in more
+detail in [README.prod.md](https://github.com/zulip/zulip/blob/master/README.prod.md).
 
-Zulip also has a [development discussion mailing list](https://groups.google.com/forum/#!forum/zulip-devel)
+## Ways to contribute
 
-Feel free to send any questions or suggestions of areas where you'd
-love to see more documentation to the list!
+Zulip welcomes all forms of contributions!  The page documents the
+Zulip development process.
 
-We recommend sending proposals for large features or refactorings to
-the zulip-devel list for discussion and advice before getting too deep
-into implementation.
+* **Pull requests**. Before a pull request can be merged, you need to
+to sign the [Dropbox Contributor License Agreement][cla].  Also,
+please skim our [commit message style guidelines][doc-commit-style].
 
-Please report any security issues you discover to support@zulip.com.
+* **Testing**. The Zulip automated tests all run automatically when
+you submit a pull request, but you can also run them all in your
+development environment following the instructions in the [testing
+docs][doc-test].
 
-Running Zulip in production
-===========================
+* **Developer Documentation**.  Zulip has a growing collection of
+developer documentation on [Read The Docs][doc].  Recommended reading
+for new contributors includes the [directory structure][doc-dirstruct]
+and [new feature tutorial][doc-newfeat]. You can also improve
+[Zulip.org][z-org].
 
-This is documented in https://zulip.org/server.html and README.prod.md.
+* **Mailing lists and bug tracker**. Zulip has a [development
+discussion mailing list][gg-devel] and uses [GitHub issues
+][gh-issues].  There are also lists for the [Android][email-android]
+and [iOS][email-ios] apps.  Feel free to send any questions or
+suggestions of areas where you'd love to see more documentation to the
+relevant list!  Please report any security issues you discover to
+zulip-security@googlegroups.com.
 
-Installing the Zulip Development environment
-============================================
+* **App codebases**. This repository is for the Zulip server and web
+app (including most integrations); the [desktop][], [Android][], and
+[iOS][] apps, are separate repositories, as is [our experimental React
+Native iOS app][ios-exp].
 
-You will need a machine with at least 2GB of RAM available (see
-https://github.com/zulip/zulip/issues/32 for a plan for how to
-dramatically reduce this requirement).
+* **Glue code**. We maintain a [Hubot adapter][hubot-adapter] and several
+integrations ([Phabricator][phab], [Jenkins][], [Puppet][], [Redmine][],
+and [Trello][]), plus [node.js API bindings][node], and a [full-text search
+PostgreSQL extension][tsearch], as separate repos.
 
-Using Vagrant
--------------
+* **Translations**.  Zulip is in the process of being translated into
+10+ languages, and we love contributions to our translations.  See our
+[translating documentation](transifex) if you're interested in
+contributing!
 
-This is the recommended approach, and is tested on OS X 10.10 as well as Ubuntu 14.04.
+[cla]: https://opensource.dropbox.com/cla/
+[doc]: https://zulip.readthedocs.io/
+[doc-commit-style]: http://zulip.readthedocs.io/en/latest/code-style.html#commit-messages
+[doc-dirstruct]: http://zulip.readthedocs.io/en/latest/directory-structure.html
+[doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
+[doc-test]: https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+[gg-devel]: https://groups.google.com/forum/#!forum/zulip-devel
+[gh-issues]: https://github.com/zulip/zulip/issues
+[desktop]: https://github.com/zulip/zulip-desktop
+[android]: https://github.com/zulip/zulip-android
+[ios]: https://github.com/zulip/zulip-ios
+[ios-exp]: https://github.com/zulip/zulip-mobile
+[email-android]: https://groups.google.com/forum/#!forum/zulip-android
+[email-ios]: https://groups.google.com/forum/#!forum/zulip-ios
+[hubot-adapter]: https://github.com/zulip/hubot-zulip
+[jenkins]: https://github.com/zulip/zulip-jenkins-plugin
+[node]: https://github.com/zulip/zulip-node
+[phab]: https://github.com/zulip/phabricator-to-zulip
+[puppet]: https://github.com/matthewbarr/puppet-zulip
+[redmine]: https://github.com/zulip/zulip-redmine-plugin
+[trello]: https://github.com/zulip/trello-to-zulip
+[tsearch]: https://github.com/zulip/tsearch_extras
+[transifex]: https://www.transifex.com/zulip/zulip/
+[z-org]: https://github.com/zulip/zulip.github.io
 
-* The best performing way to run the Zulip development environment is
-  using an LXC container.  If your host is Ubuntu 14.04 (or newer;
-  what matters is having support for LXC containers), you'll want to
-  install and configure the LXC Vagrant provider like this:
-  `sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir && vagrant plugin install vagrant-lxc`
+## How to get involved with contributing to Zulip
 
-* If your host is OS X, download VirtualBox from
-  <http://download.virtualbox.org/virtualbox/4.3.30/VirtualBox-4.3.30-101610-OSX.dmg>
-  and install it.
+First, subscribe to the Zulip [development discussion mailing
+list][gg-devel].
 
-Once that's done, simply change to your zulip directory and run
-`vagrant up` in your terminal.  That will install the development
-server inside a Vagrant guest.
+The Zulip project uses a system of labels in our [issue
+tracker][gh-issues] to make it easy to find a project if you don't
+have your own project idea in mind or want to get some experience with
+working on Zulip before embarking on a larger project you have in
+mind:
 
-Once that finishes, you can run the development server as follows:
+* [Integrations](https://github.com/zulip/zulip/labels/integrations).
+  Integrate Zulip with another piece of software and contribute it
+  back to the community!  Writing an integration can be a great first
+  contribution.  There's detailed documentation on how to write
+  integrations in [the Zulip integration writing
+  guide](https://zulip.readthedocs.io/en/latest/integration-guide.html).
 
-```
-vagrant ssh -- -L9991:localhost:9991
-# Now inside the container
-cd /srv/zulip
-source /srv/zulip-venv/bin/activate
-./tools/run-dev.py --interface=''
-```
+* [Bite Size](https://github.com/zulip/zulip/labels/bite%20size):
+  Smaller projects that might be a great first contribution.
 
-You can now visit <http://localhost:9991/> in your browser.  To get
-shell access to the virtual machine running the server, use `vagrant ssh`.
+* [Documentation](https://github.com/zulip/zulip/labels/documentation):
+  The Zulip project loves contributions of new documentation.
 
-(A small note on tools/run-dev.py: the `--interface=''` option will make
-the development server listen on all network interfaces.  While this
-is correct for the Vagrant guest sitting behind a NAT, you probably
-don't want to use that option when using run-dev.py in other environments).
+* [Help Wanted](https://github.com/zulip/zulip/labels/help%20wanted):
+  A broader list of projects that nobody is currently working on.
 
-The run-dev.py console output will show any errors your Zulip
-development server encounters.  It runs on top of Django's "manage.py
-runserver" tool, which will automatically restart the Zulip server
-whenever you save changes to Python code.
+* [Platform support](https://github.com/zulip/zulip/labels/Platform%20support):
+  These are open issues about making it possible to install Zulip on a
+  wider range of platforms.
 
-Using provision.py without Vagrant
-----------------------------------
+* [Bugs](https://github.com/zulip/zulip/labels/bug): Open bugs.
 
-If you'd like to install a Zulip development environment on a server
-that's already running Ubuntu 14.04 Trusty, you can do that by just
-running:
+* [Feature requests](https://github.com/zulip/zulip/labels/enhancement):
+  Browsing this list can be a great way to find feature ideas to
+  implement that other Zulip users are excited about.
 
-```
-sudo apt-get update
-sudo apt-get install -y python-pbs
-python /srv/zulip/provision.py
+* [2016 roadmap milestone](http://zulip.readthedocs.io/en/latest/roadmap.html): The
+  projects that are [priorities for the Zulip project](https://zulip.readthedocs.io/en/latest/roadmap.html).  These are great projects if you're looking to make an impact.
 
-cd /srv/zulip
-source /srv/zulip-venv/bin/activate
-./tools/run-dev.py
-```
+If you're excited about helping with an open issue, just post on the
+conversation thread that you're working on it.  You're encouraged to
+ask questions on how to best implement or debug your changes -- the
+Zulip maintainers are excited to answer questions to help you stay
+unblocked and working efficiently.
 
-By hand
--------
-If you really want to install everything by hand, the below
-instructions should work.
+We also welcome suggestions of features that you feel would be
+valuable or changes that you feel would make Zulip a better open
+source project, and are happy to support you in adding new features or
+other user experience improvements to Zulip.
 
-Install the following non-Python dependencies:
- * libffi-dev — needed for some Python extensions
- * postgresql 9.1 or later — our database (also install development headers)
- * memcached (and headers)
- * rabbitmq-server
- * libldap2-dev
- * python-dev
- * redis-server — rate limiting
- * tsearch-extras — better text search
+If you have a new feature you'd like to add, we recommend you start by
+opening a GitHub issue about the feature idea explaining the problem
+that you're hoping to solve and that you're excited to work on it.  A
+Zulip maintainer will usually reply within a day with feedback on the
+idea, notes on any important issues or concerns, and and often tips on
+how to implement or test it.  Please feel free to ping the thread if
+you don't hear a response from the maintainers -- we try to be very
+responsive so this usually means we missed your message.
 
-On Debian or Ubuntu systems:
+For significant changes to the visual design, user experience, data
+model, or architecture, we highly recommend posting a mockup,
+screenshot, or description of what you have in mind to zulip-devel@ to
+get broad feedback before you spend too much time on implementation
+details.
 
-```
-sudo apt-get install libffi-dev memcached rabbitmq-server libldap2-dev python-dev redis-server postgresql-server-dev-all libmemcached-dev
+Finally, before implementing a larger feature, we highly recommend
+looking at the new feature tutorial and coding style guidelines on
+ReadTheDocs.
 
-# If on 12.04 or wheezy:
-sudo apt-get install postgresql-9.1
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
+Feedback on how to make this development process more efficient, fun,
+and friendly to new contributors is very welcome!  Just send an email
+to the Zulip Developers list with your thoughts.
 
-# If on 14.04:
-sudo apt-get install postgresql-9.3
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-
-# If on 15.04 or jessie:
-sudo apt-get install postgresql-9.4
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb
-sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
-
-# Then, all versions:
-pip install -r requirements.txt
-tools/download-zxcvbn
-./tools/emoji_dump/build_emoji
-generate_secrets.py -d
-./scripts/setup/configure-rabbitmq
-./tools/postgres-init-db
-./tools/do-destroy-rebuild-database
-./tools/postgres-init-test-db
-./tools/do_destroy_rebuild_test_database
-```
-
-To start the development server:
-
-```
-./tools/run-dev.py
-```
-
-… and visit http://localhost:9991/.
-
-
-Running the test suite
-======================
-
-Run all tests:
-
-```
-./tools/test-all
-```
-
-This runs the linter plus all of our test suites; they can all be run
-separately (just read `tools/test-all` to see them).  You can also run
-individual tests, e.g.:
-
-```
-./tools/test-backend zerver.test_bugdown.BugdownTest.test_inline_youtube
-./tools/test-js-with-casper 10-navigation.js
-```
-
-The above instructions include the first-time setup of test databases,
-but you may need to rebuild the test database occasionally if you're
-working on new database migrations.  To do this, run:
-
-```
-./tools/postgres-init-test-db
-./tools/do-destroy-rebuild-test-database
-```
-
-Possible testing issues
-=======================
-
-- The Casper tests are flaky on the Virtualbox environment (probably
-  due to some performance-sensitive races; they work reliably in
-  Travis CI).  Until this issue is debugged, you may need to rerun
-  them to get them to pass.
-
-- When running the test suite, if you get an error like this:
-
-  ```
-      sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not   exist
-      LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
-                                                                   ^
-  ```
-
-  … then you need to install tsearch-extras, described
-  above. Afterwards, re-run the `init*-db` and the
-  `do-destroy-rebuild*-database` scripts.
-
-- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000.
-  This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner
-
-License
-=======
+## License
 
 Copyright 2011-2015 Dropbox, Inc.
 

THIRDPARTY

@@ -2,7 +2,7 @@ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: Zulip
 Upstream-Contact: Zulip Development Discussion <zulip-devel@googlegroups.com>
 Source: https://zulip.org/
-Comment: 
+Comment:
  Unless otherwise noted, the Zulip software is distributed under the Apache
  License, Version 2.0. The software includes some works released by third
  parties under other free and open source licenses. Those works are
@@ -38,10 +38,6 @@ Files: confirmation/*
 Copyright: 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 License: BSD-3-Clause
 
-Files: node_modules/handlebars/*
-Copyright: 2011 Yehuda Katz
-License: Expat
-
 Files: puppet/apt/*
 Copyright: 2011, Evolving Web Inc.
 License: Expat
@@ -55,28 +51,28 @@ Comment: https://github.com/DavidS/puppet-common
 
 Files: puppet/stdlib/*
 Copyright: 2011, Krzysztof Wilczynski
- 2011, Puppet Labs Inc 
+ 2011, Puppet Labs Inc
 License: Apache-2.0
 
 File: puppet/zulip_internal/files/mediawiki/Auth_remoteuser.php
-Copyright: 2006 Otheus Shelling 
+Copyright: 2006 Otheus Shelling
  2007 Rusty Burchfield
- 2009 James Kinsman   
- 2010 Daniel Thomas   
+ 2009 James Kinsman
+ 2010 Daniel Thomas
  2010 Ian Ward Comfort
 License: GPL-2.0
 Comment: Not linked.
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_debian_packages
+Files: puppet/zulip/files/nagios_plugins/zulip_base/check_debian_packages
 Copyright: 2005 Francesc Guasch
 License: GPL-2.0
 Comment: Not linked.
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_postgres.pl
+Files: puppet/zulip/files/nagios_plugins/zulip_postgres_appdb/check_postgres.pl
 Copyright: 2007-2015 Greg Sabino Mullane
 License: BSD-2-Clause
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_website_response.sh
+Files: puppet/zulip/files/nagios_plugins/zulip_nagios_server/check_website_response.sh
 Copyright: 2011 Chris Freeman
 License: GPL-2.0
 
@@ -126,15 +122,11 @@ Copyright: 2013 Nijiko Yonskai
 License: Apache-2.0
 Comment: The software has been modified.
 
-Files: static/third/gemoji/images/emoji/unicode/* tools/emoji_dump/*.ttf
+Files: static/third/gemoji/images/emoji/unicode/* tools/setup/emoji_dump/*.ttf
 Copyright: Google, Inc.
 License: Apache-2.0
 Comment: These are actually Noto Emoji, not gemoji.
 
-Files: static/third/handlebars/handlebars.runtime.js
-Copyright: 2011 Yehuda Katz
-License: Expat
-
 Files: static/third/html5-formdata/formdata.js
 Copyright: 2010 François de Metz
 License: Expat
@@ -150,7 +142,7 @@ Copyright: 2013 Jack Moore
 License: Expat
 
 Files: static/third/jquery-caret/*
-Copyright: 2010 C.F., Wong
+Copyright: 2012, 2013 Andrew C. Dvorak
 License: Expat
 
 Files: static/third/jquery-filedrop/jquery.filedrop.js
@@ -205,14 +197,14 @@ License: SIL-OFL-1.1
 
 Files: static/third/spectrum/*
 Copyright: 2013 Brian Grinstead
-License: Expat 
+License: Expat
 
 Files: static/third/spin/spin.js
 Copyright: 2011-2013 Felix Gnass
 License: Expat
 
 Files: static/third/underscore/underscore.js
-Copyright: 2009-2013 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
+Copyright: 2009-2015 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 License: Expat
 Comment: https://github.com/jashkenas/underscore/blob/master/LICENSE
 
@@ -226,7 +218,7 @@ Copyright: 2010 C. F., Wong
 License: Expat
 
 Files: static/third/zocial/*
-Copyright: Sam Collins 
+Copyright: Sam Collins
 License: Expat
 
 Files: tools/inject-messages/othello
@@ -237,10 +229,6 @@ Files: tools/jslint/jslint.js
 Copyright: 2002 Douglas Crockford
 License: XXX-good-not-evil
 
-Files: tools/python-proxy
-Copyright: 2009 F.bio Domingues
-License: Expat
-
 Files: tools/review
 Copyright: 2010 Ksplice, Inc.
 License: Apache-2.0
@@ -254,11 +242,329 @@ Files: zerver/lib/ccache.py
 Copyright: 2013 David Benjamin and Alan Huang
 License: Expat
 
-Files: zerver/tests/frontend/casperjs/*
+Files: zerver/lib/decorator.py zerver/management/commands/runtornado.py scripts/setup/generate_secrets.py
+Copyright: Django Software Foundation and individual contributors
+License: BSD-3-Clause
+
+Files: frontend_tests/casperjs/*
 Copyright: 2011-2012 Nicolas Perriault
  Joyent, Inc. and other Node contributors
 License: Expat
 
-Files: zerver/tests/frontend/casperjs/modules/vendors/*
+Files: frontend_tests/casperjs/modules/vendors/*
 Copyright: 2011, Jeremy Ashkenas
 License: Expat
+
+
+License: Apache-2.0
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ .
+     http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ .
+ On Debian systems, the full text of the Apache License version 2 can
+ be found in /usr/share/common-licenses/Apache-2.0.
+
+License: BSD-2-clause
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+ 1. Redistributions of source code must retain the above copyright
+    notice(s), this list of conditions and the following disclaimer
+    unmodified other than the allowable addition of one or more
+    copyright notices.
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice(s), this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+ .
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
+ EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+ WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
+ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+License: BSD-3-Clause
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+ .
+ 1. Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+ .
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+ .
+ 3. Neither the name of the copyright holder nor the names of its
+    contributors may be used to endorse or promote products derived from
+    this software without specific prior written permission.
+ .
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+ CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
+ OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+ WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
+ OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+License: CC-0-1.0
+ Creative Commons CC0 1.0 Universal
+ CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+ LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+ ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION
+ ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE
+ USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED HEREUNDER, AND
+ DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM THE USE OF THIS DOCUMENT
+ OR THE INFORMATION OR WORKS PROVIDED HEREUNDER.
+ .
+ Statement of Purpose
+ .
+ The laws of most jurisdictions throughout the world automatically confer
+ exclusive Copyright and Related Rights (defined below) upon the creator
+ and subsequent owner(s) (each and all, an "owner") of an original work
+ of authorship and/or a database (each, a "Work").
+ .
+ Certain owners wish to permanently relinquish those rights to a Work for
+ the purpose of contributing to a commons of creative, cultural and
+ scientific works ("Commons") that the public can reliably and without
+ fear of later claims of infringement build upon, modify, incorporate in
+ other works, reuse and redistribute as freely as possible in any form
+ whatsoever and for any purposes, including without limitation commercial
+ purposes. These owners may contribute to the Commons to promote the
+ ideal of a free culture and the further production of creative, cultural
+ and scientific works, or to gain reputation or greater distribution for
+ their Work in part through the use and efforts of others.
+ .
+ For these and/or other purposes and motivations, and without any
+ expectation of additional consideration or compensation, the person
+ associating CC0 with a Work (the "Affirmer"), to the extent that he or
+ she is an owner of Copyright and Related Rights in the Work, voluntarily
+ elects to apply CC0 to the Work and publicly distribute the Work under
+ its terms, with knowledge of his or her Copyright and Related Rights in
+ the Work and the meaning and intended legal effect of CC0 on those
+ rights.
+ .
+ 1. Copyright and Related Rights. A Work made available under CC0 may be
+ protected by copyright and related or neighboring rights ("Copyright and
+ Related Rights"). Copyright and Related Rights include, but are not
+ limited to, the following:
+ .
+ i. the right to reproduce, adapt, distribute, perform, display,
+ communicate, and translate a Work;
+ .
+ ii. moral rights retained by the original author(s) and/or performer(s);
+ .
+ iii. publicity and privacy rights pertaining to a person's image or
+ likeness depicted in a Work;
+ .
+ iv. rights protecting against unfair competition in regards to a Work,
+ subject to the limitations in paragraph 4(a), below;
+ .
+ v. rights protecting the extraction, dissemination, use and reuse of
+ data in a Work;
+ .
+ vi. database rights (such as those arising under Directive 96/9/EC of
+ the European Parliament and of the Council of 11 March 1996 on the legal
+ protection of databases, and under any national implementation thereof,
+ including any amended or successor version of such directive); and
+ .
+ vii. other similar, equivalent or corresponding rights throughout the
+ world based on applicable law or treaty, and any national
+ implementations thereof.
+ .
+ 2. Waiver. To the greatest extent permitted by, but not in contravention
+ of, applicable law, Affirmer hereby overtly, fully, permanently,
+ irrevocably and unconditionally waives, abandons, and surrenders all of
+ Affirmer's Copyright and Related Rights and associated claims and causes
+ of action, whether now known or unknown (including existing as well as
+ future claims and causes of action), in the Work (i) in all territories
+ worldwide, (ii) for the maximum duration provided by applicable law or
+ treaty (including future time extensions), (iii) in any current or
+ future medium and for any number of copies, and (iv) for any purpose
+ whatsoever, including without limitation commercial, advertising or
+ promotional purposes (the "Waiver"). Affirmer makes the Waiver for the
+ benefit of each member of the public at large and to the detriment of
+ Affirmer's heirs and successors, fully intending that such Waiver shall
+ not be subject to revocation, rescission, cancellation, termination, or
+ any other legal or equitable action to disrupt the quiet enjoyment of
+ the Work by the public as contemplated by Affirmer's express Statement
+ of Purpose.
+ .
+ 3. Public License Fallback. Should any part of the Waiver for any reason
+ be judged legally invalid or ineffective under applicable law, then the
+ Waiver shall be preserved to the maximum extent permitted taking into
+ account Affirmer's express Statement of Purpose. In addition, to the
+ extent the Waiver is so judged Affirmer hereby grants to each affected
+ person a royalty-free, non transferable, non sublicensable, non
+ exclusive, irrevocable and unconditional license to exercise Affirmer's
+ Copyright and Related Rights in the Work (i) in all territories
+ worldwide, (ii) for the maximum duration provided by applicable law or
+ treaty (including future time extensions), (iii) in any current or
+ future medium and for any number of copies, and (iv) for any purpose
+ whatsoever, including without limitation commercial, advertising or
+ promotional purposes (the "License"). The License shall be deemed
+ effective as of the date CC0 was applied by Affirmer to the Work. Should
+ any part of the License for any reason be judged legally invalid or
+ ineffective under applicable law, such partial invalidity or
+ ineffectiveness shall not invalidate the remainder of the License, and
+ in such case Affirmer hereby affirms that he or she will not (i)
+ exercise any of his or her remaining Copyright and Related Rights in the
+ Work or (ii) assert any associated claims and causes of action with
+ respect to the Work, in either case contrary to Affirmer's express
+ Statement of Purpose.
+ .
+ 4. Limitations and Disclaimers.
+ .
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+ surrendered, licensed or otherwise affected by this document.
+ .
+ b. Affirmer offers the Work as-is and makes no representations or
+ warranties of any kind concerning the Work, express, implied, statutory
+ or otherwise, including without limitation warranties of title,
+ merchantability, fitness for a particular purpose, non infringement, or
+ the absence of latent or other defects, accuracy, or the present or
+ absence of errors, whether or not discoverable, all to the greatest
+ extent permissible under applicable law.
+ .
+ c. Affirmer disclaims responsibility for clearing rights of other
+ persons that may apply to the Work or any use thereof, including without
+ limitation any person's Copyright and Related Rights in the Work.
+ Further, Affirmer disclaims responsibility for obtaining any necessary
+ consents, permissions or other rights required for any use of the Work.
+ .
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+ party to this document and has no duty or obligation with respect to
+ this CC0 or use of the Work.
+
+License: Expat
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+ .
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE.
+
+License: GPL-2.0
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; version 2, dated June, 1991.
+ .
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ GNU General Public License for more details.
+ .
+ On Debian systems, the complete text of the GNU General Public License
+ can be found in /usr/share/common-licenses/GPL-2 file.
+
+License: SIL-OFL-1.1
+ ---------------------------------------------------------------------------
+ SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+ ---------------------------------------------------------------------------
+ .
+ PREAMBLE
+ .
+ The goals of the Open Font License (OFL) are to stimulate worldwide development
+ of collaborative font projects, to support the font creation efforts of academic
+ and linguistic communities, and to provide a free and open framework in which
+ fonts may be shared and improved in partnership with others.
+ .
+ The OFL allows the licensed fonts to be used, studied, modified and redistributed
+ freely as long as they are not sold by themselves. The fonts, including any
+ derivative works, can be bundled, embedded, redistributed and/or sold with any
+ software provided that any reserved names are not used by derivative works. The
+ fonts and derivatives, however, cannot be released under any other type of license.
+ The requirement for fonts to remain under this license does not apply to any
+ document created using the fonts or their derivatives.
+ .
+ DEFINITIONS
+ .
+ "Font Software" refers to the set of files released by the Copyright Holder(s) under
+ this license and clearly marked as such. This may include source files, build
+ scripts and documentation.
+ .
+ "Reserved Font Name" refers to any names specified as such after the copyright
+ statement(s).
+ .
+ "Original Version" refers to the collection of Font Software components as
+ distributed by the Copyright Holder(s).
+ .
+ "Modified Version" refers to any derivative made by adding to, deleting, or
+ substituting -- in part or in whole -- any of the components of the Original Version,
+ by changing formats or by porting the Font Software to a new environment.
+ .
+ "Author" refers to any designer, engineer, programmer, technical writer or other
+ person who contributed to the Font Software.
+ .
+ PERMISSION & CONDITIONS
+ .
+ Permission is hereby granted, free of charge, to any person obtaining a copy of the
+ Font Software, to use, study, copy, merge, embed, modify, redistribute, and sell
+ modified and unmodified copies of the Font Software, subject to the following
+ conditions:
+ .
+ 1) Neither the Font Software nor any of its individual components, in Original or
+ Modified Versions, may be sold by itself.
+ .
+ 2) Original or Modified Versions of the Font Software may be bundled, redistributed
+ and/or sold with any software, provided that each copy contains the above copyright
+ notice and this license. These can be included either as stand-alone text files,
+ human-readable headers or in the appropriate machine-readable metadata fields within
+ text or binary files as long as those fields can be easily viewed by the user.
+ .
+ 3) No Modified Version of the Font Software may use the Reserved Font Name(s) unless
+ explicit written permission is granted by the corresponding Copyright Holder. This
+ restriction only applies to the primary font name as presented to the users.
+ .
+ 4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font Software shall
+ not be used to promote, endorse or advertise any Modified Version, except to
+ acknowledge the contribution(s) of the Copyright Holder(s) and the Author(s) or with
+ their explicit written permission.
+ .
+ 5) The Font Software, modified or unmodified, in part or in whole, must be distributed
+ entirely under this license, and must not be distributed under any other license. The
+ requirement for fonts to remain under this license does not apply to any document
+ created using the Font Software.
+ .
+ TERMINATION
+ .
+ This license becomes null and void if any of the above conditions are not met.
+ .
+ DISCLAIMER
+ .
+ THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+ INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+ PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER
+ RIGHT. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES,
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR
+ INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.

Vagrantfile

@@ -2,6 +2,11 @@
 
 VAGRANTFILE_API_VERSION = "2"
 
+def command?(name)
+  `which #{name}`
+  $?.success?
+end
+
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
   # For LXC. VirtualBox hosts use a different box, described below.
@@ -13,18 +18,60 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.synced_folder ".", "/vagrant", disabled: true
   config.vm.synced_folder ".", "/srv/zulip"
 
+  proxy_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
+  if File.file?(proxy_config_file)
+    http_proxy = https_proxy = no_proxy = ""
+
+    IO.foreach(proxy_config_file) do |line|
+      line.chomp!
+      key, value = line.split(nil, 2)
+      case key
+      when /^([#;]|$)/; # ignore comments
+      when "HTTP_PROXY"; http_proxy = value
+      when "HTTPS_PROXY"; https_proxy = value
+      when "NO_PROXY"; no_proxy = value
+      end
+    end
+
+    if Vagrant.has_plugin?("vagrant-proxyconf")
+      if http_proxy != ""
+        config.proxy.http = http_proxy
+      end
+      if https_proxy != ""
+        config.proxy.https = https_proxy
+      end
+      if https_proxy != ""
+        config.proxy.no_proxy = no_proxy
+      end
+    end
+  end
+
+  # Specify LXC provider before VirtualBox provider so it's preferred.
+  config.vm.provider "lxc" do |lxc|
+    if command? "lxc-ls"
+      LXC_VERSION = `lxc-ls --version`.strip unless defined? LXC_VERSION
+      if LXC_VERSION >= "1.1.0"
+        # Allow start without AppArmor, otherwise Box will not Start on Ubuntu 14.10
+        # see https://github.com/fgrehm/vagrant-lxc/issues/333
+        lxc.customize 'aa_allow_incomplete', 1
+      end
+      if LXC_VERSION >= "2.0.0"
+        lxc.backingstore = 'dir'
+      end
+    end
+  end
+
   config.vm.provider "virtualbox" do |vb, override|
     override.vm.box = "ubuntu/trusty64"
-    # 2GiB seemed reasonable here. The VM OOMs with only 1024MiB.
-    vb.memory = 2048
+    # It's possible we can get away with just 1GB; more testing needed
+    vb.memory = 1280
   end
 
 $provision_script = <<SCRIPT
 set -x
 set -e
-sudo apt-get update
-sudo apt-get install -y python-pbs
-python /srv/zulip/provision.py
+ln -nsf /srv/zulip ~/zulip
+/usr/bin/python /srv/zulip/provision.py
 SCRIPT
 
   config.vm.provision "shell",

analytics/management/commands/active_user_stats.py

@@ -1,6 +1,8 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from django.core.management.base import BaseCommand
+from typing import Any
 
 from zerver.models import UserPresence, UserActivity
 from zerver.lib.utils import statsd, statsd_key
@@ -14,6 +16,7 @@ class Command(BaseCommand):
     Run as a cron job that runs every 10 minutes."""
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         # Get list of all active users in the last 1 week
         cutoff = datetime.now() - timedelta(minutes=30, hours=168)
 
@@ -21,7 +24,7 @@ class Command(BaseCommand):
 
         # Calculate 10min, 2hrs, 12hrs, 1day, 2 business days (TODO business days), 1 week bucket of stats
         hour_buckets = [0.16, 2, 12, 24, 48, 168]
-        user_info = defaultdict(dict)
+        user_info = defaultdict(dict) # type: Dict[str, Dict[float, List[str]]]
 
         for last_presence in users:
             if last_presence.status == UserPresence.IDLE:
@@ -30,28 +33,28 @@ class Command(BaseCommand):
                 known_active = last_presence.timestamp
 
             for bucket in hour_buckets:
-                if not bucket in user_info[last_presence.user_profile.realm.domain]:
+                if bucket not in user_info[last_presence.user_profile.realm.domain]:
                     user_info[last_presence.user_profile.realm.domain][bucket] = []
                 if datetime.now(known_active.tzinfo) - known_active < timedelta(hours=bucket):
                     user_info[last_presence.user_profile.realm.domain][bucket].append(last_presence.user_profile.email)
 
         for realm, buckets in user_info.items():
-            print("Realm %s" % realm)
+            print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.active.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.active.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))
 
         # Also do stats for how many users have been reading the app.
-        users_reading = UserActivity.objects.select_related().filter(query="/json/update_message_flags")
+        users_reading = UserActivity.objects.select_related().filter(query="/json/messages/flags")
         user_info = defaultdict(dict)
         for activity in users_reading:
             for bucket in hour_buckets:
-                if not bucket in user_info[activity.user_profile.realm.domain]:
+                if bucket not in user_info[activity.user_profile.realm.domain]:
                     user_info[activity.user_profile.realm.domain][bucket] = []
                 if datetime.now(activity.last_visit.tzinfo) - activity.last_visit < timedelta(hours=bucket):
                     user_info[activity.user_profile.realm.domain][bucket].append(activity.user_profile.email)
         for realm, buckets in user_info.items():
-            print("Realm %s" % realm)
+            print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers reading for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.reading.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.reading.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))

analytics/management/commands/active_user_stats_by_day.py

@@ -1,9 +1,11 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 import datetime
 import pytz
 
 from optparse import make_option
+from typing import Any
 from django.core.management.base import BaseCommand
 from zerver.lib.statistics import activity_averages_during_day
 
@@ -15,10 +17,11 @@ class Command(BaseCommand):
                      help="Day to query in format 2013-12-05.  Default is yesterday"),)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options["date"] is None:
             date = datetime.datetime.now() - datetime.timedelta(days=1)
         else:
             date = datetime.datetime.strptime(options["date"], "%Y-%m-%d")
-        print "Activity data for", date
-        print activity_averages_during_day(date)
-        print "Please note that the total registered user count is a total for today"
+        print("Activity data for", date)
+        print(activity_averages_during_day(date))
+        print("Please note that the total registered user count is a total for today")

analytics/management/commands/analyze_mit.py

@@ -1,4 +1,7 @@
 from __future__ import absolute_import
+from __future__ import print_function
+
+from typing import Any
 
 from optparse import make_option
 from django.core.management.base import BaseCommand
@@ -9,6 +12,7 @@ import time
 import logging
 
 def compute_stats(log_level):
+    # type: (int) -> None
     logger = logging.getLogger()
     logger.setLevel(log_level)
 
@@ -26,15 +30,15 @@ def compute_stats(log_level):
                            "bitcoin@mit.edu", "lp@mit.edu", "clocks@mit.edu",
                            "root@mit.edu", "nagios@mit.edu",
                            "www-data|local-realm@mit.edu"])
-    user_counts = {}
+    user_counts = {} # type: Dict[str, Dict[str, int]]
     for m in mit_query.select_related("sending_client", "sender"):
         email = m.sender.email
         user_counts.setdefault(email, {})
         user_counts[email].setdefault(m.sending_client.name, 0)
         user_counts[email][m.sending_client.name] += 1
 
-    total_counts = {}
-    total_user_counts = {}
+    total_counts = {} # type: Dict[str, int]
+    total_user_counts = {} # type: Dict[str, int]
     for email, counts in user_counts.items():
         total_user_counts.setdefault(email, 0)
         for client_name, count in counts.items():
@@ -43,9 +47,9 @@ def compute_stats(log_level):
             total_user_counts[email] += count
 
     logging.debug("%40s | %10s | %s" % ("User", "Messages", "Percentage Zulip"))
-    top_percents = {}
+    top_percents = {} # type: Dict[int, float]
     for size in [10, 25, 50, 100, 200, len(total_user_counts.keys())]:
-        top_percents[size] = 0
+        top_percents[size] = 0.0
     for i, email in enumerate(sorted(total_user_counts.keys(),
                                      key=lambda x: -total_user_counts[x])):
         percent_zulip = round(100 - (user_counts[email].get("zephyr_mirror", 0)) * 100. /
@@ -63,7 +67,7 @@ def compute_stats(log_level):
         logging.info("Top %6s | %s%%" % (size, round(top_percents[size], 1)))
 
     grand_total = sum(total_counts.values())
-    print grand_total
+    print(grand_total)
     logging.info("%15s | %s" % ("Client", "Percentage"))
     for client in total_counts.keys():
         logging.info("%15s | %s%%" % (client, round(100. * total_counts[client] / grand_total, 1)))
@@ -75,6 +79,7 @@ class Command(BaseCommand):
     help = "Compute statistics on MIT Zephyr usage."
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         level = logging.INFO
         if options["verbose"]:
             level = logging.DEBUG

analytics/management/commands/analyze_user_activity.py

@@ -1,4 +1,8 @@
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from typing import Any, Dict
 
 from zerver.lib.statistics import seconds_usage_between
 
@@ -9,6 +13,7 @@ import datetime
 from django.utils.timezone import utc
 
 def analyze_activity(options):
+    # type: (Dict[str, Any]) -> None
     day_start = datetime.datetime.strptime(options["date"], "%Y-%m-%d").replace(tzinfo=utc)
     day_end = day_start + datetime.timedelta(days=options["duration"])
 
@@ -16,7 +21,7 @@ def analyze_activity(options):
     if options["realm"]:
         user_profile_query = user_profile_query.filter(realm__domain=options["realm"])
 
-    print "Per-user online duration:\n"
+    print("Per-user online duration:\n")
     total_duration = datetime.timedelta(0)
     for user_profile in user_profile_query:
         duration = seconds_usage_between(user_profile, day_start, day_end)
@@ -25,11 +30,11 @@ def analyze_activity(options):
             continue
 
         total_duration += duration
-        print "%-*s%s" % (37, user_profile.email, duration, )
+        print("%-*s%s" % (37, user_profile.email, duration,))
 
-    print "\nTotal Duration:                      %s" % (total_duration,)
-    print "\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,)
-    print "Total Duration amortized to a month: %s" % (total_duration.total_seconds() * 30. / 60.,)
+    print("\nTotal Duration:                      %s" % (total_duration,))
+    print("\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,))
+    print("Total Duration amortized to a month: %s" % (total_duration.total_seconds() * 30. / 60.,))
 
 class Command(BaseCommand):
     help = """Report analytics of user activity on a per-user and realm basis.
@@ -54,4 +59,5 @@ is shown for all realms"""
         )
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         analyze_activity(options)

analytics/management/commands/client_activity.py

@@ -1,7 +1,11 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
-from django.db.models import Count
+from django.db.models import Count, QuerySet
 
 from zerver.models import UserActivity, UserProfile, Realm, \
     get_realm, get_user_profile_by_email
@@ -18,10 +22,12 @@ python manage.py client_activity zulip.com
 python manage.py client_activity jesstess@zulip.com"""
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('arg', metavar='<arg>', type=str, nargs='?', default=None,
                             help="realm or user to estimate client activity for")
 
     def compute_activity(self, user_activity_objects):
+        # type: (QuerySet) -> None
         # Report data from the past week.
         #
         # This is a rough report of client activity because we inconsistently
@@ -48,11 +54,12 @@ python manage.py client_activity jesstess@zulip.com"""
         counts.sort()
 
         for count in counts:
-            print "%25s %15d" % (count[1], count[0])
-        print "Total:", total
+            print("%25s %15d" % (count[1], count[0]))
+        print("Total:", total)
 
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['arg'] is None:
             # Report global activity.
             self.compute_activity(UserActivity.objects.all())
@@ -70,5 +77,5 @@ python manage.py client_activity jesstess@zulip.com"""
                     self.compute_activity(UserActivity.objects.filter(
                             user_profile__realm=realm))
                 except Realm.DoesNotExist:
-                    print "Unknown user or domain %s" % (arg,)
+                    print("Unknown user or domain %s" % (arg,))
                     exit(1)

analytics/management/commands/realm_stats.py

@@ -1,12 +1,17 @@
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 import datetime
 import pytz
 
 from django.core.management.base import BaseCommand
 from django.db.models import Count
 from zerver.models import UserProfile, Realm, Stream, Message, Recipient, UserActivity, \
-    Subscription, UserMessage
+    Subscription, UserMessage, get_realm
 
 MOBILE_CLIENT_LIST = ["Android", "ios"]
 HUMAN_CLIENT_LIST = MOBILE_CLIENT_LIST + ["website"]
@@ -17,100 +22,110 @@ class Command(BaseCommand):
     help = "Generate statistics on realm activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def active_users(self, realm):
+        # type: (Realm) -> List[UserProfile]
         # Has been active (on the website, for now) in the last 7 days.
         activity_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=7)
         return [activity.user_profile for activity in \
                     UserActivity.objects.filter(user_profile__realm=realm,
                                                 user_profile__is_active=True,
                                                 last_visit__gt=activity_cutoff,
-                                                query="/json/update_pointer",
+                                                query="/json/users/me/pointer",
                                                 client__name="website")]
 
     def messages_sent_by(self, user, days_ago):
+        # type: (UserProfile, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender=user, pub_date__gt=sent_time_cutoff).count()
 
     def total_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return Message.objects.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def human_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def api_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         return (self.total_messages(realm, days_ago) - self.human_messages(realm, days_ago))
 
     def stream_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff,
                                      recipient__type=Recipient.STREAM).count()
 
     def private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.HUDDLE).count()
 
     def group_private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.PERSONAL).count()
 
     def report_percentage(self, numerator, denominator, text):
+        # type: (float, float, str) -> None
         if not denominator:
             fraction = 0.0
         else:
             fraction = numerator / float(denominator)
-        print "%.2f%% of" % (fraction * 100,), text
+        print("%.2f%% of" % (fraction * 100,), text)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
-                realms = [Realm.objects.get(domain=domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
-                print e
+                realms = [get_realm(domain) for domain in options['realms']]
+            except Realm.DoesNotExist as e:
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
+            print(realm.domain)
 
             user_profiles = UserProfile.objects.filter(realm=realm, is_active=True)
             active_users = self.active_users(realm)
             num_active = len(active_users)
 
-            print "%d active users (%d total)" % (num_active, len(user_profiles))
+            print("%d active users (%d total)" % (num_active, len(user_profiles)))
             streams = Stream.objects.filter(realm=realm).extra(
                 tables=['zerver_subscription', 'zerver_recipient'],
                 where=['zerver_subscription.recipient_id = zerver_recipient.id',
                        'zerver_recipient.type = 2',
                        'zerver_recipient.type_id = zerver_stream.id',
                        'zerver_subscription.active = true']).annotate(count=Count("name"))
-            print "%d streams" % (streams.count(),)
+            print("%d streams" % (streams.count(),))
 
             for days_ago in (1, 7, 30):
-                print "In last %d days, users sent:" % (days_ago,)
+                print("In last %d days, users sent:" % (days_ago,))
                 sender_quantities = [self.messages_sent_by(user, days_ago) for user in user_profiles]
                 for quantity in sorted(sender_quantities, reverse=True):
-                    print quantity,
-                print ""
+                    print(quantity, end=' ')
+                print("")
 
-                print "%d stream messages" % (self.stream_messages(realm, days_ago),)
-                print "%d one-on-one private messages" % (self.private_messages(realm, days_ago),)
-                print "%d messages sent via the API" % (self.api_messages(realm, days_ago),)
-                print "%d group private messages" % (self.group_private_messages(realm, days_ago),)
+                print("%d stream messages" % (self.stream_messages(realm, days_ago),))
+                print("%d one-on-one private messages" % (self.private_messages(realm, days_ago),))
+                print("%d messages sent via the API" % (self.api_messages(realm, days_ago),))
+                print("%d group private messages" % (self.group_private_messages(realm, days_ago),))
 
-            num_notifications_enabled = len(filter(lambda x: x.enable_desktop_notifications == True,
-                                                   active_users))
+            num_notifications_enabled = len([x for x in active_users if x.enable_desktop_notifications == True])
             self.report_percentage(num_notifications_enabled, num_active,
                                    "active users have desktop notifications enabled")
 
-            num_enter_sends = len(filter(lambda x: x.enter_sends, active_users))
+            num_enter_sends = len([x for x in active_users if x.enter_sends])
             self.report_percentage(num_enter_sends, num_active,
                                    "active users have enter-sends")
 
@@ -124,8 +139,8 @@ class Command(BaseCommand):
             starrers = UserMessage.objects.filter(user_profile__in=user_profiles,
                                                   flags=UserMessage.flags.starred).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users have starred %d messages" % (
-                len(starrers), sum([elt["count"] for elt in starrers]))
+            print("%d users have starred %d messages" % (
+                len(starrers), sum([elt["count"] for elt in starrers])))
 
             active_user_subs = Subscription.objects.filter(
                 user_profile__in=user_profiles, active=True)
@@ -133,20 +148,20 @@ class Command(BaseCommand):
             # Streams not in home view
             non_home_view = active_user_subs.filter(in_home_view=False).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users have %d streams not in home view" % (
-                len(non_home_view), sum([elt["count"] for elt in non_home_view]))
+            print("%d users have %d streams not in home view" % (
+                len(non_home_view), sum([elt["count"] for elt in non_home_view])))
 
             # Code block markup
             markup_messages = human_messages.filter(
                 sender__realm=realm, content__contains="~~~").values(
                 "sender").annotate(count=Count("sender"))
-            print "%d users have used code block markup on %s messages" % (
-                len(markup_messages), sum([elt["count"] for elt in markup_messages]))
+            print("%d users have used code block markup on %s messages" % (
+                len(markup_messages), sum([elt["count"] for elt in markup_messages])))
 
             # Notifications for stream messages
             notifications = active_user_subs.filter(notifications=True).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users receive desktop notifications for %d streams" % (
-                len(notifications), sum([elt["count"] for elt in notifications]))
+            print("%d users receive desktop notifications for %d streams" % (
+                len(notifications), sum([elt["count"] for elt in notifications])))
 
-            print ""
+            print("")

analytics/management/commands/stream_stats.py

@@ -1,40 +1,46 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
 from django.db.models import Q
-from zerver.models import Realm, Stream, Message, Subscription, Recipient
+from zerver.models import Realm, Stream, Message, Subscription, Recipient, get_realm
 
 class Command(BaseCommand):
     help = "Generate statistics on the streams for a realm."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['realms']:
             try:
-                realms = [Realm.objects.get(domain=domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
-                print e
+                realms = [get_realm(domain) for domain in options['realms']]
+            except Realm.DoesNotExist as e:
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
-            print "------------"
-            print "%25s %15s %10s" % ("stream", "subscribers", "messages")
+            print(realm.domain)
+            print("------------")
+            print("%25s %15s %10s" % ("stream", "subscribers", "messages"))
             streams = Stream.objects.filter(realm=realm).exclude(Q(name__istartswith="tutorial-"))
             invite_only_count = 0
             for stream in streams:
                 if stream.invite_only:
                     invite_only_count += 1
                     continue
-                print "%25s" % (stream.name,),
+                print("%25s" % (stream.name,), end=' ')
                 recipient = Recipient.objects.filter(type=Recipient.STREAM, type_id=stream.id)
-                print "%10d" % (len(Subscription.objects.filter(recipient=recipient, active=True)),),
+                print("%10d" % (len(Subscription.objects.filter(recipient=recipient, active=True)),), end=' ')
                 num_messages = len(Message.objects.filter(recipient=recipient))
-                print "%12d" % (num_messages,)
-            print "%d invite-only streams" % (invite_only_count,)
-            print ""
+                print("%12d" % (num_messages,))
+            print("%d invite-only streams" % (invite_only_count,))
+            print("")

analytics/management/commands/user_stats.py

@@ -1,41 +1,48 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
+from argparse import ArgumentParser
 import datetime
 import pytz
+from typing import Any
 
 from django.core.management.base import BaseCommand
-from zerver.models import UserProfile, Realm, Stream, Message
+from zerver.models import UserProfile, Realm, Stream, Message, get_realm
+from six.moves import range
 
 class Command(BaseCommand):
     help = "Generate statistics on user activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def messages_sent_by(self, user, week):
+        # type: (UserProfile, int) -> int
         start = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=(week + 1)*7)
         end = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=week*7)
         return Message.objects.filter(sender=user, pub_date__gt=start, pub_date__lte=end).count()
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
-                realms = [Realm.objects.get(domain=domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
-                print e
+                realms = [get_realm(domain) for domain in options['realms']]
+            except Realm.DoesNotExist as e:
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
+            print(realm.domain)
             user_profiles = UserProfile.objects.filter(realm=realm, is_active=True)
-            print "%d users" % (len(user_profiles),)
-            print "%d streams" % (len(Stream.objects.filter(realm=realm)),)
+            print("%d users" % (len(user_profiles),))
+            print("%d streams" % (len(Stream.objects.filter(realm=realm)),))
 
             for user_profile in user_profiles:
-                print "%35s" % (user_profile.email,),
+                print("%35s" % (user_profile.email,), end=' ')
                 for week in range(10):
-                    print "%5d" % (self.messages_sent_by(user_profile, week)),
-                print ""
+                    print("%5d" % (self.messages_sent_by(user_profile, week)), end=' ')
+                print("")

analytics/urls.py

@@ -1,8 +1,9 @@
 from django.conf.urls import patterns, url
 
-urlpatterns = patterns('analytics.views',
-    url(r'^activity$', 'get_activity'),
-    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'get_realm_activity'),
-    url(r'^user_activity/(?P<email>[\S]+)/$', 'get_user_activity'),
-)
+i18n_urlpatterns = [
+    url(r'^activity$', 'analytics.views.get_activity'),
+    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'analytics.views.get_realm_activity'),
+    url(r'^user_activity/(?P<email>[\S]+)/$', 'analytics.views.get_user_activity'),
+]
 
+urlpatterns = patterns('', *i18n_urlpatterns)

analytics/views.py

@@ -1,9 +1,14 @@
+from __future__ import absolute_import
+from __future__ import division
+from six import text_type
+from typing import Any, Dict, List, Tuple, Optional, Sequence, Callable, Union
+
 from django.db import connection
+from django.db.models.query import QuerySet
 from django.template import RequestContext, loader
-from django.utils.html import mark_safe
-from django.shortcuts import render_to_response
 from django.core import urlresolvers
-from django.http import HttpResponseNotFound
+from django.http import HttpResponseNotFound, HttpRequest, HttpResponse
+from jinja2 import Markup as mark_safe
 
 from zerver.decorator import has_request_variables, REQ, zulip_internal
 from zerver.models import get_realm, UserActivity, UserActivityInterval, Realm
@@ -15,14 +20,22 @@ import itertools
 import time
 import re
 import pytz
+from six.moves import filter
+from six.moves import map
+from six.moves import range
+from six.moves import zip
 eastern_tz = pytz.timezone('US/Eastern')
 
+from zproject.jinja2 import render_to_response
+
 def make_table(title, cols, rows, has_row_class=False):
+    # type: (str, List[str], List[Any], bool) -> str
 
     if not has_row_class:
         def fix_row(row):
+            # type: (Any) -> Dict[str, Any]
             return dict(cells=row, row_class=None)
-        rows = map(fix_row, rows)
+        rows = list(map(fix_row, rows))
 
     data = dict(title=title, cols=cols, rows=rows)
 
@@ -34,15 +47,17 @@ def make_table(title, cols, rows, has_row_class=False):
     return content
 
 def dictfetchall(cursor):
+    # type: (connection.cursor) -> List[Dict[str, Any]]
     "Returns all rows from a cursor as a dict"
     desc = cursor.description
     return [
-        dict(zip([col[0] for col in desc], row))
+        dict(list(zip([col[0] for col in desc], row)))
         for row in cursor.fetchall()
     ]
 
 
 def get_realm_day_counts():
+    # type: () -> Dict[str, Dict[str, str]]
     query = '''
         select
             r.domain,
@@ -70,18 +85,19 @@ def get_realm_day_counts():
     rows = dictfetchall(cursor)
     cursor.close()
 
-    counts = defaultdict(dict)
+    counts = defaultdict(dict) # type: Dict[str, Dict[int, int]]
     for row in rows:
         counts[row['domain']][row['age']] = row['cnt']
 
 
     result = {}
     for domain in counts:
-        cnts = [counts[domain].get(age, 0) for age in range(8)]
-        min_cnt = min(cnts)
-        max_cnt = max(cnts)
+        raw_cnts = [counts[domain].get(age, 0) for age in range(8)]
+        min_cnt = min(raw_cnts)
+        max_cnt = max(raw_cnts)
 
         def format_count(cnt):
+            # type: (int) -> str
             if cnt == min_cnt:
                 good_bad = 'bad'
             elif cnt == max_cnt:
@@ -91,12 +107,13 @@ def get_realm_day_counts():
 
             return '<td class="number %s">%s</td>' % (good_bad, cnt)
 
-        cnts = ''.join(map(format_count, cnts))
+        cnts = ''.join(map(format_count, raw_cnts))
         result[domain] = dict(cnts=cnts)
 
     return result
 
 def realm_summary_table(realm_minutes):
+    # type: (Dict[str, float]) -> str
     query = '''
         SELECT
             realm.domain,
@@ -132,7 +149,8 @@ def realm_summary_table(realm_minutes):
                         '/json/send_message',
                         'send_message_backend',
                         '/api/v1/send_message',
-                        '/json/update_pointer'
+                        '/json/update_pointer',
+                        '/json/users/me/pointer'
                     )
                 AND
                     last_visit > now() - interval '1 day'
@@ -161,8 +179,9 @@ def realm_summary_table(realm_minutes):
                         ua.query in (
                             '/json/send_message',
                             'send_message_backend',
-                           '/api/v1/send_message',
-                            '/json/update_pointer'
+                            '/api/v1/send_message',
+                            '/json/update_pointer',
+                            '/json/users/me/pointer'
                         )
                     GROUP by realm.id, up.email
                     HAVING max(last_visit) between
@@ -182,7 +201,8 @@ def realm_summary_table(realm_minutes):
                         '/json/send_message',
                         '/api/v1/send_message',
                         'send_message_backend',
-                        '/json/update_pointer'
+                        '/json/update_pointer',
+                        '/json/users/me/pointer'
                     )
                 AND
                     up.realm_id = realm.id
@@ -206,10 +226,10 @@ def realm_summary_table(realm_minutes):
             row['history'] = ''
 
     # augment data with realm_minutes
-    total_hours = 0
+    total_hours = 0.0
     for row in rows:
         domain = row['domain']
-        minutes = realm_minutes.get(domain, 0)
+        minutes = realm_minutes.get(domain, 0.0)
         hours = minutes / 60.0
         total_hours += hours
         row['hours'] = str(int(hours))
@@ -224,18 +244,21 @@ def realm_summary_table(realm_minutes):
 
     # Count active sites
     def meets_goal(row):
+        # type: (Dict[str, int]) -> bool
         return row['active_user_count'] >= 5
 
-    num_active_sites = len(filter(meets_goal, rows))
+    num_active_sites = len(list(filter(meets_goal, rows)))
 
     # create totals
     total_active_user_count = 0
     total_user_profile_count = 0
     total_bot_count = 0
+    total_at_risk_count = 0
     for row in rows:
         total_active_user_count += int(row['active_user_count'])
         total_user_profile_count += int(row['user_profile_count'])
         total_bot_count += int(row['bot_count'])
+        total_at_risk_count += int(row['at_risk_count'])
 
 
     rows.append(dict(
@@ -243,7 +266,8 @@ def realm_summary_table(realm_minutes):
         active_user_count=total_active_user_count,
         user_profile_count=total_user_profile_count,
         bot_count=total_bot_count,
-        hours=int(total_hours)
+        hours=int(total_hours),
+        at_risk_count=total_at_risk_count,
     ))
 
     content = loader.render_to_string(
@@ -254,6 +278,7 @@ def realm_summary_table(realm_minutes):
 
 
 def user_activity_intervals():
+    # type: () -> Tuple[mark_safe, Dict[str, float]]
     day_end = timestamp_to_datetime(time.time())
     day_start = day_end - timedelta(hours=24)
 
@@ -293,7 +318,7 @@ def user_activity_intervals():
 
             total_duration += duration
             realm_duration += duration
-            output += "  %-*s%s\n" % (37, email, duration, )
+            output += "  %-*s%s\n" % (37, email, duration)
 
         realm_minutes[domain] = realm_duration.total_seconds() / 60
 
@@ -304,6 +329,7 @@ def user_activity_intervals():
     return content, realm_minutes
 
 def sent_messages_report(realm):
+    # type: (str) -> str
     title = 'Recently sent messages for ' + realm
 
     cols = [
@@ -371,14 +397,17 @@ def sent_messages_report(realm):
     return make_table(title, cols, rows)
 
 def ad_hoc_queries():
+    # type: () -> List[Dict[str, str]]
     def get_page(query, cols, title):
+        # type: (str, List[str], str) -> Dict[str, str]
         cursor = connection.cursor()
         cursor.execute(query)
         rows = cursor.fetchall()
-        rows = map(list, rows)
+        rows = list(map(list, rows))
         cursor.close()
 
         def fix_rows(i, fixup_func):
+            # type: (int, Union[Callable[[Realm], mark_safe], Callable[[datetime], str]]) -> None
             for row in rows:
                 row[i] = fixup_func(row[i])
 
@@ -541,8 +570,9 @@ def ad_hoc_queries():
 @zulip_internal
 @has_request_variables
 def get_activity(request):
-    duration_content, realm_minutes = user_activity_intervals()
-    counts_content = realm_summary_table(realm_minutes)
+    # type: (HttpRequest) -> HttpResponse
+    duration_content, realm_minutes = user_activity_intervals() # type: Tuple[mark_safe, Dict[str, float]]
+    counts_content = realm_summary_table(realm_minutes) # type: str
     data = [
         ('Counts', counts_content),
         ('Durations', duration_content),
@@ -555,10 +585,11 @@ def get_activity(request):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title, is_home=True),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 def get_user_activity_records_for_realm(realm, is_bot):
+    # type: (str, bool) -> QuerySet
     fields = [
         'user_profile__full_name',
         'user_profile__email',
@@ -578,6 +609,7 @@ def get_user_activity_records_for_realm(realm, is_bot):
     return records
 
 def get_user_activity_records_for_email(email):
+    # type: (str) -> List[QuerySet]
     fields = [
         'user_profile__full_name',
         'query',
@@ -594,6 +626,7 @@ def get_user_activity_records_for_email(email):
     return records
 
 def raw_user_activity_table(records):
+    # type: (List[QuerySet]) -> str
     cols = [
         'query',
         'client',
@@ -602,6 +635,7 @@ def raw_user_activity_table(records):
     ]
 
     def row(record):
+        # type: (QuerySet) -> List[Any]
         return [
                 record.query,
                 record.client.name,
@@ -609,13 +643,20 @@ def raw_user_activity_table(records):
                 format_date_for_activity_reports(record.last_visit)
         ]
 
-    rows = map(row, records)
+    rows = list(map(row, records))
     title = 'Raw Data'
     return make_table(title, cols, rows)
 
 def get_user_activity_summary(records):
-    summary = {}
+    # type: (List[QuerySet]) -> Dict[str, Dict[str, Any]]
+    #: `Any` used above should be `Union(int, datetime)`.
+    #: However current version of `Union` does not work inside other function.
+    #: We could use something like:
+    # `Union[Dict[str, Dict[str, int]], Dict[str, Dict[str, datetime]]]`
+    #: but that would require this long `Union` to carry on throughout inner functions.
+    summary = {} # type: Dict[str, Dict[str, Any]]
     def update(action, record):
+        # type: (str, QuerySet) -> None
         if action not in summary:
             summary[action] = dict(
                     count=record.count,
@@ -649,7 +690,7 @@ def get_user_activity_summary(records):
             update('website', record)
         if ('send_message' in query) or re.search('/api/.*/external/.*', query):
             update('send', record)
-        if query in ['/json/update_pointer', '/api/v1/update_pointer']:
+        if query in ['/json/update_pointer', '/json/users/me/pointer', '/api/v1/update_pointer']:
             update('pointer', record)
         update(client, record)
 
@@ -657,24 +698,28 @@ def get_user_activity_summary(records):
     return summary
 
 def format_date_for_activity_reports(date):
+    # type: (Optional[datetime]) -> str
     if date:
         return date.astimezone(eastern_tz).strftime('%Y-%m-%d %H:%M')
     else:
         return ''
 
 def user_activity_link(email):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_user_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(email=email))
     email_link = '<a href="%s">%s</a>' % (url, email)
     return mark_safe(email_link)
 
 def realm_activity_link(realm):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_realm_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(realm=realm))
     realm_link = '<a href="%s">%s</a>' % (url, realm)
     return mark_safe(realm_link)
 
 def realm_client_table(user_summaries):
+    # type: (Dict[str, Dict[str, Dict[str, Any]]]) -> str
     exclude_keys = [
             'internal',
             'name',
@@ -719,6 +764,7 @@ def realm_client_table(user_summaries):
     return make_table(title, cols, rows)
 
 def user_activity_summary_table(user_summary):
+    # type: (Dict[str, Dict[str, Any]]) -> str
     rows = []
     for k, v in user_summary.items():
         if k == 'name':
@@ -745,28 +791,33 @@ def user_activity_summary_table(user_summary):
     return make_table(title, cols, rows)
 
 def realm_user_summary_table(all_records, admin_emails):
+    # type: (List[QuerySet], Set[text_type]) -> Tuple[Dict[str, Dict[str, Any]], str]
     user_records = {}
 
     def by_email(record):
+        # type: (QuerySet) -> str
         return record.user_profile.email
 
     for email, records in itertools.groupby(all_records, by_email):
         user_records[email] = get_user_activity_summary(list(records))
 
     def get_last_visit(user_summary, k):
+        # type: (Dict[str, Dict[str, datetime]], str) -> Optional[datetime]
         if k in user_summary:
             return user_summary[k]['last_visit']
         else:
             return None
 
     def get_count(user_summary, k):
+        # type: (Dict[str, Dict[str, str]], str) -> str
         if k in user_summary:
             return user_summary[k]['count']
         else:
             return ''
 
     def is_recent(val):
-        age = datetime.now(val.tzinfo) - val
+        # type: (Optional[datetime]) -> bool
+        age = datetime.now(val.tzinfo) - val # type: ignore # datetie.now tzinfo bug.
         return age.total_seconds() < 5 * 60
 
     rows = []
@@ -776,18 +827,19 @@ def realm_user_summary_table(all_records, admin_emails):
         cells = [user_summary['name'], email_link, sent_count]
         row_class = ''
         for field in ['use', 'send', 'pointer', 'desktop', 'ZulipiOS', 'Android']:
-            val = get_last_visit(user_summary, field)
+            visit = get_last_visit(user_summary, field)
             if field == 'use':
-                if val and is_recent(val):
+                if visit and is_recent(visit):
                     row_class += ' recently_active'
                 if email in admin_emails:
                     row_class += ' admin'
-            val = format_date_for_activity_reports(val)
+            val = format_date_for_activity_reports(visit)
             cells.append(val)
         row = dict(cells=cells, row_class=row_class)
         rows.append(row)
 
     def by_used_time(row):
+        # type: (Dict[str, Sequence[str]]) -> str
         return row['cells'][3]
 
     rows = sorted(rows, key=by_used_time, reverse=True)
@@ -811,20 +863,19 @@ def realm_user_summary_table(all_records, admin_emails):
 
 @zulip_internal
 def get_realm_activity(request, realm):
-    data = []
-    all_records = {}
-    all_user_records = {}
+    # type: (HttpRequest, str) -> HttpResponse
+    data = [] # type: List[Tuple[str, str]]
+    all_user_records = {} # type: Dict[str, Any]
 
     try:
-        admins = Realm.objects.get(domain=realm).get_admin_users()
+        admins = get_realm(realm).get_admin_users()
     except Realm.DoesNotExist:
         return HttpResponseNotFound("Realm %s does not exist" % (realm,))
 
     admin_emails = {admin.email for admin in admins}
 
     for is_bot, page_title in [(False,  'Humans'), (True, 'Bots')]:
-        all_records = get_user_activity_records_for_realm(realm, is_bot)
-        all_records = list(all_records)
+        all_records = list(get_user_activity_records_for_realm(realm, is_bot))
 
         user_records, content = realm_user_summary_table(all_records, admin_emails)
         all_user_records.update(user_records)
@@ -849,14 +900,15 @@ def get_realm_activity(request, realm):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, realm_link=realm_link, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 @zulip_internal
 def get_user_activity(request, email):
+    # type: (HttpRequest, str) -> HttpResponse
     records = get_user_activity_records_for_email(email)
 
-    data = []
+    data = [] # type: List[Tuple[str, str]]
     user_summary = get_user_activity_summary(records)
     content = user_activity_summary_table(user_summary)
 
@@ -869,5 +921,5 @@ def get_user_activity(request, email):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )

api/MANIFEST.in

@@ -8,3 +8,4 @@ include examples/unsubscribe
 include examples/list-members
 include examples/list-subscriptions
 include examples/print-messages
+include examples/recent-messages

api/README.md

@@ -31,11 +31,24 @@ file is as follows:
     key=<api key from the web interface>
     email=<your email address>
     site=<your Zulip server's URI>
+    insecure=<true or false, true means do not verify the server certificate>
+    cert_bundle=<path to a file containing CA or server certificates to trust>
+
+If omitted, these settings have the following defaults:
+
+    site=https://api.zulip.com
+    insecure=false
+    cert_bundle=<the default CA bundle trusted by Python>
 
 Alternatively, you may explicitly use "--user" and "--api-key" in our
 examples, which is especially useful if you are running several bots
-which share a home directory.  There is also a "--site" option for
-setting the Zulip server on the command line.
+which share a home directory.
+
+The command line equivalents for other configuration options are:
+
+    --site=<your Zulip server's URI>
+    --insecure
+    --cert-bundle=<file>
 
 You can obtain your Zulip API key, create bots, and manage bots all
 from your Zulip [settings page](https://zulip.com/#settings).
@@ -101,3 +114,46 @@ Alternatively, if you don't want to use your ~/.zuliprc file:
         --api-key a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5 \
         hamlet@example.com cordelia@example.com -m \
         "Conscience doth make cowards of us all."
+
+#### Working with an untrusted server certificate
+
+If your server has either a self-signed certificate, or a certificate signed
+by a CA that you don't wish to globally trust then by default the API will
+fail with an SSL verification error.
+
+You can add `insecure=true` to your .zuliprc file.
+
+    [api]
+    site=https://zulip.example.com
+    insecure=true
+
+This disables verification of the server certificate, so connections are
+encrypted but unauthenticated. This is not secure, but may be good enough
+for a development environment.
+
+
+You can explicitly trust the server certificate using `cert_bundle=<filename>`
+in your .zuliprc file.
+
+    [api]
+    site=https://zulip.example.com
+    cert_bundle=/home/bots/certs/zulip.example.com.crt
+
+You can also explicitly trust a different set of Certificate Authorities from
+the default bundle that is trusted by Python. For example to trust a company
+internal CA.
+
+    [api]
+    site=https://zulip.example.com
+    cert_bundle=/home/bots/certs/example.com.ca-bundle
+
+Save the server certificate (or the CA certificate) in its own file,
+converting to PEM format first if necessary.
+Verify that the certificate you have saved is the same as the one on the
+server.
+
+The `cert_bundle` option trusts the server / CA certificate only for
+interaction with the zulip site, and is relatively secure.
+
+Note that a certificate bundle is merely one or more certificates combined
+into a single file.

api/examples/create-user

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012-2014 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 from os import path
 import optparse
@@ -46,9 +47,9 @@ parser.add_option('--new-short-name')
 
 client = zulip.init_from_options(options)
 
-print client.create_user({
+print(client.create_user({
         'email': options.new_email,
         'password': options.new_password,
         'full_name': options.new_full_name,
         'short_name': options.new_short_name
-        })
+        }))

api/examples/edit-message

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -53,4 +54,4 @@ if options.subject != "":
     message_data["subject"] = options.subject
 if options.content != "":
     message_data["content"] = options.content
-print client.update_message(message_data)
+print(client.update_message(message_data))

api/examples/get-public-streams

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,4 +44,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.get_streams(include_public=True, include_subscribed=False)
+print(client.get_streams(include_public=True, include_subscribed=False))

api/examples/list-members

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 for user in client.get_members()["members"]:
-    print user["full_name"], user["email"]
+    print(user["full_name"], user["email"])

api/examples/list-subscriptions

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.list_subscriptions()
+print(client.list_subscriptions())

api/examples/print-events

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,7 +44,7 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 def print_event(event):
-    print event
+    print(event)
 
 # This is a blocking call, and will continuously poll for new events
 # Note also the filter here is messages to the stream Denmark; if you

api/examples/print-messages

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,7 +44,7 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 def print_message(message):
-    print message
+    print(message)
 
 # This is a blocking call, and will continuously poll for new messages
 client.call_on_each_message(print_message)

api/examples/print-next-message

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.get_messages({})
+print(client.get_messages({}))

api/examples/recent-messages

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright © 2012 Zulip, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from __future__ import print_function
+import sys
+import os
+import json
+import optparse
+
+usage = """recent-messages [options] --count=<no. of previous messages> --user=<sender's email address> --api-key=<sender's api key>
+
+Prints out last count messages recieved by the indicated bot or user
+
+Example: recent-messages --count=101 --user=username@example.com --api-key=a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5
+
+You can omit --user and --api-key arguments if you have a properly set up ~/.zuliprc
+"""
+sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
+import zulip
+
+parser = optparse.OptionParser(usage=usage)
+parser.add_option('--count', default=100)
+parser.add_option_group(zulip.generate_option_group(parser))
+(options, args) = parser.parse_args()
+
+client = zulip.init_from_options(options)
+
+req = {
+    'narrow': [["stream", "Denmark"]],
+    'num_before': options.count,
+    'num_after': 0,
+    'anchor': 1000000000,
+    'apply_markdown': False
+}
+
+old_messages = client.do_api_query(req, zulip.API_VERSTRING + 'messages', method='GET')
+if 'messages' in old_messages:
+    for message in old_messages['messages']:
+        print(json.dumps(message, indent=4))
+else:
+    print([])

api/examples/send-message

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -54,4 +55,4 @@ message_data = {
     "subject": options.subject,
     "to": args,
 }
-print client.send_message(message_data)
+print(client.send_message(message_data))

api/examples/subscribe

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -45,8 +46,8 @@ parser.add_option('--streams', default='')
 client = zulip.init_from_options(options)
 
 if options.streams == "":
-    print >>sys.stderr, "Usage:", parser.usage
+    print("Usage:", parser.usage, file=sys.stderr)
     sys.exit(1)
 
-print client.add_subscriptions([{"name": stream_name} for stream_name in
-                                options.streams.split()])
+print(client.add_subscriptions([{"name": stream_name} for stream_name in
+                                options.streams.split()]))

api/examples/unsubscribe

@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -45,7 +46,7 @@ parser.add_option('--streams', default='')
 client = zulip.init_from_options(options)
 
 if options.streams == "":
-    print >>sys.stderr, "Usage:", parser.usage
+    print("Usage:", parser.usage, file=sys.stderr)
     sys.exit(1)
 
-print client.remove_subscriptions(options.streams.split())
+print(client.remove_subscriptions(options.streams.split()))

api/integrations/asana/zulip_asana_mirror

@@ -30,6 +30,7 @@
 #
 # python-dateutil is a dependency for this script.
 
+from __future__ import print_function
 import base64
 from datetime import datetime, timedelta
 
@@ -37,16 +38,16 @@ import json
 import logging
 import os
 import time
-import urllib2
+from six.moves import urllib
 
 import sys
 
 try:
     import dateutil.parser
     import dateutil.tz
-except ImportError, e:
-    print >>sys.stderr, e
-    print >>sys.stderr, "Please install the python-dateutil package."
+except ImportError as e:
+    print(e, file=sys.stderr)
+    print("Please install the python-dateutil package.", file=sys.stderr)
     exit(1)
 
 sys.path.insert(0, os.path.dirname(__file__))
@@ -74,8 +75,8 @@ def fetch_from_asana(path):
     headers = {"Authorization": "Basic %s" % auth}
 
     url = "https://app.asana.com/api/1.0" + path
-    request = urllib2.Request(url, None, headers)
-    result = urllib2.urlopen(request)
+    request = urllib.request.Request(url, None, headers)
+    result = urllib.request.urlopen(request)
 
     return json.load(result)
 
@@ -189,7 +190,7 @@ def since():
                 timestamp = float(datestring)
                 max_timestamp_processed = datetime.fromtimestamp(timestamp)
                 logging.info("Reading from resume file: " + datestring)
-        except (ValueError,IOError) as e:
+        except (ValueError, IOError) as e:
             logging.warn("Could not open resume file: %s" % (
                     e.message or e.strerror,))
             max_timestamp_processed = default_since()

api/integrations/basecamp/zulip_basecamp_mirror

@@ -26,6 +26,7 @@
 # or preferably on a server.
 # You may need to install the python-requests library.
 
+from __future__ import absolute_import
 import requests
 import logging
 import time
@@ -33,7 +34,8 @@ import re
 import sys
 import os
 from datetime import datetime, timedelta
-from HTMLParser import HTMLParser
+from six.moves.html_parser import HTMLParser
+import six
 
 sys.path.insert(0, os.path.dirname(__file__))
 import zulip_basecamp_config as config
@@ -80,7 +82,7 @@ def check_permissions():
 
 # builds the message dict for sending a message with the Zulip API
 def build_message(event):
-    if not (event.has_key('bucket') and event.has_key('creator') and event.has_key('html_url')):
+    if not ('bucket' in event and 'creator' in event and 'html_url' in event):
         logging.error("Perhaps the Basecamp API changed behavior? "
                       "This event doesn't have the expected format:\n%s" %(event,))
         return None
@@ -92,7 +94,7 @@ def build_message(event):
     action = htmlParser.unescape(re.sub(r"<[^<>]+>", "", event.get('action', '')))
     target = htmlParser.unescape(event.get('target', ''))
     # Some events have "excerpts", which we blockquote
-    excerpt = htmlParser.unescape(event.get('excerpt',''))
+    excerpt = htmlParser.unescape(event.get('excerpt', ''))
     if excerpt.strip() == "":
         message = '**%s** %s [%s](%s).' % (event['creator']['name'], action, target, event['html_url'])
     else:
@@ -116,13 +118,13 @@ def run_mirror():
         since = re.search(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}-\d{2}:\d{2}", since)
         assert since, "resume file does not meet expected format"
         since = since.string
-    except (AssertionError,IOError) as e:
+    except (AssertionError, IOError) as e:
         logging.warn("Could not open resume file: %s" % (e.message or e.strerror,))
         since = (datetime.utcnow() - timedelta(hours=config.BASECAMP_INITIAL_HISTORY_HOURS)).isoformat() + "-00:00"
     try:
         # we use an exponential backoff approach when we get 429 (Too Many Requests).
         sleepInterval = 1
-        while 1:
+        while True:
             time.sleep(sleepInterval)
             response = requests.get("https://basecamp.com/%s/api/v1/events.json" % (config.BASECAMP_ACCOUNT_ID),
                                     params={'since': since},
@@ -170,7 +172,7 @@ def run_mirror():
 
 
 if __name__ == "__main__":
-    if not isinstance(config.RESUME_FILE, basestring):
+    if not isinstance(config.RESUME_FILE, six.string_types):
         sys.stderr("RESUME_FILE path not given; refusing to continue")
     check_permissions()
     if config.LOG_FILE:

api/integrations/codebase/zulip_codebase_mirror

@@ -29,6 +29,8 @@
 #
 # python-dateutil is a dependency for this script.
 
+from __future__ import print_function
+from __future__ import absolute_import
 import requests
 import logging
 import time
@@ -36,13 +38,14 @@ import sys
 import os
 
 from datetime import datetime, timedelta
+import six
 
 
 try:
     import dateutil.parser
-except ImportError, e:
-    print >>sys.stderr, e
-    print >>sys.stderr, "Please install the python-dateutil package."
+except ImportError as e:
+    print(e, file=sys.stderr)
+    print("Please install the python-dateutil package.", file=sys.stderr)
     exit(1)
 
 sys.path.insert(0, os.path.dirname(__file__))
@@ -110,7 +113,7 @@ def handle_event(event):
         project_name = raw_props.get('name')
         project_repo_type = raw_props.get('scm_type')
 
-        url = make_url("projects/%s" % project_link)
+        url = make_url("projects/%s" % (project_link,))
         scm = "of type %s" % (project_repo_type,) if project_repo_type else ""
 
 
@@ -132,7 +135,7 @@ def handle_event(event):
             content = "%s deleted branch %s from %s" % (actor_name, branch, project)
         else:
             if new_ref:
-                branch = "new branch %s" % (branch, )
+                branch = "new branch %s" % (branch,)
             content = "%s pushed %s commit(s) to %s in project %s:\n\n" % \
                         (actor_name, num_commits, branch, project)
             for commit in raw_props.get('commits'):
@@ -271,13 +274,13 @@ def run_mirror():
         else:
             timestamp = int(timestamp, 10)
             since = datetime.fromtimestamp(timestamp)
-    except (ValueError,IOError) as e:
+    except (ValueError, IOError) as e:
         logging.warn("Could not open resume file: %s" % (e.message or e.strerror,))
         since = default_since()
 
     try:
         sleepInterval = 1
-        while 1:
+        while True:
             events = make_api_call("activity")[::-1]
             if events is not None:
                 sleepInterval = 1
@@ -314,7 +317,7 @@ def check_permissions():
         sys.stderr(e)
 
 if __name__ == "__main__":
-    if not isinstance(config.RESUME_FILE, basestring):
+    if not isinstance(config.RESUME_FILE, six.string_types):
         sys.stderr("RESUME_FILE path not given; refusing to continue")
     check_permissions()
     if config.LOG_FILE:

api/integrations/git/post-receive

@@ -29,13 +29,14 @@
 # For example:
 #  aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master
 
+from __future__ import absolute_import
 import os
 import sys
 import subprocess
 import os.path
 
 sys.path.insert(0, os.path.dirname(__file__))
-import zulip_git_config as config
+from . import zulip_git_config as config
 VERSION = "0.9"
 
 if config.ZULIP_API_PATH is not None:

api/integrations/hg/zulip-changegroup.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 #
 # Zulip hook for Mercurial changeset pushes.
-# Copyright © 2012-2014 Zulip, Inc.
+# Copyright © 2012-2014 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -25,8 +25,10 @@
 #
 # This hook is called when changesets are pushed to the master repository (ie
 # `hg push`). See https://zulip.com/integrations for installation instructions.
+from __future__ import absolute_import
 
 import zulip
+from six.moves import range
 
 VERSION = "0.9"
 

api/integrations/jira/org/humbug/jira/ZulipListener.groovy

@@ -67,7 +67,7 @@ class ZulipListener extends AbstractIssueEventListener {
                                   author, issueUrlMd, comment)
           break
         case ISSUE_CREATED_ID:
-          content = String.format("%s **created** %s priority %s, assigned to **%s**: \n\n> %s",
+          content = String.format("%s **created** %s priority %s, assigned to @**%s**: \n\n> %s",
                                   author, issueUrlMd, event.issue.priorityObject.name,
                                   assignee, title)
           break

api/integrations/perforce/zulip_change-commit.py

@@ -33,6 +33,7 @@ For example:
   1234 //depot/security/src/
 
 '''
+from __future__ import print_function
 
 import os
 import sys
@@ -59,12 +60,12 @@ try:
     changelist = int(sys.argv[1])
     changeroot = sys.argv[2]
 except IndexError:
-    print >> sys.stderr, "Wrong number of arguments.\n\n",
-    print >> sys.stderr,  __doc__
+    print("Wrong number of arguments.\n\n", end=' ', file=sys.stderr)
+    print(__doc__, file=sys.stderr)
     sys.exit(-1)
 except ValueError:
-    print >> sys.stderr, "First argument must be an integer.\n\n",
-    print >> sys.stderr, __doc__
+    print("First argument must be an integer.\n\n", end=' ', file=sys.stderr)
+    print(__doc__, file=sys.stderr)
     sys.exit(-1)
 
 metadata = git_p4.p4_describe(changelist)

api/integrations/rss/rss-bot

@@ -23,16 +23,17 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import calendar
 import errno
 import hashlib
-from HTMLParser import HTMLParser
+from six.moves.html_parser import HTMLParser
 import logging
 import optparse
 import os
 import sys
 import time
-import urlparse
+from six.moves import urllib
 
 import feedparser
 import zulip
@@ -87,7 +88,7 @@ def mkdir_p(path):
     # Python doesn't have an analog to `mkdir -p` < Python 3.2.
     try:
         os.makedirs(path)
-    except OSError, e:
+    except OSError as e:
         if e.errno == errno.EEXIST and os.path.isdir(path):
             pass
         else:
@@ -97,7 +98,7 @@ try:
     mkdir_p(opts.data_dir)
 except OSError:
     # We can't write to the logfile, so just print and give up.
-    print >>sys.stderr, "Unable to store RSS data at %s." % (opts.data_dir,)
+    print("Unable to store RSS data at %s." % (opts.data_dir,), file=sys.stderr)
     exit(1)
 
 log_file = os.path.join(opts.data_dir, "rss-bot.log")
@@ -169,7 +170,7 @@ client = zulip.Client(email=opts.email, api_key=opts.api_key,
 first_message = True
 
 for feed_url in feed_urls:
-    feed_file = os.path.join(opts.data_dir, urlparse.urlparse(feed_url).netloc)
+    feed_file = os.path.join(opts.data_dir, urllib.parse.urlparse(feed_url).netloc)
 
     try:
         with open(feed_file, "r") as f:

api/integrations/trac/zulip_trac.py

@@ -107,8 +107,8 @@ class ZulipPlugin(Component):
         field_changes = []
         for key in old_values.keys():
             if key == "description":
-                content += '- Changed %s from %s to %s' % (key, markdown_block(old_values.get(key)),
-                                                           markdown_block(ticket.values.get(key)))
+                content += '- Changed %s from %s\n\nto %s' % (key, markdown_block(old_values.get(key)),
+                                                              markdown_block(ticket.values.get(key)))
             elif old_values.get(key) == "":
                 field_changes.append('%s: => **%s**' % (key, ticket.values.get(key)))
             elif ticket.values.get(key) == "":

api/integrations/trac/zulip_trac_config.py

@@ -34,13 +34,13 @@ TRAC_BASE_TICKET_URL = "https://trac.example.com/ticket"
 # and annoying.  We solve this issue by only sending a notification
 # for changes to the fields listed below.
 #
-# Total list of possible fields is:
+# TRAC_NOTIFY_FIELDS lets you specify which fields will trigger a
+# Zulip notification in response to a trac update; you should change
+# this list to match your team's workflow.  The complete list of
+# possible fields is:
+#
 # (priority, milestone, cc, owner, keywords, component, severity,
 #  type, versions, description, resolution, summary, comment)
-#
-# The following is the list of fields which can be changed without
-# triggering a Zulip notification; change these to match your team's
-# workflow.
 TRAC_NOTIFY_FIELDS = ["description", "summary", "resolution", "comment", "owner"]
 
 ## If properly installed, the Zulip API should be in your import

api/integrations/twitter/twitter-bot

@@ -23,10 +23,11 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import os
 import sys
 import optparse
-import ConfigParser
+import six.moves.configparser
 
 import zulip
 VERSION = "0.9"
@@ -85,14 +86,14 @@ if not options.twitter_id:
     parser.error('You must specify --twitter-id')
 
 try:
-    config = ConfigParser.ConfigParser()
+    config = six.moves.configparser.ConfigParser()
     config.read(CONFIGFILE)
 
     consumer_key = config.get('twitter', 'consumer_key')
     consumer_secret = config.get('twitter', 'consumer_secret')
     access_token_key = config.get('twitter', 'access_token_key')
     access_token_secret = config.get('twitter', 'access_token_secret')
-except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+except (six.moves.configparser.NoSectionError, six.moves.configparser.NoOptionError):
    parser.error("Please provide a ~/.zulip_twitterrc")
 
 if not consumer_key or not consumer_secret or not access_token_key or not access_token_secret:
@@ -112,17 +113,17 @@ api = twitter.Api(consumer_key=consumer_key,
 user = api.VerifyCredentials()
 
 if not user.GetId():
-    print "Unable to log in to twitter with supplied credentials. Please double-check and try again"
+    print("Unable to log in to twitter with supplied credentials. Please double-check and try again")
     sys.exit()
 
 try:
     since_id = config.getint('twitter', 'since_id')
-except ConfigParser.NoOptionError:
+except six.moves.configparser.NoOptionError:
     since_id = -1
 
 try:
     user_id = config.get('twitter', 'user_id')
-except ConfigParser.NoOptionError:
+except six.moves.configparser.NoOptionError:
     user_id = options.twitter_id
 
 client = zulip.Client(
@@ -154,7 +155,7 @@ for status in statuses[::-1][:options.limit_tweets]:
 
     if ret['result'] == 'error':
         # If sending failed (e.g. no such stream), abort and retry next time
-        print "Error sending message to zulip: %s" % ret['msg']
+        print("Error sending message to zulip: %s" % ret['msg'])
         break
     else:
         since_id = status.GetId()

api/integrations/twitter/twitter-search-bot

@@ -23,10 +23,11 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import os
 import sys
 import optparse
-import ConfigParser
+import six.moves.configparser
 
 import zulip
 VERSION = "0.9"
@@ -107,14 +108,14 @@ if not opts.search_terms:
     parser.error('You must specify a search term.')
 
 try:
-    config = ConfigParser.ConfigParser()
+    config = six.moves.configparser.ConfigParser()
     config.read(CONFIGFILE)
 
     consumer_key = config.get('twitter', 'consumer_key')
     consumer_secret = config.get('twitter', 'consumer_secret')
     access_token_key = config.get('twitter', 'access_token_key')
     access_token_secret = config.get('twitter', 'access_token_secret')
-except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+except (six.moves.configparser.NoSectionError, six.moves.configparser.NoOptionError):
    parser.error("Please provide a ~/.zulip_twitterrc")
 
 if not (consumer_key and consumer_secret and access_token_key and access_token_secret):
@@ -122,7 +123,7 @@ if not (consumer_key and consumer_secret and access_token_key and access_token_s
 
 try:
     since_id = config.getint('search', 'since_id')
-except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+except (six.moves.configparser.NoOptionError, six.moves.configparser.NoSectionError):
     since_id = 0
 
 try:
@@ -138,8 +139,8 @@ api = twitter.Api(consumer_key=consumer_key,
 user = api.VerifyCredentials()
 
 if not user.GetId():
-    print "Unable to log in to Twitter with supplied credentials.\
-Please double-check and try again."
+    print("Unable to log in to Twitter with supplied credentials.\
+Please double-check and try again.")
     sys.exit()
 
 client = zulip.Client(
@@ -182,7 +183,7 @@ for status in statuses[::-1][:opts.limit_tweets]:
 
     if ret['result'] == 'error':
         # If sending failed (e.g. no such stream), abort and retry next time
-        print "Error sending message to zulip: %s" % ret['msg']
+        print("Error sending message to zulip: %s" % ret['msg'])
         break
     else:
         since_id = status.GetId()

api/setup.py

@@ -1,6 +1,9 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+from __future__ import print_function
+from typing import Any, Generator, List, Tuple
+
 import os
 import sys
 
@@ -9,12 +12,13 @@ import itertools
 def version():
     version_py = os.path.join(os.path.dirname(__file__), "zulip", "__init__.py")
     with open(version_py) as in_handle:
-        version_line = itertools.dropwhile(lambda x: not x.startswith("__version__"),
-                                           in_handle).next()
+        version_line = next(itertools.dropwhile(lambda x: not x.startswith("__version__"),
+                                           in_handle))
     version = version_line.split('=')[-1].strip().replace('"', '')
     return version
 
 def recur_expand(target_root, dir):
+    # type: (Any, Any) -> Generator[Tuple[str, List[str]], None, None]
     for root, _, files in os.walk(dir):
         paths = [os.path.join(root, f) for f in files]
         if len(paths):
@@ -39,7 +43,7 @@ package_info = dict(
     data_files=[('share/zulip/examples', ["examples/zuliprc", "examples/send-message", "examples/subscribe",
                                            "examples/get-public-streams", "examples/unsubscribe",
                                            "examples/list-members", "examples/list-subscriptions",
-                                           "examples/print-messages"])] + \
+                                           "examples/print-messages", "examples/recent-messages"])] + \
         list(recur_expand('share/zulip', 'integrations/')),
     scripts=["bin/zulip-send"],
 )
@@ -47,6 +51,8 @@ package_info = dict(
 setuptools_info = dict(
     install_requires=['requests>=0.12.1',
                       'simplejson',
+                      'six',
+                      'typing',
     ],
 )
 
@@ -60,13 +66,13 @@ except ImportError:
     try:
         import simplejson
     except ImportError:
-        print >>sys.stderr, "simplejson is not installed"
+        print("simplejson is not installed", file=sys.stderr)
         sys.exit(1)
     try:
         import requests
-        assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
+        assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1')) # type: ignore # https://github.com/JukkaL/mypy/issues/1165
     except (ImportError, AssertionError):
-        print >>sys.stderr, "requests >=0.12.1 is not installed"
+        print("requests >=0.12.1 is not installed", file=sys.stderr)
         sys.exit(1)
 
 

api/zulip/__init__.py

@@ -20,30 +20,34 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
+from __future__ import absolute_import
+from __future__ import division
 import simplejson
 import requests
 import time
 import traceback
-import urlparse
 import sys
 import os
 import optparse
 import platform
-import urllib
 import random
 from distutils.version import LooseVersion
 
-from ConfigParser import SafeConfigParser
+from six.moves.configparser import SafeConfigParser
+from six.moves import urllib
 import logging
+import six
+from typing import Any, Dict
 
 
-__version__ = "0.2.4"
+__version__ = "0.2.5"
 
 logger = logging.getLogger(__name__)
 
 # Check that we have a recent enough version
 # Older versions don't provide the 'json' attribute on responses.
-assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
+assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1')) # type: ignore # https://github.com/python/mypy/issues/1165 and https://github.com/python/typeshed/pull/206
 # In newer versions, the 'json' attribute is a function, not a property
 requests_json_is_function = callable(requests.Response.json)
 
@@ -87,7 +91,7 @@ class RandomExponentialBackoff(CountingBackoff):
         try:
             logger.warning(message)
         except NameError:
-            print message
+            print(message)
         time.sleep(delay)
 
 def _default_client():
@@ -117,6 +121,19 @@ def generate_option_group(parser, prefix=''):
                      default=None,
                      dest="zulip_client",
                      help=optparse.SUPPRESS_HELP)
+    group.add_option('--insecure',
+                     action='store_true',
+                     dest='insecure',
+                     help='''Do not verify the server certificate.
+                          The https connection will not be secure.''')
+    group.add_option('--cert-bundle',
+                     action='store',
+                     dest='cert_bundle',
+                     help='''Specify a file containing either the
+                          server certificate, or a set of trusted
+                          CA certificates. This will be used to
+                          verify the server's identity. All
+                          certificates should be PEM encoded.''')
     return group
 
 def init_from_options(options, client=None):
@@ -126,7 +143,8 @@ def init_from_options(options, client=None):
         client = _default_client()
     return Client(email=options.zulip_email, api_key=options.zulip_api_key,
                   config_file=options.zulip_config_file, verbose=options.verbose,
-                  site=options.zulip_site, client=client)
+                  site=options.zulip_site, client=client,
+                  cert_bundle=options.cert_bundle, insecure=options.insecure)
 
 def get_default_config_filename():
     config_file = os.path.join(os.environ["HOME"], ".zuliprc")
@@ -138,17 +156,16 @@ def get_default_config_filename():
 class Client(object):
     def __init__(self, email=None, api_key=None, config_file=None,
                  verbose=False, retry_on_errors=True,
-                 site=None, client=None):
+                 site=None, client=None,
+                 cert_bundle=None, insecure=None):
         if client is None:
             client = _default_client()
-        if None in (api_key, email):
-            if config_file is None:
-                config_file = get_default_config_filename()
-            if not os.path.exists(config_file):
-                raise RuntimeError("api_key or email not specified and %s does not exist"
-                                   % (config_file,))
+
+        if config_file is None:
+            config_file = get_default_config_filename()
+        if os.path.exists(config_file):
             config = SafeConfigParser()
-            with file(config_file, 'r') as f:
+            with open(config_file, 'r') as f:
                 config.readfp(f, config_file)
             if api_key is None:
                 api_key = config.get("api", "key")
@@ -156,6 +173,22 @@ class Client(object):
                 email = config.get("api", "email")
             if site is None and config.has_option("api", "site"):
                 site = config.get("api", "site")
+            if cert_bundle is None and config.has_option("api", "cert_bundle"):
+                cert_bundle = config.get("api", "cert_bundle")
+            if insecure is None and config.has_option("api", "insecure"):
+                # Be quite strict about what is accepted so that users don't
+                # disable security unintentionally.
+                insecure_setting = config.get("api", "insecure").lower()
+                if insecure_setting == "true":
+                    insecure = True
+                elif insecure_setting == "false":
+                    insecure = False
+                else:
+                    raise RuntimeError("insecure is set to '%s', it must be 'true' or 'false' if it is used in %s"
+                                       % (insecure_setting, config_file))
+        elif None in (api_key, email):
+            raise RuntimeError("api_key or email not specified and %s does not exist"
+                               % (config_file,))
 
         self.api_key = api_key
         self.email = email
@@ -175,6 +208,17 @@ class Client(object):
         self.retry_on_errors = retry_on_errors
         self.client_name = client
 
+        if insecure:
+            self.tls_verification=False
+        elif cert_bundle is not None:
+            if not os.path.isfile(cert_bundle):
+                raise RuntimeError("tls bundle '%s' does not exist"
+                                   %(cert_bundle,))
+            self.tls_verification=cert_bundle
+        else:
+            # Default behavior: verify against system CA certificates
+            self.tls_verification=True
+
     def get_user_agent(self):
         vendor = ''
         vendor_version = ''
@@ -202,8 +246,8 @@ class Client(object):
     def do_api_query(self, orig_request, url, method="POST", longpolling = False):
         request = {}
 
-        for (key, val) in orig_request.iteritems():
-            if not (isinstance(val, str) or isinstance(val, unicode)):
+        for (key, val) in six.iteritems(orig_request):
+            if not (isinstance(val, str) or isinstance(val, six.text_type)):
                 request[key] = simplejson.dumps(val)
             else:
                 request[key] = val
@@ -212,7 +256,7 @@ class Client(object):
             'had_error_retry': False,
             'request': request,
             'failures': 0,
-        }
+        } # type: Dict[str, Any]
 
         def error_retry(error_string):
             if not self.retry_on_errors or query_state["failures"] >= 10:
@@ -233,9 +277,9 @@ class Client(object):
         def end_error_retry(succeeded):
             if query_state["had_error_retry"] and self.verbose:
                 if succeeded:
-                    print "Success!"
+                    print("Success!")
                 else:
-                    print "Failed!"
+                    print("Failed!")
 
         while True:
             try:
@@ -246,10 +290,10 @@ class Client(object):
                 kwargs = {kwarg: query_state["request"]}
                 res = requests.request(
                         method,
-                        urlparse.urljoin(self.base_url, url),
+                        urllib.parse.urljoin(self.base_url, url),
                         auth=requests.auth.HTTPBasicAuth(self.email,
                                                          self.api_key),
-                        verify=True, timeout=90,
+                        verify=self.tls_verification, timeout=90,
                         headers={"User-agent": self.get_user_agent()},
                         **kwargs)
 
@@ -300,10 +344,15 @@ class Client(object):
                     "status_code": res.status_code}
 
     @classmethod
-    def _register(cls, name, url=None, make_request=(lambda request={}: request),
+    def _register(cls, name, url=None, make_request=None,
                   method="POST", computed_url=None, **query_kwargs):
         if url is None:
             url = name
+        if make_request is None:
+            def make_request(request=None):
+                if request is None:
+                    request = {}
+                return request
         def call(self, *args, **kwargs):
             request = make_request(*args, **kwargs)
             if computed_url is not None:
@@ -311,10 +360,12 @@ class Client(object):
             else:
                 req_url = url
             return self.do_api_query(request, API_VERSTRING + req_url, method=method, **query_kwargs)
-        call.func_name = name
+        call.__name__ = name
         setattr(cls, name, call)
 
-    def call_on_each_event(self, callback, event_types=None, narrow=[]):
+    def call_on_each_event(self, callback, event_types=None, narrow=None):
+        if narrow is None:
+            narrow = []
         def do_register():
             while True:
                 if event_types is None:
@@ -324,7 +375,7 @@ class Client(object):
 
                 if 'error' in res.get('result'):
                     if self.verbose:
-                        print "Server returned error:\n%s" % res['msg']
+                        print("Server returned error:\n%s" % res['msg'])
                     time.sleep(1)
                 else:
                     return (res['queue_id'], res['last_event_id'])
@@ -338,13 +389,13 @@ class Client(object):
             if 'error' in res.get('result'):
                 if res["result"] == "http-error":
                     if self.verbose:
-                        print "HTTP error fetching events -- probably a server restart"
+                        print("HTTP error fetching events -- probably a server restart")
                 elif res["result"] == "connection-error":
                     if self.verbose:
-                        print "Connection error fetching events -- probably server is temporarily down?"
+                        print("Connection error fetching events -- probably server is temporarily down?")
                 else:
                     if self.verbose:
-                        print "Server returned error:\n%s" % res["msg"]
+                        print("Server returned error:\n%s" % res["msg"])
                     if res["msg"].startswith("Bad event queue id:"):
                         # Our event queue went away, probably because
                         # we were asleep or the server restarted
@@ -382,9 +433,11 @@ def _mk_rm_subs(streams):
 def _mk_deregister(queue_id):
     return {'queue_id': queue_id}
 
-def _mk_events(event_types=None, narrow=[]):
+def _mk_events(event_types=None, narrow=None):
     if event_types is None:
         return dict()
+    if narrow is None:
+        narrow = []
     return dict(event_types=event_types, narrow=narrow)
 
 def _kwargs_to_dict(**kwargs):
@@ -425,7 +478,7 @@ Client._register('list_subscriptions', method='GET', url='users/me/subscriptions
 Client._register('add_subscriptions', url='users/me/subscriptions', make_request=_mk_subs)
 Client._register('remove_subscriptions', method='PATCH', url='users/me/subscriptions', make_request=_mk_rm_subs)
 Client._register('get_subscribers', method='GET',
-                 computed_url=lambda request: 'streams/%s/members' % (urllib.quote(request['stream'], safe=''),),
+                 computed_url=lambda request: 'streams/%s/members' % (urllib.parse.quote(request['stream'], safe=''),),
                  make_request=_kwargs_to_dict)
 Client._register('render_message', method='GET', url='messages/render')
 Client._register('create_user', method='POST', url='users')

assets/favicon/generate

@@ -1,6 +1,8 @@
 #!/usr/bin/env python
+from __future__ import absolute_import
 import xml.etree.ElementTree as ET
 import subprocess
+from six.moves import range
 
 # Generates the favicon images containing unread message counts.
 
@@ -10,7 +12,7 @@ elems = [tree.getroot().findall(
     ".//*[@id='%s']/{http://www.w3.org/2000/svg}tspan" % (name,))[0]
      for name in ('number_back', 'number_front')]
 
-for i in xrange(1,100):
+for i in range(1, 100):
     # Prepare a modified SVG
     s = '%2d' % (i,)
     for e in elems:

bots/check-mirroring

@@ -1,4 +1,6 @@
 #!/usr/bin/env python
+from __future__ import print_function
+from __future__ import absolute_import
 import sys
 import time
 import optparse
@@ -7,6 +9,7 @@ import random
 import logging
 import subprocess
 import hashlib
+from six.moves import range
 
 parser = optparse.OptionParser()
 parser.add_option('--verbose',
@@ -100,7 +103,7 @@ def print_status_and_exit(status):
     # e.g. true success and punting due to a SERVNAK, result in a
     # non-alert case, so to give us something unambiguous to check in
     # Nagios, print the exit status.
-    print status
+    print(status)
     sys.exit(status)
 
 def send_zulip(message):
@@ -149,7 +152,7 @@ for (stream, test) in test_streams:
         zephyr_subs_to_add.append((stream, '*', '*'))
 
 actually_subscribed = False
-for tries in xrange(10):
+for tries in range(10):
     try:
         zephyr.init()
         zephyr._z.subAll(zephyr_subs_to_add)
@@ -163,7 +166,7 @@ for tries in xrange(10):
         if missing == 0:
             actually_subscribed = True
             break
-    except IOError, e:
+    except IOError as e:
         if "SERVNAK received" in e:
             logger.error("SERVNAK repeatedly received, punting rest of test")
         else:
@@ -276,7 +279,7 @@ logger.info("Finished receiving Zulip messages!")
 receive_zephyrs()
 logger.info("Finished receiving Zephyr messages!")
 
-all_keys = set(zhkeys.keys() + hzkeys.keys())
+all_keys = set(list(zhkeys.keys()) + list(hzkeys.keys()))
 def process_keys(content_list):
     # Start by filtering out any keys that might have come from
     # concurrent check-mirroring processes

bots/gcal-bot

@@ -1,9 +1,10 @@
 #!/usr/bin/env python
+from __future__ import print_function
 import sys
 import time
 import datetime
 import optparse
-import urlparse
+from six.moves import urllib
 import itertools
 import traceback
 import os
@@ -56,13 +57,13 @@ except ImportError:
     parser.error('Install python-gdata')
 
 def get_calendar_url():
-    parts = urlparse.urlparse(options.calendar)
+    parts = urllib.parse.urlparse(options.calendar)
     pat = os.path.split(parts.path)
     if pat[1] != 'basic':
         parser.error('The --calendar URL should be the XML "Private Address" ' +
                      'from your calendar settings')
-    return urlparse.urlunparse((parts.scheme, parts.netloc, pat[0] + '/full',
-                                '', 'futureevents=true&orderby=startdate', ''))
+    return urllib.parse.urlunparse((parts.scheme, parts.netloc, pat[0] + '/full',
+                                   '', 'futureevents=true&orderby=startdate', ''))
 
 calendar_url = get_calendar_url()
 
@@ -99,7 +100,7 @@ def send_reminders():
             key = (uid, start)
             if key not in sent:
                 line = '%s starts at %s' % (title, start.strftime('%H:%M'))
-                print 'Sending reminder:', line
+                print('Sending reminder:', line)
                 messages.append(line)
                 keys.add(key)
 

bots/irc-mirror.py

@@ -1,4 +1,4 @@
-#! /usr/bin/env python
+#!/usr/bin/env python
 #
 # EXPERIMENTAL
 # IRC <=> Zulip mirroring bot
@@ -6,6 +6,7 @@
 # Setup: First, you need to install python-irc version 8.5.3
 # (https://bitbucket.org/jaraco/irc)
 
+from __future__ import print_function
 import irc.bot
 import irc.strings
 from irc.client import ip_numstr_to_quad, ip_quad_to_numstr
@@ -53,12 +54,12 @@ class IRCBot(irc.bot.SingleServerIRCBot):
             return
 
         # Forward the PM to Zulip
-        print zulip_client.send_message({
+        print(zulip_client.send_message({
                 "sender": sender,
                 "type": "private",
                 "to": "username@example.com",
                 "content": content,
-                })
+                }))
 
     def on_pubmsg(self, c, e):
         content = e.arguments[0]
@@ -68,14 +69,14 @@ class IRCBot(irc.bot.SingleServerIRCBot):
             return
 
         # Forward the stream message to Zulip
-        print zulip_client.send_message({
+        print(zulip_client.send_message({
                 "forged": "yes",
                 "sender": sender,
                 "type": "stream",
                 "to": stream,
                 "subject": "IRC",
                 "content": content,
-                })
+                }))
 
     def on_dccmsg(self, c, e):
         c.privmsg("You said: " + e.arguments[0])

bots/jabber_mirror.py

@@ -21,6 +21,7 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+from __future__ import print_function
 import sys
 import subprocess
 import os
@@ -39,7 +40,7 @@ args.extend(sys.argv[1:])
 
 backoff = RandomExponentialBackoff(timeout_success_equivalent=300)
 while backoff.keep_going():
-    print "Starting Jabber mirroring bot"
+    print("Starting Jabber mirroring bot")
     try:
         ret = subprocess.call(args)
     except:
@@ -51,9 +52,9 @@ while backoff.keep_going():
 
     backoff.fail()
 
-print ""
-print ""
-print "ERROR: The Jabber mirroring bot is unable to continue mirroring Jabber."
-print "Please contact zulip-devel@googlegroups.com if you need assistance."
-print ""
+print("")
+print("")
+print("ERROR: The Jabber mirroring bot is unable to continue mirroring Jabber.")
+print("Please contact zulip-devel@googlegroups.com if you need assistance.")
+print("")
 sys.exit(1)

bots/jabber_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright (C) 2013 Permabit, Inc.
 # Copyright (C) 2013--2014 Zulip, Inc.
@@ -37,6 +37,7 @@
 #               | other sender|  x  |    |        |
 # public mode   +-------------+-----+----+--------+----
 #               | self sender |     |    |        |
+from typing import Set
 
 import logging
 import threading
@@ -44,7 +45,7 @@ import optparse
 
 from sleekxmpp import ClientXMPP, InvalidJID, JID
 from sleekxmpp.exceptions import IqError, IqTimeout
-from ConfigParser import SafeConfigParser
+from six.moves.configparser import SafeConfigParser
 import os, sys, zulip, getpass
 import re
 
@@ -78,11 +79,11 @@ class JabberToZulipBot(ClientXMPP):
             self.nick = jid.username
             jid.resource = "zulip"
         ClientXMPP.__init__(self, jid, password)
-        self.rooms = set()
+        self.rooms = set() # type: Set[str]
         self.rooms_to_join = rooms
         self.add_event_handler("session_start", self.session_start)
         self.add_event_handler("message", self.message)
-        self.zulip = None
+        self.zulip = None # type: zulip.Client
         self.use_ipv6 = False
 
         self.register_plugin('xep_0045') # Jabber chatrooms
@@ -195,7 +196,7 @@ class JabberToZulipBot(ClientXMPP):
 class ZulipToJabberBot(object):
     def __init__(self, zulip_client):
         self.client = zulip_client
-        self.jabber = None
+        self.jabber = None # type: JabberToZulipBot
 
     def set_jabber_client(self, client):
         self.jabber = client
@@ -376,7 +377,7 @@ option does not affect login credentials.'''.replace("\n", " "))
 
     config = SafeConfigParser()
     try:
-        with file(config_file, 'r') as f:
+        with open(config_file, 'r') as f:
             config.readfp(f, config_file)
     except IOError:
         pass

bots/log2zulip

@@ -1,4 +1,5 @@
-#!/usr/bin/python
+#!/usr/bin/env python
+from __future__ import print_function
 import subprocess
 import os
 import sys
@@ -20,7 +21,7 @@ def mkdir_p(path):
     # Python doesn't have an analog to `mkdir -p` < Python 3.2.
     try:
         os.makedirs(path)
-    except OSError, e:
+    except OSError as e:
         if e.errno == errno.EEXIST and os.path.isdir(path):
             pass
         else:
@@ -55,14 +56,14 @@ def process_logs():
         data_file_path = "/var/tmp/log2zulip.state"
     mkdir_p(os.path.dirname(data_file_path))
     if not os.path.exists(data_file_path):
-        file(data_file_path, "w").write("{}")
-    last_data = ujson.loads(file(data_file_path).read())
+        open(data_file_path, "w").write("{}")
+    last_data = ujson.loads(open(data_file_path).read())
     new_data = {}
     for log_file in log_files:
         file_data = last_data.get(log_file, {})
         if not os.path.exists(log_file):
             # If the file doesn't exist, log an error and then move on to the next file
-            print "Log file %s does not exist!" % (log_file,)
+            print("Log file does not exist or could not stat log file: %s" % (log_file,))
             continue
         length = int(subprocess.check_output(["wc", "-l", log_file]).split()[0])
         if file_data.get("last") is None:
@@ -78,26 +79,26 @@ def process_logs():
             process_lines(new_lines, filename)
             file_data["last"] += len(new_lines)
         new_data[log_file] = file_data
-    file(data_file_path, "w").write(ujson.dumps(new_data))
+    open(data_file_path, "w").write(ujson.dumps(new_data))
 
 if __name__ == "__main__":
     if os.path.exists(lock_path):
-        print "Log2zulip lock held; not doing anything"
+        print("Log2zulip lock held; not doing anything")
         sys.exit(0)
 
     try:
-        file(lock_path, "w").write("1")
+        open(lock_path, "w").write("1")
         zulip_client = zulip.Client(config_file="/etc/log2zulip.zuliprc")
         try:
-            log_files = ujson.loads(file(control_path, "r").read())
+            log_files = ujson.loads(open(control_path, "r").read())
         except Exception:
-            print "Could not load control data from %s" % (control_path,)
+            print("Could not load control data from %s" % (control_path,))
             traceback.print_exc()
             sys.exit(1)
         process_logs()
     finally:
         try:
             os.remove(lock_path)
-        except OSError, IOError:
+        except OSError as IOError:
             pass
 

bots/process_ccache

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 import sys
 import subprocess
 import base64
@@ -9,27 +9,27 @@ ccache_data_encoded = sys.argv[3]
 
 # Update the Kerberos ticket cache file
 program_name = "zmirror-%s" % (short_user,)
-with file("/home/zulip/ccache/%s" % (program_name,), "w") as f:
+with open("/home/zulip/ccache/%s" % (program_name,), "w") as f:
     f.write(base64.b64decode(ccache_data_encoded))
 
 # Setup API key
 api_key_path = "/home/zulip/api-keys/%s" % (program_name,)
-file(api_key_path, "w").write(api_key + "\n")
+open(api_key_path, "w").write(api_key + "\n")
 
 # Setup supervisord configuration
 supervisor_path = "/etc/supervisor/conf.d/%s.conf" % (program_name,)
 template = "/home/zulip/zulip/bots/zmirror_private.conf.template"
-template_data = file(template).read()
+template_data = open(template).read()
 session_path = "/home/zulip/zephyr_sessions/%s" % (program_name,)
 
 # Preserve mail zephyrs forwarding setting across rewriting the config file
 
 try:
-    if "--forward-mail-zephyrs" in file(supervisor_path, "r").read():
+    if "--forward-mail-zephyrs" in open(supervisor_path, "r").read():
         template_data = template_data.replace("--use-sessions", "--use-sessions --forward-mail-zephyrs")
 except Exception:
     pass
-file(supervisor_path, "w").write(template_data.replace("USERNAME", short_user))
+open(supervisor_path, "w").write(template_data.replace("USERNAME", short_user))
 
 # Delete your session
 subprocess.check_call(["rm", "-f", session_path])

bots/summarize_stream.py

@@ -1,3 +1,5 @@
+from __future__ import print_function
+from typing import Any, Dict, List
 # This is hacky code to analyze data on our support stream.  The main
 # reusable bits are get_recent_messages and get_words.
 
@@ -31,7 +33,7 @@ def analyze_messages(msgs, word_count, email_count):
         if False:
             if ' ack' in msg['content']:
                 name = msg['sender_full_name'].split()[0]
-                print 'ACK', name
+                print('ACK', name)
         m = re.search('ticket (Z....).*email: (\S+).*~~~(.*)', msg['content'], re.M | re.S)
         if m:
             ticket, email, req = m.groups()
@@ -40,22 +42,22 @@ def analyze_messages(msgs, word_count, email_count):
                 word_count[word] += 1
             email_count[email] += 1
         if False:
-            print
+            print()
             for k, v in msg.items():
-                print '%-20s: %s' % (k, v)
+                print('%-20s: %s' % (k, v))
 
 def generate_support_stats():
     client = zulip.Client()
     narrow = 'stream:support'
     count = 2000
     msgs = get_recent_messages(client, narrow, count)
-    msgs_by_topic = collections.defaultdict(list)
+    msgs_by_topic = collections.defaultdict(list) # type: Dict[str, List[Dict[str, Any]]]
     for msg in msgs:
         topic = msg['subject']
         msgs_by_topic[topic].append(msg)
 
-    word_count = collections.defaultdict(int)
-    email_count = collections.defaultdict(int)
+    word_count = collections.defaultdict(int) # type: Dict[str, int]
+    email_count = collections.defaultdict(int) # type: Dict[str, int]
 
     if False:
         for topic in msgs_by_topic:
@@ -63,17 +65,15 @@ def generate_support_stats():
     analyze_messages(msgs, word_count, email_count)
 
     if True:
-        words = word_count.keys()
-        words = filter(lambda w: word_count[w] >= 10, words)
-        words = filter(lambda w: len(w) >= 5, words)
+        words = [w for w in word_count.keys() if word_count[w] >= 10 and len(w) >= 5]
         words = sorted(words, key=lambda w: word_count[w], reverse=True)
         for word in words:
-            print word, word_count[word]
+            print(word, word_count[word])
 
     if False:
-        emails = email_count.keys()
-        emails = sorted(emails, key=lambda w: email_count[w], reverse=True)
+        emails = sorted(list(email_count.keys()),
+                        key=lambda w: email_count[w], reverse=True)
         for email in emails:
-            print email, email_count[email]
+            print(email, email_count[email])
 
 generate_support_stats()

bots/sync-public-streams

@@ -59,7 +59,7 @@ if __name__ == "__main__":
         if public_streams is None:
             continue
 
-        f = file("/home/zulip/public_streams.tmp", "w")
+        f = open("/home/zulip/public_streams.tmp", "w")
         f.write(simplejson.dumps(list(public_streams)) + "\n")
         f.close()
 

bots/zephyr_mirror.py

@@ -21,13 +21,15 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+from __future__ import absolute_import
+from __future__ import print_function
 import sys
 import subprocess
 import os
 import traceback
 import signal
 
-from zephyr_mirror_backend import parse_args
+from .zephyr_mirror_backend import parse_args
 
 def die(signal, frame):
     # We actually want to exit, so run os._exit (so as not to be caught and restarted)
@@ -52,30 +54,30 @@ if options.forward_class_messages and not options.noshard:
     if options.on_startup_command is not None:
         subprocess.call([options.on_startup_command])
     from zerver.lib.parallel import run_parallel
-    print "Starting parallel zephyr class mirroring bot"
+    print("Starting parallel zephyr class mirroring bot")
     jobs = list("0123456789abcdef")
     def run_job(shard):
         subprocess.call(args + ["--shard=%s" % (shard,)])
         return 0
     for (status, job) in run_parallel(run_job, jobs, threads=16):
-        print "A mirroring shard died!"
+        print("A mirroring shard died!")
         pass
     sys.exit(0)
 
 backoff = RandomExponentialBackoff(timeout_success_equivalent=300)
 while backoff.keep_going():
-    print "Starting zephyr mirroring bot"
+    print("Starting zephyr mirroring bot")
     try:
         subprocess.call(args)
     except:
         traceback.print_exc()
     backoff.fail()
 
-print ""
-print ""
-print "ERROR: The Zephyr mirroring bot is unable to continue mirroring Zephyrs."
-print "This is often caused by failing to maintain unexpired Kerberos tickets"
-print "or AFS tokens.  See https://zulip.com/zephyr for documentation on how to"
-print "maintain unexpired Kerberos tickets and AFS tokens."
-print ""
+print("")
+print("")
+print("ERROR: The Zephyr mirroring bot is unable to continue mirroring Zephyrs.")
+print("This is often caused by failing to maintain unexpired Kerberos tickets")
+print("or AFS tokens.  See https://zulip.com/zephyr for documentation on how to")
+print("maintain unexpired Kerberos tickets and AFS tokens.")
+print("")
 sys.exit(1)

bots/zephyr_mirror_backend.py

@@ -20,12 +20,16 @@
 # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
+from __future__ import absolute_import
+from typing import Any, List
 
 import sys
+from six.moves import map
+from six.moves import range
 try:
     import simplejson
 except ImportError:
-    import json as simplejson
+    import json as simplejson # type: ignore
 import re
 import time
 import subprocess
@@ -41,10 +45,12 @@ import select
 
 DEFAULT_SITE = "https://api.zulip.com"
 
-class States:
-    Startup, ZulipToZephyr, ZephyrToZulip, ChildSending = range(4)
+class States(object):
+    Startup, ZulipToZephyr, ZephyrToZulip, ChildSending = list(range(4))
 CURRENT_STATE = States.Startup
 
+logger = None # type: logging.Logger
+
 def to_zulip_username(zephyr_username):
     if "@" in zephyr_username:
         (user, realm) = zephyr_username.split("@")
@@ -188,7 +194,7 @@ def zephyr_bulk_subscribe(subs):
 
 def update_subscriptions():
     try:
-        f = file(options.stream_file_path, "r")
+        f = open(options.stream_file_path, "r")
         public_streams = simplejson.loads(f.read())
         f.close()
     except:
@@ -284,7 +290,7 @@ def parse_zephyr_body(zephyr_data):
 
 def parse_crypt_table(zephyr_class, instance):
     try:
-        crypt_table = file(os.path.join(os.environ["HOME"], ".crypt-table"))
+        crypt_table = open(os.path.join(os.environ["HOME"], ".crypt-table"))
     except IOError:
         return None
 
@@ -346,7 +352,7 @@ def process_notice(notice, log):
 
     if zephyr_class == options.nagios_class:
         # Mark that we got the message and proceed
-        with file(options.nagios_path, "w") as f:
+        with open(options.nagios_path, "w") as f:
             f.write("0\n")
         return
 
@@ -465,7 +471,7 @@ def zephyr_load_session_autoretry(session_path):
     backoff = zulip.RandomExponentialBackoff()
     while backoff.keep_going():
         try:
-            session = file(session_path, "r").read()
+            session = open(session_path, "r").read()
             zephyr._z.initialize()
             zephyr._z.load_session(session)
             zephyr.__inited = True
@@ -507,7 +513,7 @@ def zephyr_to_zulip(options):
         if options.nagios_class:
             zephyr_subscribe_autoretry((options.nagios_class, "*", "*"))
         if options.use_sessions:
-            file(options.session_path, "w").write(zephyr._z.dump_session())
+            open(options.session_path, "w").write(zephyr._z.dump_session())
 
     if options.logs_to_resend is not None:
         with open(options.logs_to_resend, 'r') as log:
@@ -801,9 +807,9 @@ def add_zulip_subscriptions(verbose):
         unauthorized = res.get("unauthorized")
         if verbose:
             if already is not None and len(already) > 0:
-                logger.info("\nAlready subscribed to: %s" % (", ".join(already.values()[0]),))
+                logger.info("\nAlready subscribed to: %s" % (", ".join(list(already.values())[0]),))
             if new is not None and len(new) > 0:
-                logger.info("\nSuccessfully subscribed to: %s" % (", ".join(new.values()[0]),))
+                logger.info("\nSuccessfully subscribed to: %s" % (", ".join(list(new.values())[0]),))
             if unauthorized is not None and len(unauthorized) > 0:
                 logger.info("\n" + "\n".join(textwrap.wrap("""\
 The following streams you have NOT been subscribed to,
@@ -854,7 +860,7 @@ def parse_zephyr_subs(verbose=False):
             logger.error("Couldn't find ~/.zephyr.subs!")
         return []
 
-    for line in file(subs_file, "r").readlines():
+    for line in open(subs_file, "r").readlines():
         line = line.strip()
         if len(line) == 0:
             continue
@@ -875,6 +881,7 @@ def parse_zephyr_subs(verbose=False):
     return zephyr_subscriptions
 
 def open_logger():
+    # type: () -> logging.Logger
     if options.log_path is not None:
         log_file = options.log_path
     elif options.forward_class_messages:
@@ -1022,7 +1029,7 @@ if __name__ == "__main__":
 
     signal.signal(signal.SIGINT, die_gracefully)
 
-    (options, args) = parse_args()
+    (options, args) = parse_args() # type: Any, List[str]
 
     logger = open_logger()
     configure_logger(logger, "parent")
@@ -1047,7 +1054,7 @@ Could not find API key file.
 You need to either place your api key file at %s,
 or specify the --api-key-file option.""" % (options.api_key_file,))))
             sys.exit(1)
-        api_key = file(options.api_key_file).read().strip()
+        api_key = open(options.api_key_file).read().strip()
         # Store the API key in the environment so that our children
         # don't need to read it in
         os.environ["HUMBUG_API_KEY"] = api_key
@@ -1110,7 +1117,7 @@ or specify the --api-key-file option.""" % (options.api_key_file,))))
         options.session_path = "/var/tmp/%s" % (options.user,)
 
     if options.forward_from_zulip:
-        child_pid = os.fork()
+        child_pid = os.fork() # type: int
         if child_pid == 0:
             CURRENT_STATE = States.ZulipToZephyr
             # Run the zulip => zephyr mirror in the child

confirmation/__init__.py

@@ -2,23 +2,23 @@
 
 # Copyright: (c) 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 
-# Permission is hereby granted, free of charge, to any person obtaining a                                                       
-# copy of this software and associated documentation files (the                                                                 
-# "Software"), to deal in the Software without restriction, including                                                           
-# without limitation the rights to use, copy, modify, merge, publish, dis-                                                      
-# tribute, sublicense, and/or sell copies of the Software, and to permit                                                        
-# persons to whom the Software is furnished to do so, subject to the fol-                                                       
-# lowing conditions:                                                                                                            
-#                                                                                                                               
-# The above copyright notice and this permission notice shall be included                                                       
-# in all copies or substantial portions of the Software.                                                                        
-#                                                                                                                               
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS                                                       
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-                                                      
-# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT                                                        
-# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,                                                         
-# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,                                                            
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS                                                        
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 
 VERSION = (0, 9, 'pre')

confirmation/management/commands/cleanupconfirmation.py

@@ -4,6 +4,7 @@
 
 __revision__ = '$Id: cleanupconfirmation.py 5 2008-11-18 09:10:12Z jarek.zgoda $'
 
+from typing import Any
 
 from django.core.management.base import NoArgsCommand
 
@@ -14,4 +15,5 @@ class Command(NoArgsCommand):
     help = 'Delete expired confirmations from database'
 
     def handle_noargs(self, **options):
+        # type: (**Any) -> None
         Confirmation.objects.delete_expired_confirmations()

confirmation/models.py

@@ -105,9 +105,9 @@ class Confirmation(models.Model):
 
     objects = ConfirmationManager()
 
-    class Meta:
+    class Meta(object):
         verbose_name = _('confirmation email')
         verbose_name_plural = _('confirmation emails')
 
     def __unicode__(self):
-        return _('confirmation email for %s') % self.content_object
+        return _('confirmation email for %s') % (self.content_object,)

confirmation/settings.py

@@ -2,9 +2,10 @@
 
 # Copyright: (c) 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 
+from typing import Any, Dict
+
 __revision__ = '$Id: settings.py 12 2008-11-23 19:38:52Z jarek.zgoda $'
 
 STATUS_ACTIVE = 1
 
-STATUS_FIELDS = {
-}
+STATUS_FIELDS = {} # type: Dict[Any, Any]

confirmation/util.py

@@ -9,4 +9,4 @@ from django.conf import settings
 def get_status_field(app_label, model_name):
     model = '%s.%s' % (app_label, model_name)
     mapping = getattr(settings, 'STATUS_FIELDS', {})
-    return mapping.get(model, 'status')
+    return mapping.get(model, 'status')

confirmation/views.py

@@ -8,11 +8,14 @@ __revision__ = '$Id: views.py 21 2008-12-05 09:21:03Z jarek.zgoda $'
 from django.shortcuts import render_to_response
 from django.template import RequestContext
 from django.conf import settings
+from django.http import HttpRequest, HttpResponse
 
 from confirmation.models import Confirmation
+from zproject.jinja2 import render_to_response
 
 
 def confirm(request, confirmation_key):
+    # type: (HttpRequest, str) -> HttpResponse
     confirmation_key = confirmation_key.lower()
     obj = Confirmation.objects.confirm(confirmation_key)
     confirmed = True
@@ -38,6 +41,5 @@ def confirm(request, confirmation_key):
     ]
     if obj:
         # if we have an object, we can use specific template
-        templates.insert(0, 'confirmation/confirm_%s.html' % obj._meta.model_name)
-    return render_to_response(templates, ctx,
-        context_instance=RequestContext(request))
+        templates.insert(0, 'confirmation/confirm_%s.html' % (obj._meta.model_name,))
+    return render_to_response(templates, ctx, request=request)

corporate/urls.py

@@ -1,9 +1,16 @@
 from django.conf.urls import patterns, url
 from django.views.generic import TemplateView, RedirectView
 
-urlpatterns = patterns('',
+i18n_urlpatterns = [
     # Zephyr/MIT
     url(r'^zephyr/$', TemplateView.as_view(template_name='corporate/zephyr.html')),
     url(r'^mit/$', TemplateView.as_view(template_name='corporate/mit.html')),
     url(r'^zephyr-mirror/$', TemplateView.as_view(template_name='corporate/zephyr-mirror.html')),
-)
+
+    # Terms of service and privacy policy
+    url(r'^terms/$',   TemplateView.as_view(template_name='corporate/terms.html')),
+    url(r'^terms-enterprise/$',  TemplateView.as_view(template_name='corporate/terms-enterprise.html')),
+    url(r'^privacy/$', TemplateView.as_view(template_name='corporate/privacy.html')),
+]
+
+urlpatterns = patterns('', *i18n_urlpatterns)

docs/README.md

@@ -1,23 +1,23 @@
-These docs are written in rST, and are included on the zulip.org website
-as well as on each development installation.  Many of these docs
-have been ported from the internal docs of Zulip Inc.,
-and may need to be updated for use in the open source project.
+These docs are written in [Commonmark
+Markdown](http://commonmark.org/) with a small bit of rST.  We've
+chosen Markdown because it is [easy to
+write](http://commonmark.org/help).  The docs are served in production
+at [zulip.readthedocs.io](https://zulip.readthedocs.io/en/latest/).
 
-To generate HTML docs locally from rST:
+If you want to build the documentation locally (e.g. to test your
+changes), the dependencies are automatically installed as part of
+Zulip development environment provisioning, and you can build the
+documentation using:
 
-   * `pip install sphinx`
-   * In this directory, `make html`. Output appears in a `_build/html` subdirectory.
+```
+cd docs/
+make html
+```
 
-To create rST from MediaWiki input:
-
-   * Use `pandoc -r mediawiki -w rst` on MediaWiki source.
-   * Use unescape.py to remove any leftover HTML entities (often inside <pre>
-     tags and the like).
-
-We can use pandoc to translate mediawiki into reStructuredText, but some things need fixing up:
-
-   * Add page titles.
-   * Review pages for formatting (especially inline code chunks) and content.
-   * Fix wiki links?
-   * Add pages to the table of contents (`index.rst`).
+You can also usually test your changes by pushing a branch to GitHub
+and looking at the content on the GitHub web UI, since GitHub renders
+Markdown.
 
+When editing dependencies for the Zulip documentation, you should edit
+`requirements/docs.txt` (which is used by ReadTheDocs to build the
+documentation quickly, without installing all of Zulip's dependencies).

docs/architecture-overview.md

@@ -0,0 +1,234 @@
+Zulip architectural overview
+============================
+
+Key Codebases
+-------------
+
+The core Zulip application is at
+[<https://github.com/zulip/zulip>](https://github.com/zulip/zulip) and
+is a web application written in Python 2.7 (soon to also support
+Python 3) and using the Django framework. That codebase includes
+server-side code and the web client, as well as Python API bindings
+and most of our integrations with other services and applications (see
+[the directory structure
+guide](https://zulip.readthedocs.io/en/latest/directory-structure.html)).
+
+We maintain several separate repositories for integrations and other
+glue code: a [Hubot adapter](https://github.com/zulip/hubot-zulip);
+integrations with
+[Phabricator](https://github.com/zulip/phabricator-to-zulip),
+[Jenkins](https://github.com/zulip/zulip-jenkins-plugin),
+[Puppet](https://github.com/matthewbarr/puppet-zulip),
+[Redmine](https://github.com/zulip/zulip-redmine-plugin), and
+[Trello](https://github.com/zulip/trello-to-zulip); [node.js API
+bindings](https://github.com/zulip/zulip-node); and our [full-text
+search PostgreSQL extension](https://github.com/zulip/tsearch_extras) .
+
+Our mobile clients are separate code repositories:
+[Android](https://github.com/zulip/zulip-android), [iOS
+(stable)](https://github.com/zulip/zulip-ios) , and [our experimental
+React Native iOS app](https://github.com/zulip/zulip-mobile). Our
+[desktop application](https://github.com/zulip/zulip-desktop) is also a
+separate repository.
+
+We use [Transifex](https://www.transifex.com/zulip/zulip/) to do
+translations.
+
+In this overview we'll mainly discuss the core Zulip server and web
+application.
+
+Usage assumptions and concepts
+------------------------------
+
+Zulip is a real-time web-based chat application meant for companies and
+similar groups ranging in size from a small team to more than a thousand
+users. It features real-time notifications, message persistence and
+search, public group conversations (*streams*), invite-only streams,
+private one-on-one and group conversations, inline image previews, team
+presence/a buddy list, a rich API, Markdown message support, and several
+integrations with other services. The maintainer team aims to support
+users who connect to Zulip using dedicated iOS, Android, Linux, Windows,
+and Mac OS X clients, as well as people using modern web browsers or
+dedicated Zulip API clients.
+
+A server can host multiple Zulip *realms* (organizations) at the same
+domain, each of which is a private chamber with its own users, streams,
+customizations, and so on. This means that one person might be a user of
+multiple Zulip realms. The administrators of a realm can choose whether
+to allow anyone to register an account and join, or only allow people
+who have been invited, or restrict registrations to members of
+particular groups (using email domain names or corporate single-sign-on
+login for verification). For more on scalability and security
+considerations, see [Zulip in
+production](https://github.com/zulip/zulip/blob/master/README.prod.md).
+
+The default Zulip home screen is like a chronologically ordered inbox;
+it displays messages, starting at the oldest message that the user
+hasn't viewed yet. The home screen displays the most recent messages in
+all the streams a user has joined (except for the streams they've
+muted), as well as private messages from other users, in strict
+chronological order. A user can *narrow* to view only the messages in a
+single stream, and can further narrow to focus on a *topic* (thread)
+within that stream. Each narrow has its own URL.
+
+Zulip's philosophy is to provide sensible defaults but give the user
+fine-grained control over their incoming information flow; a user can
+mute topics and streams, and can make fine-grained choices to reduce
+real-time notifications they find irrelevant.
+
+Components
+----------
+
+### Tornado and Django
+
+We use both the [Tornado](http://www.tornadoweb.org) and
+[Django](https://www.djangoproject.com/) Python web frameworks.
+
+Django is the main web application server; Tornado runs the
+server-to-client real-time push system. The app servers are configured
+by the Supervisor configuration (which explains how to start the server
+processes; see "Supervisor" below) and the nginx configuration (which
+explains which HTTP requests get sent to which app server).
+
+Tornado is an asynchronous server and is meant specifically to hold open
+tens of thousands of long-lived (long-polling or websocket) connections
+-- that is to say, routes that maintain a persistent connection from
+every running client. For this reason, it's responsible for event
+(message) delivery, but not much else. We try to avoid any blocking
+calls in Tornado because we don't want to delay delivery to thousands of
+other connections (as this would make Zulip very much not real-time).
+For instance, we avoid doing cache or database queries inside the
+Tornado code paths, since those blocking requests carry a very high
+performance penalty for a single-threaded, asynchronous server.
+
+The parts that are activated relatively rarely (e.g. when people type or
+click on something) are processed by the Django application server. One
+exception to this is that Zulip uses websockets through Tornado to
+minimize latency on the code path for **sending** messages.
+
+### nginx
+
+nginx is the front-end web server to all Zulip traffic; it serves static
+assets and proxies to Django and Tornado. It handles HTTP requests
+according to the rules laid down in the many config files found in
+`zulip/puppet/zulip/files/nginx/`.
+
+`zulip/puppet/zulip/files/nginx/zulip-include-frontend/app` is the most
+important of these files. It explains what happens when requests come in
+from outside.
+
+-   In production, all requests to URLs beginning with `/static/` are
+    served from the corresponding files in `/home/zulip/prod-static/`,
+    and the production build process (`tools/build-release-tarball`)
+    compiles, minifies, and installs the static assets into the
+    `prod-static/` tree form. In development, files are served directly
+    from `/static/` in the git repository.
+-   Requests to `/json/get_events`, `/api/v1/events`, and `/sockjs` are
+    sent to the Tornado server. These are requests to the real-time push
+    system, because the user's web browser sets up a long-lived TCP
+    connection with Tornado to serve as [a channel for push
+    notifications](https://en.wikipedia.org/wiki/Push_technology#Long_Polling).
+    nginx gets the hostname for the Tornado server via
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`.
+-   Requests to all other paths are sent to the Django app via the UNIX
+    socket `unix:/home/zulip/deployments/fastcgi-socket` (defined in
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`). We use
+    `zproject/wsgi.py` to implement FastCGI here (see
+    `django.core.wsgi`).
+
+### Supervisor
+
+We use [supervisord](http://supervisord.org/) to start server processes,
+restart them automatically if they crash, and direct logging.
+
+The config file is
+`zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf`. This is where
+Tornado and Django are set up, as well as a number of background
+processes that process event queues. We use event queues for the kinds
+of tasks that are best run in the background because they are
+expensive (in terms of performance) and don't have to be synchronous
+-- e.g., sending emails or updating analytics. Also see [the queuing
+guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+
+### memcached
+
+memcached is used to cache database model objects. `zerver/lib/cache.py`
+and `zerver/lib/cache_helpers.py` manage putting things into memcached,
+and invalidating the cache when values change. The memcached
+configuration is in `puppet/zulip/files/memcached.conf`.
+
+### Redis
+
+Redis is used for a few very short-term data stores, such as in the
+basis of `zerver/lib/rate_limiter.py`, a per-user rate limiting scheme
+[example](http://blog.domaintools.com/2013/04/rate-limiting-with-redis/)),
+and the [email-to-Zulip
+integration](https://zulip.com/integrations/#email).
+
+Redis is configured in `zulip/puppet/zulip/files/redis` and it's a
+pretty standard configuration except for the last line, which turns off
+persistence:
+
+    # Zulip-specific configuration: disable saving to disk.
+    save ""
+
+memcached was used first and then we added Redis specifically to
+implement rate limiting. [We're discussing switching everything over to
+Redis.](https://github.com/zulip/zulip/issues/16)
+
+### RabbitMQ
+
+RabbitMQ is a queueing system. Its config files live in
+`zulip/puppet/zulip/files/rabbitmq`. Initial configuration happens in
+`zulip/scripts/setup/configure-rabbitmq`.
+
+We use RabbitMQ for queuing expensive work (e.g. sending emails
+triggered by a message, push notifications, some analytics, etc.) that
+require reliable delivery but which we don't want to do on the main
+thread. It's also used for communication between the application server
+and the Tornado push system.
+
+Two simple wrappers around `pika` (the Python RabbitMQ client) are in
+`zulip/server/lib/queue.py`. There's an asynchronous client for use in
+Tornado and a more general client for use elsewhere.
+
+`zerver/lib/event_queue.py` has helper functions for putting events into
+one queue or another. Most of the processes started by Supervisor are
+queue processors that continually pull things out of a RabbitMQ queue
+and handle them.
+
+Also see [the queuing
+guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+
+### PostgreSQL
+
+PostgreSQL (also known as Postgres) is the database that stores all
+persistent data, that is, data that's expected to live beyond a user's
+current session.
+
+In production, Postgres is installed with a default configuration. The
+directory that would contain configuration files
+(`puppet/zulip/files/postgresql`) has only a utility script and a custom
+list of stopwords used by a Postgresql extension.
+
+In a development environment, configuration of that postgresql extension
+is handled by `tools/postgres-init-dev-db` (invoked by `provision.py`).
+That file also manages setting up the development postgresql user.
+
+`provision.py` also invokes `tools/do-destroy-rebuild-database` to
+create the actual database with its schema.
+
+### Nagios
+
+Nagios is an optional component used for notifications to the system
+administrator, e.g., in case of outages.
+
+`zulip/puppet/zulip/manifests/nagios.pp` installs Nagios plugins from
+puppet/`zulip/files/nagios_plugins/`.
+
+This component is intended to install Nagios plugins intended to be run
+on a Nagios server; most of the Zulip Nagios plugins are intended to be
+run on the Zulip servers themselves, and are included with the relevant
+component of the Zulip server (e.g.
+`puppet/zulip/manifests/postgres_common.pp` installs a few under
+`/usr/lib/nagios/plugins/zulip_postgres_common`).

docs/changelog.md

@@ -0,0 +1,125 @@
+# Version History
+
+All notable changes to the Zulip server are documented in this file.
+
+### Unreleased
+
+### 1.3.13 - 2016-06-21
+- Added nearly complete internationalization of the Zulip UI.
+- Added warning when using @all/@everyone.
+- Added button offering to subscribe at bottom of narrows to streams
+  the user is not subscribed to.
+- Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT,
+  Transifex, and Updown.io.
+- Added menu option to mark all messages in a stream or topic as read.
+- Added new Attachment model to keep track of uploaded files.
+- Added caching of virtualenvs in development.
+- Added mypy static type annotations to about 85% of the Zulip Python codebase.
+- Added automated test of backend templates to test for regressions.
+- Added lots of detailed documentation on the Zulip development environment.
+- Added setting allowing only administrators to create new streams.
+- Added button to exit the Zulip tutorial early.
+- Added web UI for configuring default streams.
+- Added new OPEN_REALM_CREATION setting (default off), providing a UI
+  for creating additional realms on a Zulip server.
+- Fixed email_gateway_password secret not working properly.
+- Fixed missing helper scripts for RabbitMQ Nagios plugins.
+- Fixed skipping forward to latest messages ("More messages below" button).
+- Fixed netcat issue causing Zulip installation to hang on Scaleway machines.
+- Fixed rendering of /me status messages after message editing.
+- Fixed case sensitivity of right sidebar fading when compose is open.
+- Fixed error messages when composing to invalid PM recipients.
+- Fixed LDAP auth backend not working with Zulip mobile apps.
+- Fixed erroneous WWW-Authenticate headers with expired sessions.
+- Changed "coworkers" to "users" in the Zulip UI.
+- Changed add_default_stream REST API to correctly use PUT rather than PATCH.
+- Updated the Zulip emoji set (the Android Emoji) to a modern version.
+- Made numerous small improvements to the Zulip development experience.
+- Migrated backend templates to the faster Jinja2 templating system.
+- Migrated development environment setup scripts to tools/setup/.
+- Expanded test coverage for several areas of the product.
+- Simplified the API for writing new webhook integrations.
+- Removed most of the remaining javascript global variables.
+
+### 1.3.12 - 2016-05-10
+- CVE-2016-4426: Bot API keys were accessible to other users in the same realm.
+- CVE-2016-4427: Deactivated users could access messages if SSO was enabled.
+- Fixed a RabbitMQ configuration bug that resulted in reordered messages.
+- Added expansive test suite for authentication backends and decorators.
+- Added an option to logout_all_users to delete only sessions for deactivated users.
+
+### 1.3.11 - 2016-05-02
+- Moved email digest support into the default Zulip production configuration.
+- Added options for configuring Postgres, RabbitMQ, Redis, and memcached
+  in settings.py.
+- Added documentation on using Hubot to integrate with useful services
+  not yet integrated with Zulip directly (e.g. Google Hangouts).
+- Added new management command to test sending email from Zulip.
+- Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.
+- Added Nagios plugins to the main distribution.
+- Added ability for realm administrators to manage custom emoji.
+- Added guide to writing new integrations.
+- Enabled camo image proxy to fix mixed-content warnings for http images.
+- Refactored the Zulip puppet modules to be more modular.
+- Refactored the Tornado event system, fixing old memory leaks.
+- Removed many old-style /json API endpoints
+- Implemented running queue processors multithreaded in development,
+  decreasing RAM requirements for a Zulip development environment from
+  ~1GB to ~300MB.
+- Fixed rerendering the complete buddy list whenever a user came back from
+  idle, which was a significant performance issue in larger realms.
+- Fixed the disabling of desktop notifications from 1.3.7 for new users.
+- Fixed the (admin) create_user API enforcing restricted_to_domain, even
+  if that setting was disabled for the realm.
+- Fixed bugs changing certain settings in administration pages.
+- Fixed collapsing messages in narrowed views.
+- Fixed 500 errors when uploading a non-image file as an avatar.
+- Fixed Jira integration incorrectly not @-mentioning assignee.
+
+### 1.3.10 - 2016-01-21
+- Added new integration for Travis CI.
+- Added settings option to control maximum file upload size.
+- Added support for running Zulip development environment in Docker.
+- Added easy configuration support for a remote postgres database.
+- Added extensive documentation on scalability, backups, and security.
+- Recent private message threads are now displayed expanded similar to
+  the pre-existing recent topics feature.
+- Made it possible to set LDAP and EMAIL_HOST passwords in
+  /etc/zulip/secrets.conf.
+- Improved the styling for the Administration page and added tabs.
+- Substantially improved loading performance on slow networks by enabling
+  GZIP compression on more assets.
+- Changed the page title in narrowed views to include the current narrow.
+- Fixed several backend performance issues affecting very large realms.
+- Fixed bugs where draft compose content might be lost when reloading site.
+- Fixed support for disabling the "zulip" notifications stream.
+- Fixed missing step in postfix_localmail installation instructions.
+- Fixed several bugs/inconveniences in the production upgrade process.
+- Fixed realm restrictions for servers with a unique, open realm.
+- Substantially cleaned up console logging from run-dev.py.
+
+### 1.3.9 - 2015-11-16
+- Fixed buggy #! lines in upgrade scripts.
+
+### 1.3.8 - 2015-11-15
+- Added options to the Python api for working with untrusted server certificates.
+- Added a lot of documentation on the development environment and testing.
+- Added partial support for translating the Zulip UI.
+- Migrated installing Node dependencies to use npm.
+- Fixed LDAP integration breaking autocomplete of @-mentions.
+- Fixed admin panel reactivation/deactivation of bots.
+- Fixed inaccurate documentation for downloading the desktop apps.
+- Fixed various minor bugs in production installation process.
+- Fixed security issue where recent history on private streams might
+  be visible to new users (to the Zulip team) who were invited with that
+  private stream as one of their initial streams
+  (https://github.com/zulip/zulip/issues/230).
+- Major preliminary progress towards supporting Python 3.
+
+### 1.3.7 - 2015-10-19
+- Turn off desktop and audible notifications for streams by default.
+- Added support for the LDAP authentication integration creating new users.
+- Added new endpoint to support Google auth on mobile.
+- Fixed desktop notifications in modern Firefox.
+- Fixed several installation issues for both production and development environments.
+- Improved documentation for outgoing SMTP and the email mirror integration.

docs/code-contribution-checklist.rst

@@ -0,0 +1,24 @@
+=======================
+Code contribution guide
+=======================
+
+Thanks for contributing to Zulip! We hope this guide helps make the
+process smooth sailing for you.
+
+The "Ways to Contribute" and "How to get involved" sections of
+:doc:`readme-symlink` are a good place to start! After that, check out
+the tips below.
+
+
+.. toctree::
+   :maxdepth: 2
+
+   code-style
+   markdown
+   testing
+   logging
+   mypy
+   settings
+   front-end-build-process
+   queuing
+   schema-migrations

docs/code-style.md

@@ -0,0 +1,482 @@
+Code style and conventions
+==========================
+
+Be consistent!
+--------------
+
+Look at the surrounding code, or a similar part of the project, and try
+to do the same thing. If you think the other code has actively bad
+style, fix it (in a separate commit).
+
+When in doubt, send an email to <zulip-devel@googlegroups.com> with your
+question.
+
+Lint tools
+----------
+
+You can run them all at once with
+
+    ./tools/lint-all
+
+You can set this up as a local Git commit hook with
+
+    ``tools/setup-git-repo``
+
+The Vagrant setup process runs this for you.
+
+`lint-all` runs many lint checks in parallel, including
+
+-   Javascript ([JSLint](http://www.jslint.com/))
+
+    > `tools/jslint/check-all.js` contains a pretty fine-grained set of
+    > JSLint options, rule exceptions, and allowed global variables. If
+    > you add a new global, you'll need to add it to the list.
+
+-   Python ([Pyflakes](http://pypi.python.org/pypi/pyflakes))
+-   templates
+-   Puppet configuration
+-   custom checks (e.g. trailing whitespace and spaces-not-tabs)
+
+Secrets
+-------
+
+Please don't put any passwords, secret access keys, etc. inline in the
+code. Instead, use the `get_secret` function in `zproject/settings.py`
+to read secrets from `/etc/zulip/secrets.conf`.
+
+Dangerous constructs
+--------------------
+
+### Misuse of database queries
+
+Look out for Django code like this:
+
+    [Foo.objects.get(id=bar.x.id)
+    for bar in Bar.objects.filter(...)
+    if  bar.baz < 7]
+
+This will make one database query for each `Bar`, which is slow in
+production (but not in local testing!). Instead of a list comprehension,
+write a single query using Django's [QuerySet
+API](https://docs.djangoproject.com/en/dev/ref/models/querysets/).
+
+If you can't rewrite it as a single query, that's a sign that something
+is wrong with the database schema. So don't defer this optimization when
+performing schema changes, or else you may later find that it's
+impossible.
+
+### UserProfile.objects.get() / Client.objects.get / etc.
+
+In our Django code, never do direct `UserProfile.objects.get(email=foo)`
+database queries. Instead always use `get_user_profile_by_{email,id}`.
+There are 3 reasons for this:
+
+1.  It's guaranteed to correctly do a case-inexact lookup
+2.  It fetches the user object from remote cache, which is faster
+3.  It always fetches a UserProfile object which has been queried using
+    .selected\_related(), and thus will perform well when one later
+    accesses related models like the Realm.
+
+Similarly we have `get_client` and `get_stream` functions to fetch those
+commonly accessed objects via remote cache.
+
+### Using Django model objects as keys in sets/dicts
+
+Don't use Django model objects as keys in sets/dictionaries -- you will
+get unexpected behavior when dealing with objects obtained from
+different database queries:
+
+For example,
+`UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])`
+is False
+
+You should work with the IDs instead.
+
+### user\_profile.save()
+
+You should always pass the update\_fields keyword argument to .save()
+when modifying an existing Django model object. By default, .save() will
+overwrite every value in the column, which results in lots of race
+conditions where unrelated changes made by one thread can be
+accidentally overwritten by another thread that fetched its UserProfile
+object before the first thread wrote out its change.
+
+### Using raw saves to update important model objects
+
+In most cases, we already have a function in zephyr/lib/actions.py with
+a name like do\_activate\_user that will correctly handle lookups,
+caching, and notifying running browsers via the event system about your
+change. So please check whether such a function exists before writing
+new code to modify a model object, since your new code has a good chance
+of getting at least one of these things wrong.
+
+### `x.attr('zid')` vs. `rows.id(x)`
+
+Our message row DOM elements have a custom attribute `zid` which
+contains the numerical message ID. **Don't access this directly as**
+`x.attr('zid')` ! The result will be a string and comparisons (e.g. with
+`<=`) will give the wrong result, occasionally, just enough to make a
+bug that's impossible to track down.
+
+You should instead use the `id` function from the `rows` module, as in
+`rows.id(x)`. This returns a number. Even in cases where you do want a
+string, use the `id` function, as it will simplify future code changes.
+In most contexts in JavaScript where a string is needed, you can pass a
+number without any explicit conversion.
+
+### Javascript var
+
+Always declare Javascript variables using `var`:
+
+    var x = ...;
+
+In a function, `var` is necessary or else `x` will be a global variable.
+For variables declared at global scope, this has no effect, but we do it
+for consistency.
+
+Javascript has function scope only, not block scope. This means that a
+`var` declaration inside a `for` or `if` acts the same as a `var`
+declaration at the beginning of the surrounding `function`. To avoid
+confusion, declare all variables at the top of a function.
+
+### Javascript `for (i in myArray)`
+
+Don't use it:
+[[1]](http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays),
+[[2]](http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop),
+[[3]](http://www.jslint.com/lint.html#forin)
+
+### jQuery global state
+
+Don't mess with jQuery global state once the app has loaded. Code like
+this is very dangerous:
+
+    $.ajaxSetup({ async: false });
+    $.get(...);
+    $.ajaxSetup({ async: true });
+
+jQuery and the browser are free to run other code while the request is
+pending, which could perform other Ajax requests with the altered
+settings.
+
+Instead, switch to the more general `$.ajax`\_ function, which can take
+options like `async`.
+
+### State and logs files
+
+Do not write state and logs files inside the current working directory
+in the production environment. This will not how you expect, because the
+current working directory for the app changes every time we do a deploy.
+Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
+settings.py for an example.
+
+JS array/object manipulation
+----------------------------
+
+For generic functions that operate on arrays or JavaScript objects, you
+should generally use [Underscore](http://underscorejs.org/). We used to
+use jQuery's utility functions, but the Underscore equivalents are more
+consistent, better-behaved and offer more choices.
+
+A quick conversion table:
+
+       $.each → _.each (parameters to the callback reversed)
+       $.inArray → _.indexOf (parameters reversed)
+       $.grep → _.filter
+       $.map → _.map
+       $.extend → _.extend
+
+There's a subtle difference in the case of `_.extend`; it will replace
+attributes with undefined, whereas jQuery won't:
+
+       $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
+       _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
+
+Also, `_.each` does not let you break out of the iteration early by
+returning false, the way jQuery's version does. If you're doing this,
+you probably want `_.find`, `_.every`, or `_.any`, rather than 'each'.
+
+Some Underscore functions have multiple names. You should always use the
+canonical name (given in large print in the Underscore documentation),
+with the exception of `_.any`, which we prefer over the less clear
+'some'.
+
+More arbitrary style things
+---------------------------
+
+### General
+
+Indentation is four space characters for Python, JS, CSS, and shell
+scripts. Indentation is two space characters for HTML templates.
+
+We never use tabs anywhere in source code we write, but we have some
+third-party files which contain tabs.
+
+Keep third-party static files under the directory
+`zephyr/static/third/`, with one subdirectory per third-party project.
+
+We don't have an absolute hard limit on line length, but we should avoid
+extremely long lines. A general guideline is: refactor stuff to get it
+under 85 characters, unless that makes the code a lot uglier, in which
+case it's fine to go up to 120 or so.
+
+Whitespace guidelines:
+
+-   Put one space (or more for alignment) around binary arithmetic and
+    equality operators.
+-   Put one space around each part of the ternary operator.
+-   Put one space between keywords like `if` and `while` and their
+    associated open paren.
+-   Put one space between the closing paren for `if` and `while`-like
+    constructs and the opening curly brace. Put the curly brace on the
+    same line unless doing otherwise improves readability.
+-   Put no space before or after the open paren for function calls and
+    no space before the close paren for function calls.
+-   For the comma operator and colon operator in languages where it is
+    used for inline dictionaries, put no space before it and at least
+    one space after. Only use more than one space for alignment.
+
+### Javascript
+
+Don't use `==` and `!=` because these operators perform type coercions,
+which can mask bugs. Always use `===` and `!==`.
+
+End every statement with a semicolon.
+
+`if` statements with no braces are allowed, if the body is simple and
+its extent is abundantly clear from context and formatting.
+
+Anonymous functions should have spaces before and after the argument
+list:
+
+    var x = function (foo, bar) { // ...
+
+When calling a function with an anonymous function as an argument, use
+this style:
+
+    $.get('foo', function (data) {
+        var x = ...;
+        // ...
+    });
+
+The inner function body is indented one level from the outer function
+call. The closing brace for the inner function and the closing
+parenthesis for the outer call are together on the same line. This style
+isn't necessarily appropriate for calls with multiple anonymous
+functions or other arguments following them.
+
+Use
+
+    $(function () { ...
+
+rather than
+
+    $(document).ready(function () { ...
+
+and combine adjacent on-ready functions, if they are logically related.
+
+The best way to build complicated DOM elements is a Mustache template
+like `zephyr/static/templates/message.handlebars`. For simpler things
+you can use jQuery DOM building APIs like so:
+
+    var new_tr = $('<tr />').attr('id', zephyr.id);
+
+Passing a HTML string to jQuery is fine for simple hardcoded things:
+
+    foo.append('<p id="selected">foo</p>');
+
+but avoid programmatically building complicated strings.
+
+We used to favor attaching behaviors in templates like so:
+
+    <p onclick="select_zephyr({{id}})">
+
+but there are some reasons to prefer attaching events using jQuery code:
+
+-   Potential huge performance gains by using delegated events where
+    possible
+-   When calling a function from an `onclick` attribute, `this` is not
+    bound to the element like you might think
+-   jQuery does event normalization
+
+Either way, avoid complicated JavaScript code inside HTML attributes;
+call a helper function instead.
+
+### HTML / CSS
+
+Don't use the `style=` attribute. Instead, define logical classes and
+put your styles in `zulip.css`.
+
+Don't use the tag name in a selector unless you have to. In other words,
+use `.foo` instead of `span.foo`. We shouldn't have to care if the tag
+type changes in the future.
+
+Don't use inline event handlers (`onclick=`, etc. attributes). Instead,
+attach a jQuery event handler
+(`$('#foo').on('click', function () {...})`) when the DOM is ready
+(inside a `$(function () {...})` block).
+
+Use this format when you have the same block applying to multiple CSS
+styles (separate lines for each selector):
+
+    selector1,
+    selector2 {
+    };
+
+### Python
+
+-   Scripts should start with `#!/usr/bin/env python` and not
+    `#/usr/bin/python` (the right Python may not be installed in
+    `/usr/bin`) or `#/usr/bin/env/python2.7` (bad for Python 3
+    compatibility).  Don't put a shebang line on a Python file unless
+    it's meaningful to run it as a script. (Some libraries can also be
+    run as scripts, e.g. to run a test suite.)
+-   The first import in a file should be
+    `from __future__ import absolute_import`, per [PEP
+    328](http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports)
+-   Put all imports together at the top of the file, absent a compelling
+    reason to do otherwise.
+-   Unpacking sequences doesn't require list brackets:
+
+        [x, y] = xs    # unnecessary
+        x, y = xs      # better
+
+-   For string formatting, use `x % (y,)` rather than `x % y`, to avoid
+    ambiguity if `y` happens to be a tuple.
+-   When selecting by id, don't use `foo.pk` when you mean `foo.id`.
+    E.g.
+
+        recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
+
+    should be written as
+
+        recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
+
+    in case we ever change the primary keys.
+
+Version Control
+---------------
+
+### Commit Discipline
+
+We follow the Git project's own commit discipline practice of "Each
+commit is a minimal coherent idea". This discipline takes a bit of work,
+but it makes it much easier for code reviewers to spot bugs, and
+makes the commit history a much more useful resource for developers
+trying to understand why the code works the way it does, which also
+helps a lot in preventing bugs.
+
+Coherency requirements for any commit:
+
+-   It should pass tests (so test updates needed by a change should be
+    in the same commit as the original change, not a separate "fix the
+    tests that were broken by the last commit" commit).
+-   It should be safe to deploy individually, or comment in detail in
+    the commit message as to why it isn't (maybe with a [manual] tag).
+    So implementing a new API endpoint in one commit and then adding the
+    security checks in a future commit should be avoided -- the security
+    checks should be there from the beginning.
+-   Error handling should generally be included along with the code that
+    might trigger the error.
+-   TODO comments should be in the commit that introduces the issue or
+    functionality with further work required.
+
+When you should be minimal:
+
+-   Significant refactorings should be done in a separate commit from
+    functional changes.
+-   Moving code from one file to another should be done in a separate
+    commits from functional changes or even refactoring within a file.
+-   2 different refactorings should be done in different commits.
+-   2 different features should be done in different commits.
+-   If you find yourself writing a commit message that reads like a list
+    of somewhat dissimilar things that you did, you probably should have
+    just done 2 commits.
+
+When not to be overly minimal:
+
+-   For completely new features, you don't necessarily need to split out
+    new commits for each little subfeature of the new feature. E.g. if
+    you're writing a new tool from scratch, it's fine to have the
+    initial tool have plenty of options/features without doing separate
+    commits for each one. That said, reviewing a 2000-line giant blob of
+    new code isn't fun, so please be thoughtful about submitting things
+    in reviewable units.
+-   Don't bother to split back end commits from front end commits, even
+    though the backend can often be coherent on its own.
+
+Other considerations:
+
+-   Overly fine commits are easily squashed, but not vice versa, so err
+    toward small commits, and the code reviewer can advise on squashing.
+-   If a commit you write doesn't pass tests, you should usually fix
+    that by amending the commit to fix the bug, not writing a new "fix
+    tests" commit on top of it.
+
+Zulip expects you to structure the commits in your pull requests to form
+a clean history before we will merge them; it's best to write your
+commits following these guidelines in the first place, but if you don't,
+you can always fix your history using git rebase -i.
+
+It can take some practice to get used to writing your commits with a
+clean history so that you don't spend much time doing interactive
+rebases. For example, often you'll start adding a feature, and discover
+you need to a refactoring partway through writing the feature. When that
+happens, we recommend stashing your partial feature, do the refactoring,
+commit it, and then finish implementing your feature.
+
+### Commit Messages
+
+-   The first line of commit messages should be written in the
+    imperative and be kept relatively short while concisely explaining
+    what the commit does. For example:
+
+Bad:
+
+    bugfix
+    gather_subscriptions was broken
+    fix bug #234.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Use present-tense action verbs in your commit messages.
+
+Bad:
+
+    Fixing gather_subscriptions throwing an exception when given bad input.
+    Fixed gather_subscriptions throwing an exception when given bad input.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Please use a complete sentence in the summary, ending with a period.
+-   The rest of the commit message should be written in full prose and
+    explain why and how the change was made. If the commit makes
+    performance improvements, you should generally include some rough
+    benchmarks showing that it actually improves the performance.
+-   When you fix a GitHub issue, [mark that you've fixed the issue in
+    your commit
+    message](https://help.github.com/articles/closing-issues-via-commit-messages/)
+    so that the issue is automatically closed when your code is merged.
+    Zulip's preferred style for this is to have the final paragraph of
+    the commit message read e.g. "Fixes: \#123."
+-   Any paragraph content in the commit message should be line-wrapped
+    to less than 76 characters per line, so that your commit message
+    will be reasonably readable in git log in a normal terminal.
+-   In your commit message, you should describe any manual testing you
+    did in addition to running the automated tests, and any aspects of
+    the commit that you think are questionable and you'd like special
+    attention applied to.
+
+### Tests
+
+All significant new features should come with tests. See testing.
+
+### Third party code
+
+When adding new third-party packages to our codebase, please include
+"[third]" at the beginning of the commit message. You don't necessarily
+need to do this when patching third-party code that's already in tree.

docs/code-style.rst

@@ -1,484 +0,0 @@
-==========================
-Code style and conventions
-==========================
-
-Be consistent!
-==============
-
-Look at the surrounding code, or a similar part of the project, and
-try to do the same thing. If you think the other code has actively bad
-style, fix it (in a separate commit).
-
-When in doubt, send an email to zulip-devel@googlegroups.com with your
-question.
-
-Lint tools
-==========
-
-You can run them all at once with
-
-::
-
-    ./tools/lint-all
-
-You can set this up as a local Git commit hook with
-
-::
-
-    ``tools/setup-git-repo``
-
-The Vagrant setup process runs this for you.
-
-``lint-all`` runs many lint checks in parallel, including
-
-- Javascript (`JSLint <http://www.jslint.com/>`__)
-
-    ``tools/jslint/check-all.js`` contains a pretty fine-grained set of
-    JSLint options, rule exceptions, and allowed global variables. If you
-    add a new global, you'll need to add it to the list.
-
-- Python (`Pyflakes <http://pypi.python.org/pypi/pyflakes>`__)
-- templates
-- Puppet configuration
-- custom checks (e.g. trailing whitespace and spaces-not-tabs)
-
-Secrets
-=======
-
-Please don't put any passwords, secret access keys, etc. inline in the
-code.  Instead, use the ``get_secret`` function in
-``zproject/settings.py`` to read secrets from ``/etc/zulip/secrets.conf``.
-
-Dangerous constructs
-====================
-
-Misuse of database queries
---------------------------
-
-Look out for Django code like this::
-
-   [Foo.objects.get(id=bar.x.id)
-   for bar in Bar.objects.filter(...)
-   if  bar.baz < 7]
-
-This will make one database query for each ``Bar``, which is slow in
-production (but not in local testing!). Instead of a list comprehension,
-write a single query using Django's `QuerySet
-API <https://docs.djangoproject.com/en/dev/ref/models/querysets/>`__.
-
-If you can't rewrite it as a single query, that's a sign that something
-is wrong with the database schema. So don't defer this optimization when
-performing schema changes, or else you may later find that it's
-impossible.
-
-UserProfile.objects.get() / Client.objects.get / etc.
------------------------------------------------------
-
-In our Django code, never do direct
-``UserProfile.objects.get(email=foo)`` database queries. Instead always
-use ``get_user_profile_by_{email,id}``. There are 3 reasons for this:
-
-#. It's guaranteed to correctly do a case-inexact lookup
-#. It fetches the user object from memcached, which is faster
-#. It always fetches a UserProfile object which has been queried using
-   .selected\_related(), and thus will perform well when one later
-   accesses related models like the Realm.
-
-Similarly we have ``get_client`` and ``get_stream`` functions to fetch
-those commonly accessed objects via memcached.
-
-Using Django model objects as keys in sets/dicts
-------------------------------------------------
-
-Don't use Django model objects as keys in sets/dictionaries -- you will
-get unexpected behavior when dealing with objects obtained from
-different database queries:
-
-For example,
-``UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])``
-is False
-
-You should work with the IDs instead.
-
-user\_profile.save()
---------------------
-
-You should always pass the update\_fields keyword argument to .save()
-when modifying an existing Django model object. By default, .save() will
-overwrite every value in the column, which results in lots of race
-conditions where unrelated changes made by one thread can be
-accidentally overwritten by another thread that fetched its UserProfile
-object before the first thread wrote out its change.
-
-Using raw saves to update important model objects
--------------------------------------------------
-
-In most cases, we already have a function in zephyr/lib/actions.py with
-a name like do\_activate\_user that will correctly handle lookups,
-caching, and notifying running browsers via the event system about your
-change. So please check whether such a function exists before writing
-new code to modify a model object, since your new code has a good chance
-of getting at least one of these things wrong.
-
-``x.attr('zid')`` vs. ``rows.id(x)``
-------------------------------------
-
-Our message row DOM elements have a custom attribute ``zid`` which
-contains the numerical message ID. **Don't access this directly as**
-``x.attr('zid')`` ! The result will be a string and comparisons (e.g.
-with ``<=``) will give the wrong result, occasionally, just enough to
-make a bug that's impossible to track down.
-
-You should instead use the ``id`` function from the ``rows`` module, as
-in ``rows.id(x)``. This returns a number. Even in cases where you do
-want a string, use the ``id`` function, as it will simplify future code
-changes. In most contexts in JavaScript where a string is needed, you
-can pass a number without any explicit conversion.
-
-Javascript var
---------------
-
-Always declare Javascript variables using ``var``::
-
-   var x = ...;
-
-In a function, ``var`` is necessary or else ``x`` will be a global
-variable. For variables declared at global scope, this has no effect,
-but we do it for consistency.
-
-Javascript has function scope only, not block scope. This means that a
-``var`` declaration inside a ``for`` or ``if`` acts the same as a
-``var`` declaration at the beginning of the surrounding ``function``. To
-avoid confusion, declare all variables at the top of a function.
-
-Javascript ``for (i in myArray)``
----------------------------------
-
-Don't use it:
-`[1] <http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays>`__,
-`[2] <http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop>`__,
-`[3] <http://www.jslint.com/lint.html#forin>`__
-
-jQuery global state
--------------------
-
-Don't mess with jQuery global state once the app has loaded. Code like
-this is very dangerous::
-
-   $.ajaxSetup({ async: false });
-   $.get(...);
-   $.ajaxSetup({ async: true });
-
-jQuery and the browser are free to run other code while the request is
-pending, which could perform other Ajax requests with the altered
-settings.
-
-Instead, switch to the more general |ajax|_ function, which can take options
-like ``async``.
-
-.. |ajax| replace:: ``$.ajax``
-.. _ajax: http://api.jquery.com/jQuery.ajax
-
-State and logs files
---------------------
-
-Do not write state and logs files inside the current working directory
-in the production environment. This will not how you expect, because the
-current working directory for the app changes every time we do a deploy.
-Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
-settings.py for an example.
-
-JS array/object manipulation
-============================
-
-For generic functions that operate on arrays or JavaScript objects, you
-should generally use `Underscore <http://underscorejs.org/>`__. We used
-to use jQuery's utility functions, but the Underscore equivalents are
-more consistent, better-behaved and offer more choices.
-
-A quick conversion table::
-
-      $.each → _.each (parameters to the callback reversed)
-      $.inArray → _.indexOf (parameters reversed)
-      $.grep → _.filter
-      $.map → _.map
-      $.extend → _.extend
-
-There's a subtle difference in the case of ``_.extend``; it will replace
-attributes with undefined, whereas jQuery won't::
-
-      $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
-      _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
-
-Also, ``_.each`` does not let you break out of the iteration early by
-returning false, the way jQuery's version does. If you're doing this,
-you probably want ``_.find``, ``_.every``, or ``_.any``, rather than
-'each'.
-
-Some Underscore functions have multiple names. You should always use the
-canonical name (given in large print in the Underscore documentation),
-with the exception of ``_.any``, which we prefer over the less clear
-'some'.
-
-More arbitrary style things
-===========================
-
-General
--------
-
-Indentation is four space characters for Python, JS, CSS, and shell
-scripts. Indentation is two space characters for HTML templates.
-
-We never use tabs anywhere in source code we write, but we have some
-third-party files which contain tabs.
-
-Keep third-party static files under the directory
-``zephyr/static/third/``, with one subdirectory per third-party project.
-
-We don't have an absolute hard limit on line length, but we should avoid
-extremely long lines. A general guideline is: refactor stuff to get it
-under 85 characters, unless that makes the code a lot uglier, in which
-case it's fine to go up to 120 or so.
-
-Whitespace guidelines:
-
--  Put one space (or more for alignment) around binary arithmetic and
-   equality operators.
--  Put one space around each part of the ternary operator.
--  Put one space between keywords like ``if`` and ``while`` and their
-   associated open paren.
--  Put one space between the closing paren for ``if`` and ``while``-like
-   constructs and the opening curly brace. Put the curly brace on the
-   same line unless doing otherwise improves readability.
--  Put no space before or after the open paren for function calls and no
-   space before the close paren for function calls.
--  For the comma operator and colon operator in languages where it is
-   used for inline dictionaries, put no space before it and at least one
-   space after. Only use more than one space for alignment.
-
-Javascript
-----------
-
-Don't use ``==`` and ``!=`` because these operators perform type
-coercions, which can mask bugs. Always use ``===`` and ``!==``.
-
-End every statement with a semicolon.
-
-``if`` statements with no braces are allowed, if the body is simple and
-its extent is abundantly clear from context and formatting.
-
-Anonymous functions should have spaces before and after the argument
-list::
-
-   var x = function (foo, bar) { // ...
-
-When calling a function with an anonymous function as an argument, use
-this style::
-
-   $.get('foo', function (data) {
-       var x = ...;
-       // ...
-   });
-
-The inner function body is indented one level from the outer function
-call. The closing brace for the inner function and the closing
-parenthesis for the outer call are together on the same line. This style
-isn't necessarily appropriate for calls with multiple anonymous
-functions or other arguments following them.
-
-Use
-
-::
-
-   $(function () { ...
-
-rather than
-
-::
-
-   $(document).ready(function () { ...
-
-and combine adjacent on-ready functions, if they are logically related.
-
-The best way to build complicated DOM elements is a Mustache template
-like ``zephyr/static/templates/message.handlebars``. For simpler things
-you can use jQuery DOM building APIs like so::
-
-   var new_tr = $('<tr />').attr('id', zephyr.id);
-
-Passing a HTML string to jQuery is fine for simple hardcoded things::
-
-   foo.append('<p id="selected">foo</p>');
-
-but avoid programmatically building complicated strings.
-
-We used to favor attaching behaviors in templates like so::
-
-    <p onclick="select_zephyr({{id}})">
-
-but there are some reasons to prefer attaching events using jQuery code:
-
--  Potential huge performance gains by using delegated events where
-   possible
--  When calling a function from an ``onclick`` attribute, ``this`` is
-   not bound to the element like you might think
--  jQuery does event normalization
-
-Either way, avoid complicated JavaScript code inside HTML attributes;
-call a helper function instead.
-
-HTML / CSS
-----------
-
-Don't use the ``style=`` attribute. Instead, define logical classes and
-put your styles in ``zulip.css``.
-
-Don't use the tag name in a selector unless you have to. In other words,
-use ``.foo`` instead of ``span.foo``. We shouldn't have to care if the
-tag type changes in the future.
-
-Don't use inline event handlers (``onclick=``, etc. attributes).
-Instead, attach a jQuery event handler
-(``$('#foo').on('click', function () {...})``) when the DOM is ready
-(inside a ``$(function () {...})`` block).
-
-Use this format when you have the same block applying to multiple CSS
-styles (separate lines for each selector)::
-
-    selector1,
-    selector2 {
-    };
-
-Python
-------
-
--  Scripts should start with ``#!/usr/bin/env python`` and not
-   ``#!/usr/bin/python``. See commit ``437d4aee`` for an explanation of
-   why. Don't put such a line on a Python file unless it's meaningful to
-   run it as a script. (Some libraries can also be run as scripts, e.g.
-   to run a test suite.)
--  The first import in a file should be
-   ``from __future__ import absolute_import``, per `PEP
-   328 <http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports>`__
--  Put all imports together at the top of the file, absent a compelling
-   reason to do otherwise.
--  Unpacking sequences doesn't require list brackets::
-
-      [x, y] = xs    # unnecessary
-      x, y = xs      # better
-
--  For string formatting, use ``x % (y,)`` rather than ``x % y``, to
-   avoid ambiguity if ``y`` happens to be a tuple.
--  When selecting by id, don't use ``foo.pk`` when you mean ``foo.id``.
-   E.g.
-
-   ::
-
-      recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
-
-   should be written as
-
-   ::
-
-      recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
-
-   in case we ever change the primary keys.
-
-Version Control
-===============
-
-Commit Discipline
------------------
-
-We follow the Git project's own commit discipline practice of "Each
-commit is a minimal coherent idea".
-
-Coherency requirements for any commit:
-
--  It should pass tests (so test updates needed by a change should be in
-   the same commit as the original change, not a separate "fix the tests
-   that were broken by the last commit" commit).
--  It should be safe to deploy individually, or comment in detail in the
-   commit message as to why it isn't (maybe with a [manual] tag). So
-   implementing a new API endpoint in one commit and then adding the
-   security checks in a future commit should be avoided -- the security
-   checks should be there from the beginning.
--  Error handling should generally be included along with the code that
-   might trigger the error.
--  TODO comments should be in the commit that introduces the
-   issue or functionality with further work required.
-
-When you should be minimal:
-
--  Significant refactorings should be done in a separate commit from
-   functional changes.
--  Moving code from one file to another should be done in a separate
-   commits from functional changes or even refactoring within a file.
--  2 different refactorings should be done in different commits.
--  2 different features should be done in different commits.
--  If you find yourself writing a commit message that reads like a list
-   of somewhat dissimilar things that you did, you probably should have
-   just done 2 commits.
-
-When not to be overly minimal:
-
--  For completely new features, you don't necessarily need to split out
-   new commits for each little subfeature of the new feature. E.g. if
-   you're writing a new tool from scratch, it's fine to have the initial
-   tool have plenty of options/features without doing separate commits
-   for each one.  That said, reviewing a 2000-line giant blob of new
-   code isn't fun, so please be thoughtful about submitting things in
-   reviewable units.
--  Don't bother to split back end commits from front end commits, even
-   though the backend can often be coherent on its own.
-
-Other considerations:
-
--  Overly fine commits are easily squashed, but not vice versa, so err
-   toward small commits, and the code reviewer can advise on squashing.
-
-It can take some practice to get used to writing your commits this
-way.  For example, often you'll start adding a feature, and discover
-you need to a refactoring partway through writing the feature.  When
-that happens, we recommend stashing your partial feature, do the
-refactoring, commit it, and then finish implementing your feature.
-
-Commit Messages
----------------
-
--  The first line of commit messages should be written in the imperative
-   and be kept relatively short while concisely explaining what the
-   commit does. For example:
-
-Bad::
-
-   bugfix
-   gather_subscriptions was broken
-
-Good::
-
-   Prevent gather_subscriptions from throwing an exception when given bad input.
-
--  Please use a complete sentence, ending with a period.
-
--  The rest of the commit message should be written in full prose and
-   explain why and how the change was made. If the commit makes
-   performance improvements, you should generally include some rough
-   benchmarks showing that it actually improves the performance.
-
--  In your commit message, you should describe any manual testing you
-   did in addition to running the automated tests, and any aspects of
-   the commit that you think are questionable and you'd like special
-   attention applied to.
-
-Tests
------
-
-All significant new features should come with tests.
-
-Third party code
-----------------
-
-When adding new third-party packages to our codebase, please include
-"[third]" at the beginning of the commit message. You don't necessarily
-need to do this when patching third-party code that's already in tree.

docs/conf.py

@@ -15,6 +15,7 @@
 import sys
 import os
 import shlex
+from typing import Dict, List, Optional
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -29,16 +30,11 @@ import shlex
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = [] # type: List[str]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
 
@@ -64,7 +60,7 @@ release = '0.1'
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = None # type: Optional[str]
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
@@ -227,7 +223,7 @@ latex_elements = {
 
 # Latex figure (float) alignment
 #'figure_align': 'htbp',
-}
+} # type: Dict[str, str]
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
@@ -293,3 +289,13 @@ texinfo_documents = [
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 #texinfo_no_detailmenu = False
+
+from recommonmark.parser import CommonMarkParser
+
+source_parsers = {
+        '.md': CommonMarkParser,
+        }
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+source_suffix = ['.rst', '.md']

docs/directory-structure.md

@@ -0,0 +1,115 @@
+Directory structure
+===================
+
+This page documents the Zulip directory structure and how to decide
+where to put a file.
+
+Scripts
+-------
+
+* `scripts/` Scripts that production deployments might run manually
+  (e.g., `restart-server`).
+
+* `scripts/lib/` Scripts that are needed on production deployments but
+  humans should never run.
+
+* `scripts/setup/` Tools that production deployments will only run
+  once, during installation.
+
+* `tools/` Development tools.
+
+---------------------------------------------------------
+
+Bots
+----
+
+* `api/integrations/` Bots distributed as part of the Zulip API bundle.
+
+* `bots/` Previously Zulip internal bots. These usually need a bit of
+   work.
+
+-----------------------------------------------------
+
+Management commands
+-------------------
+
+* `zerver/management/commands/` Management commands one might run at a
+  production deployment site (e.g. scripts to change a value or
+  deactivate a user properly)
+
+-------------------------------------------------------------------------
+
+Views
+-----
+
+* `zerver/tornadoviews.py` Tornado views
+
+* `zerver/views/webhooks.py` Webhook views
+
+* `zerver/views/messages.py` message-related views
+
+* `zerver/views/__init__.py` other Django views
+
+----------------------------------------
+
+Jinja2 Compatibility Files
+--------------------------
+
+* `zproject/jinja2/__init__.py` Jinja2 environment
+
+* `zproject/jinja2/backends.py` Jinja2 backend
+
+* `zproject/jinja2/compressors.py` Jinja2 compatible functions of
+   Django-Pipeline
+
+-----------------------------------------------------------------------
+
+Static assets
+-------------
+
+* `assets/` For assets not to be served to the web (e.g. the system to
+            generate our favicons)
+
+* `static/` For things we do want to both serve to the web and
+            distribute to production deployments (e.g. the webpages)
+
+---------------------------------------------------------------
+
+Puppet
+------
+
+* `puppet/zulip/` For configuration for production deployments
+
+-------------------------------------------------------------------
+
+Templates
+---------
+
+* `templates/zerver/` For Jinja2 templates for the backend (for zerver app)
+
+* `static/templates/` Handlebars templates for the frontend
+
+-----------------------------------------------------------------------
+
+Tests
+-----
+
+* `zerver/tests/` Backend tests
+
+* `frontend_tests/node_tests/` Node Frontend unit tests
+
+* `frontend_tests/casper_tests/` Casper frontend tests
+
+-----------------------------------------------------------------------
+
+
+Documentation
+-------------
+
+*  `docs/`        Source for this documentation
+
+--------------------------------------------------------------
+
+You can consult the repository's `.gitattributes` file to see exactly
+which components are excluded from production releases (release
+tarballs are generated using `tools/build-release-tarball`).

docs/directory-structure.rst

@@ -1,95 +0,0 @@
-===================
-Directory structure
-===================
-
-This page documents the Zulip directory structure and how to decide where to
-put a file.
-
-Scripts
-=======
-
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/``       | Scripts that production deployments might run manually (e.g. ``restart-server``)  |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``bin/``           | Scripts that are needed on production deployments but humans should never run     |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/setup/`` | Tools that production deployments will only run once, during installation         |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``tools/``         | Development tools                                                                 |
-+--------------------+-----------------------------------------------------------------------------------+
-
-Bots
-====
-
-+------------------------+----------------------------------------------------------------------+
-| ``api/integrations``   | Bots distributed as part of the Zulip API bundle.                    |
-+------------------------+----------------------------------------------------------------------+
-| ``bots/``              | Previously Zulip internal bots.  These usually need a bit of work.   |
-+------------------------+----------------------------------------------------------------------+
-
-Management commands
-===================
-
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-| ``zerver/management/commands/``     | Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly)   |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-
-Views
-=====
-
-+--------------------------------+-----------------------------------------+
-| ``zerver/tornadoviews.py``     | Tornado views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/webhooks.py``   | Webhook views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/messages.py``   | message-related views                   |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/__init__.py``   | other Django views                      |
-+--------------------------------+-----------------------------------------+
-
-Static assets
-=============
-
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``assets/``   | For assets not to be served to the web (e.g. the system to generate our favicons)                             |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``static/``   | For things we do want to both serve to the web and distribute to production deployments (e.g. the webpages)   |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-
-Puppet
-======
-
-+--------------------+----------------------------------------------------------------------------------+
-| ``puppet/zulip``   | For configuration for production deployments                                     |
-+--------------------+----------------------------------------------------------------------------------+
-
-Templates
-=========
-
-+--------------------------+--------------------------------------------------------+
-| ``templates/zerver``     | For templates related to zerver views                  |
-+--------------------------+--------------------------------------------------------+
-| ``static/templates``     | Handlebars templates for the frontend                  |
-+--------------------------+--------------------------------------------------------+
-
-Tests
-=====
-
-+------------------------+-----------------------------------+
-| ``zerver/test*.py``             | Backend tests            |       |
-+------------------------+-----------------------------------+
-| ``zerver/tests/frontend/node``  | Node Frontend unit tests |
-+------------------------+-----------------------------------+
-| ``zerver/tests/frontend/tests`` | Casper frontend tests    |
-+------------------------+-----------------------------------+
-
-Documentation
-=============
-
-+-------------+-----------------------------------------------+
-| ``docs/``   | Source for this documentation                 |
-+-------------+-----------------------------------------------+
-
-You can consult the repository's .gitattributes file to see exactly
-which components are excluded from production releases (release
-tarballs are generated using tools/build-release-tarball).

docs/front-end-build-process.md

@@ -0,0 +1,73 @@
+Front End Build Process
+=======================
+
+This page documents additional information that may be useful when
+developing new features for Zulip that require front-end changes. For a
+more general overview, see the new-feature-tutorial. The code-style
+documentation also has relevant information about how Zulip's code is
+structured.
+
+Primary build process
+---------------------
+
+Most of the existing JS in Zulip is written in IIFE-wrapped modules, one
+per file in the static/js directory. When running Zulip in development
+mode, each file is loaded seperately. In production mode (and when
+creating a release tarball using tools/build-release-tarball),
+JavaScript files are concatenated and minified.
+
+If you add a new JavaScript file, it needs to be specified in the
+JS\_SPECS dictionary defined in zproject/settings.py to be included in
+the concatenated file.
+
+Webpack/CommonJS modules
+------------------------
+
+New JS written for Zulip can be written as CommonJS modules (bundled
+using [webpack](https://webpack.github.io/), though this will taken care
+of automatically whenever `run-dev.py` is running). (CommonJS is the
+same module format that Node uses, so see the [Node
+documentation](https://nodejs.org/docs/latest/api/modules.html) for
+more information on the syntax.)
+
+Benefits of using CommonJS modules over the
+[IIFE](http://benalman.com/news/2010/11/immediately-invoked-function-expression/)
+module approach:
+
+-   namespacing/module boilerplate will be added automatically in the
+    bundling process
+-   dependencies between modules are more explicit and easier to trace
+-   no separate list of JS files needs to be maintained for
+    concatenation and minification
+-   third-party libraries can be more easily installed/versioned using
+    npm
+-   running the same code in the browser and in Node for testing is
+    simplified (as both environments use the same module syntax)
+
+The entry point file for the bundle generated by webpack is
+`static/js/src/main.js`. Any modules you add will need to be required
+from this file (or one of its dependencies) in order to be included in
+the script bundle.
+
+Adding static files
+-------------------
+
+To add a static file to the app (JavaScript, CSS, images, etc), first
+add it to the appropriate place under `static/`.
+
+-   Third-party files should all go in `static/third/`. Tag the commit
+    with "[third]" when adding or modifying a third-party package.
+-   Our own JS lives under `static/js`; CSS lives under `static/styles`.
+-   JavaScript and CSS files are combined and minified in production. In
+    this case all you need to do is add the filename to PIPELINE\_CSS or
+    JS\_SPECS in `zproject/settings.py`. (If you plan to only use the
+    JS/CSS within the app proper, and not on the login page or other
+    standalone pages, put it in the 'app' category.)
+
+If you want to test minified files in development, look for the
+`PIPELINE =` line in `zproject/settings.py` and set it to `True` -- or
+just set `DEBUG = False`.
+
+Note that `static/html/{400,5xx}.html` will only render properly if
+minification is enabled, since they hardcode the path
+`static/min/portico.css`.

docs/html_unescape.py

@@ -1,4 +1,5 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
+from __future__ import print_function
 
 # Remove HTML entity escaping left over from MediaWiki->rST conversion.
 

docs/index.rst

@@ -6,15 +6,35 @@
 Welcome to Zulip documentation!
 ===============================
 
+Zulip is a powerful, open source group chat application. Written in
+Python and using the Django framework, Zulip supports both private
+messaging and group chats via conversation streams.
+
+Zulip also supports fast search, drag-and-drop file uploads, image
+previews, group private messages, audible notifications, missed-message
+emails, desktop apps, and much more.
+
+Further information on the Zulip project and its features can be found
+at `https://www.zulip.org <https://www.zulip.org>`__ and in these
+docs.  Our code is available at `our GitHub repository
+<https://github.com/zulip/>`__.
+
+This set of documents covers installation and contribution instructions.
+
 Contents:
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 3
 
+   readme-symlink
+   architecture-overview
+   integration-guide
    new-feature-tutorial
-   code-style
+   code-contribution-checklist
    directory-structure
-   testing
+   translating
+   changelog
+   roadmap
 
 Indices and tables
 ==================

docs/integration-guide.md

@@ -0,0 +1,182 @@
+# How to write a new integration
+
+Integrations are one of the most important parts of a group chat tool
+like Zulip, and we are committed to making integrating with Zulip and
+getting you integration merged upstream so everyone else can benefit
+from it as easy as possible while maintaining the high quality of the
+Zulip integrations library.
+
+Contributions to this guide are very welcome, so if you run into any
+issues following these instructions or come up with any tips or tools
+that help writing integration, please email
+zulip-devel@googlegroups.com, open an issue, or submit a pull request
+to share your ideas!
+
+## Types of integrations
+
+We have several different ways that we integrate with 3rd part
+products, ordered here by which types we prefer to write:
+
+1. Webhook integrations (examples: Freshdesk, GitHub), where the
+third-party service supports posting content to a particular URI on
+our site with data about the event.  For these, you usually just need
+to add a new handler in `zerver/views/webhooks.py` (plus
+test/document/etc.).  An example commit implementing a new webhook is:
+https://github.com/zulip/zulip/pull/324.
+
+2. Python script integrations (examples: SVN, Git), where we can get
+the service to call our integration (by shelling out or otherwise),
+passing in the required data.  Our preferred model for these is to
+ship these integrations in our API release tarballs (by writing the
+integration in `api/integrations`).
+
+3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user
+needs to install a plugin into their existing software.  These are
+often more work, but for some products are the only way to integrate
+with the product at all.
+
+## General advice for writing integrations
+
+* Consider using our Zulip markup to make the output from your
+  integration especially attractive or useful (e.g.  emoji, markdown
+  emphasis, @-mentions, or `!avatar(email)`).
+
+* Use topics effectively to ensure sequential messages about the same
+  thing are threaded together; this makes for much better consumption
+  by users.  E.g. for a bug tracker integration, put the bug number in
+  the topic for all messages; for an integration like Nagios, put the
+  service in the topic.
+
+* Integrations that don't match a team's workflow can often be
+  uselessly spammy.  Give careful thought to providing options for
+  triggering Zulip messages only for certain message types, certain
+  projects, or sending different messages to different streams/topics,
+  to make it easy for teams to configure the integration to support
+  their workflow.
+
+* Consistently capitalize the name of the integration in the
+  documentation and the Client name the way the vendor does.  It's OK
+  to use all-lower-case in the implementation.
+
+* Sometimes it can be helpful to contact the vendor if it appears they
+  don't have an API or webhook we can use -- sometimes the right API
+  is just not properly documented.
+
+## Writing Webhook integrations
+
+New Zulip webhook integrations can take just a few hours to write,
+including tests and documentation, if you use the right process.
+Here's how we recommend doing it:
+
+* First, use http://requestb.in/ or a similar site to capture an
+  example webhook payload from the service you're integrating.  You
+  can use these captured payloads to create a set of test fixtures for
+  your integration under `zerver/fixtures`.
+
+* Then write a draft webhook handler under `zerver/views/webhooks/`;
+  there are a lot of examples in that directory.  We recommend
+  templating off a short one (like `stash.py` or `zendesk.py`), since
+  the longer ones usually just have more complex parsing which can
+  obscure what's common to all webhook integrations.  In addition to
+  writing the integration itself, you'll need to add an entry in
+  `zproject/urls.py` for your webhook; search for `webhook` in that
+  file to find the existing ones (and please add yours in the
+  alphabetically correct place).
+
+* Then write a test for your fixture in `zerver/tests/test_hooks.py`, and
+  you can iterate on the tests and webhooks handler until they work,
+  all without ever needing to post directly from the server you're
+  integrating to your Zulip development machine.  To run just the
+  tests from the test class you wrote, you can use e.g.
+
+  ```
+  test-backend zerver.tests.test_hooks.PagerDutyHookTests
+  ```
+
+  See
+  https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+  for more details on the Zulip test runner.
+
+* Once you've gotten your webhook working and passing a test, capture
+  payloads for the other common types of posts the service's webhook
+  will make, and add tests for them; usually this part of the process
+  is pretty fast.  Webhook integration tests should all use fixtures
+  (as opposed to contacting the service), since otherwise the tests
+  can't run without Internet access and some sort of credentials for
+  the service.
+
+* Finally, write documentation for the integration (see below)!
+
+## Writing Python script and plugin integrations integrations
+
+For plugin integrations, usually you will need to consult the
+documentation for the third party software in order to learn how to
+write the integration.  But we have a few notes on how to do these:
+
+* You should always send messages by POSTing to URLs of the form
+`https://zulip.example.com/v1/messages/`, not the legacy
+`/api/v1/send_message` message sending API.
+
+* We usually build Python script integration with (at least) 2 files:
+`zulip_foo_config.py`` containing the configuration for the
+integration including the bots' API keys, plus a script that reads
+from this configuration to actually do the work (that way, it's
+possible to update the script without breaking users' configurations).
+
+* Be sure to test your integration carefully and document how to
+  install it (see notes on documentation below).
+
+* You should specify a clear HTTP User-Agent for your integration. The
+user agent should at a minimum identify the integration and version
+number, separated by a slash. If possible, you should collect platform
+information and include that in `()`s after the version number. Some
+examples of ideal UAs are:
+
+```
+ZulipDesktop/0.7.0 (Ubuntu; 14.04)
+ZulipJenkins/0.1.0 (Windows; 7.2)
+ZulipMobile/0.5.4 (Android; 4.2; maguro)
+```
+
+## Documenting your integration
+
+Every Zulip integration must be documented in
+`templates/zerver/integrations.html`.  Usually, this involves a few
+steps:
+
+* Add an `integration-lozenge` class block in the alphabetically
+  correct place in the main integration list, using the logo for the
+  integrated software.
+
+* Add an `integration-instructions` class block also in the
+  alphabetically correct place, explaining all the steps required to
+  setup the integration, including what URLs to use, etc.  If there
+  are any screens in the product involved, take a few screenshots with
+  the input fields filled out with sample values in order to make the
+  instructions really easy to follow.  For the screenshots, use
+  something like `github-bot@example.com` for the email addresses and
+  an obviously fake API key like `abcdef123456790`.
+
+* Finally, generate a message sent by the integration and take a
+  screenshot of the message to provide an example message in the
+  documentation. If your new integration is a webhook integration,
+  you can generate such a message from your test fixtures
+  using `send_webhook_fixture_message`:
+
+  ```
+  ./manage.py send_webhook_fixture_message \
+       --fixture=zerver/fixtures/pingdom/pingdom_imap_down_to_up.json \
+       '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'
+  ```
+
+  When generating the screenshot of a sample message, give your test
+  bot a nice name like "GitHub Bot", use the project's logo as the
+  bot's avatar, and take the screenshots showing the stream/topic bar
+  for the message, not just the message body.
+
+When writing documentation for your integration, be sure to use the
+`{{ external_api_uri }}` template variable, so that your integration
+documentation will provide the correct URL for whatever server it is
+deployed on.  If special configuration is required to set the SITE
+variable, you should document that too, inside an `{% if
+api_site_required %}` check.

docs/logging.md

@@ -0,0 +1,24 @@
+# Logging and Performance Debugging
+
+It's good to have the terminal running `run-dev.py` up as you work since error
+messages including tracebacks along with every backend request will be printed
+there.
+
+The messages will look similar to:
+
+```
+2016-05-20 14:50:22,056 INFO     127.0.0.1       GET     302 528ms (db: 1ms/1q) (+start: 123ms) / (unauth via ?)
+[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0
+2016-05-20 14:50:22,272 INFO     127.0.0.1       GET     200 124ms (db: 3ms/2q) /login/ (unauth via ?)
+2016-05-20 14:50:26,333 INFO     127.0.0.1       POST    302  37ms (db: 6ms/7q) /accounts/login/local/ (unauth via ?)
+[20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0
+2016-05-20 14:50:26,538 INFO     127.0.0.1       GET     200  12ms (db: 1ms/2q) (+start: 53ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
+2016-05-20 14:50:26,657 INFO     127.0.0.1       GET     200  10ms (+start: 8ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
+2016-05-20 14:50:26,959 INFO     127.0.0.1       GET     200 588ms (db: 26ms/21q) / [1463769771:0] (cordelia@zulip.com via website)
+```
+
+The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status
+code, time to process, (optional perf data details, e.g. database time/queries,
+memcached time/queries, Django process startup time, markdown processing time,
+etc.), URL, and "email via client" showing user account involved (if logged in)
+and the type of client they used ("web", "Android", etc.).

docs/markdown.md

@@ -0,0 +1,136 @@
+# Zulip's Markdown implementation
+
+Zulip has a special flavor of Markdown, currently called 'bugdown'
+after Zulip's original name of "humbug". End users are using Bugdown
+within the client, not original Markdown.
+
+Zulip has two implementations of Bugdown.  The first is based on
+Python-Markdown (`zerver/lib/bugdown/`) and is used to authoritatively
+render messages on the backend (and implements expensive features like
+querying the Twitter API to render tweets nicely).  The other is in
+JavaScript, based on marked (`static/js/echo.js`), and is used to
+preview and locally echo messages the moment the sender hits enter,
+without waiting for round trip from the server.  The two
+implementations are tested for compatibility via
+`zerver/tests/test_bugdown.py` and the fixtures under
+`zerver/fixtures/bugdown-data.json`.
+
+The JavaScript implementation knows which types of messages it can
+render correctly, and thus while there is code to rerender messages
+based on the authoritative backend rendering (which would clause a
+change in the rendering visible only to the sender shortly after a
+message is sent), this should never happen, and whenever it does it is
+considered a bug.  Instead, if the frontend doesn't know how to
+correctly render a message, we simply won't echo the message for the
+sender until it's rendered by the backend.  So for example, a message
+containing a link to Twitter will not be rendered by the JavaScript
+implementation because it doesn't support doing the 3rd party API
+queries required to render tweets nicely.
+
+I should note that the below documentation is based on a comparison
+with original Markdown, not newer Markdown variants like CommonMark.
+
+## Zulip's Markdown philosophy
+
+Markdown is great for group chat for the same reason it's been
+successful in products ranging from blogs to wikis to bug trackers:
+it's close enough to how people try to express themselves when writing
+plain text (e.g. emails) that is helps more than getting in the way.
+
+The main issue for using Markdown in instant messaging is that the
+Markdown standard syntax used in a lot of wikis/blogs has nontrivial
+error rates, where the author needs to go back and edit the post to
+fix the formatting after typing it the first time.  While that's
+basically fine when writing a blog, it gets annoying very fast in a
+chat product; even though you can edit messages to fix formatting
+mistakes, you don't want to be doing that often.  There are basically
+2 types of error rates that are important for a product like Zulip:
+
+* What fraction of the time, if you pasted a short technical email
+that you wrote to your team and passed it through your Markdown
+implementation, would you need to change the text of your email for it
+to render in a reasonable way?  This is the "accidental Markdown
+syntax" problem, common with Markdown syntax like the italics syntax
+interacting with talking about `char *`s.
+
+* What fraction of the time do users attempting to use a particular
+Markdown syntax actually succeed at doing so correctly?  Syntax like
+required a blank line between text and the start of a bulleted list
+raise this figure substantially.
+
+Both of these are minor issues for most products using Markdown, but
+they are major problems in the instant messaging context, because one
+can't edit a message that has already been sent and users are
+generally writing quickly.  Zulip's Markdown strategy is based on the
+principles of giving users the power they need to express complicated
+ideas in a chat context while minimizing those two error rates.
+
+## Zulip's Changes to Markdown
+
+Below, we document the changes that Zulip has against stock
+Python-Markdown; some of the features we modify / disable may already
+be non-standard.
+
+### Basic syntax
+
+* Enable `nl2br</tt> extension: this means one newline creates a line
+  break (not paragraph break).
+
+* Disable italics entirely.  This resolves an issue where people were
+  using `*` and `_` and hitting it by mistake too often.  E.g. with
+  stock Markdown `You should use char * instead of void * there` would
+  trigger italics.
+
+* Allow only `**` syntax for bold, not `__` (easy to hit by mistake if
+  discussing Python `__init__` or something)
+
+* Disable special use of `\` to escape other syntax. Rendering `\\` as
+  `\` was hugely controversial, but having no escape syntax is also
+  controversial.  We may revisit this.  For now you can always put
+  things in code blocks.
+
+### Lists
+
+* Allow tacking a bulleted list or block quote onto the end of a
+  paragraph, i.e. without a blank line before it
+
+* Allow only `*` for bulleted lists, not `+` or `-` (previously
+  created confusion with diff-style text sloppily not included in a
+  code block)
+
+* Disable ordered list syntax: it automatically renumbers, which can
+  be really confusing when sending a numbered list across multiple
+  messages.
+
+### Links
+
+* Enable auto-linkification, both for `http://...` and guessing at
+  things like `t.co/foo`.
+
+* Force links to be absolute. `[foo](google.com)` will go to
+  `http://google.com`, and not `http://zulip.com/google.com` which
+  is the default behavior.
+
+* Set `target="_blank"` and `title=`(the url) on every link tag so
+  clicking always opens a new window
+
+* Disable link-by-reference syntax, `[foo][bar]` ... `[bar]: http://google.com`
+
+### Code
+
+* Enable fenced code block extension, with syntax highlighting
+
+* Disable line-numbering within fenced code blocks -- the `<table>`
+  output confused our web client code.
+
+### Other
+
+* Disable headings, both `# foo` and `== foo ==` syntax: they don't
+  make much sense for chat messages.
+
+* Disabled images.
+
+* Allow embedding any avatar as a tiny (list bullet size) image.  This
+  is used primarily by version control integrations.
+
+* We added the `~~~ quote` block quote syntax.

docs/mypy.md

@@ -0,0 +1,165 @@
+# Testing with the mypy Python static type checker
+
+[mypy](http://mypy-lang.org/) is a compile-time static type checker
+for Python, allowing optional, gradual typing of Python code.  Zulip
+is using mypy's Python 2 compatible syntax for type annotations, which
+means that type annotations are written inside comments that start
+with `# type: `.  Here's a brief example of the mypy syntax we're
+using in Zulip:
+
+```
+user_dict = {} # type: Dict[str, UserProfile]
+
+def get_user_profile_by_email(email):
+    # type: (str) -> UserProfile
+    ... # Actual code of the function here
+```
+
+You can learn more about it at:
+
+* [The mypy cheat
+  sheet](https://github.com/python/mypy/blob/master/docs/source/cheat_sheet.rst)
+  is the best resource for quickly understanding how to write the PEP
+  484 type annotations used by mypy correctly.
+
+* The [Python 2 type annotation syntax spec in PEP
+  484](https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code)
+
+* [Using mypy with Python 2 code](http://mypy.readthedocs.io/en/latest/python2.html)
+
+The mypy type checker is run automatically as part of Zulip's Travis
+CI testing process.
+
+## Zulip goals
+
+Zulip is hoping to reach 100% of the codebase annotated with mypy
+static types, and then enforce that it stays that way.  Our current
+coverage is shown in
+[Coveralls](https://coveralls.io/github/zulip/zulip).
+
+## Installing mypy
+
+If you installed Zulip's development environment correctly, mypy
+should already be installed inside the Python 3 virtualenv at
+`zulip-py3-venv` (mypy only supports Python 3).  If it isn't installed
+(e.g. because you haven't reprovisioned recently), you can run
+`tools/install-mypy` to install it.
+
+## Running mypy on Zulip's code locally
+
+To run mypy on Zulip's python code, run the command:
+
+    tools/run-mypy
+
+It will output errors in the same style of a compiler.  For example,
+if your code has a type error like this:
+
+```
+foo = 1
+foo = '1'
+```
+
+you'll get an error like this:
+
+```
+test.py: note: In function "test":
+test.py:200: error: Incompatible types in assignment (expression has type "str", variable has type "int")
+```
+
+If you need help interpreting or debugging mypy errors, please feel
+free to mention @sharmaeklavya2 or @timabbott on your pull request (or
+email zulip-devel@googlegroups.com) to get help; we'd love to both
+build a great troubleshooting guide in this doc and also help
+contribute improvements to error messages upstream.
+
+Since mypy is a new tool under rapid development and occasionally
+makes breaking changes, Zulip is using a pinned version of mypy from
+its [git repository](https://github.com/python/mypy) rather than
+tracking the (older) latest mypy release on pypi.
+
+## Excluded files
+
+Since several python files in Zulip's code don't pass mypy's checks
+(even for unannotated code) right now, a list of files to be excluded
+from the check for CI is present in tools/run-mypy.
+
+To run mypy on all python files, ignoring the exclude list, you can
+pass the `--all` option to tools/run-mypy.
+
+    tools/run-mypy --all
+
+If you type annotate some of those files so that they pass without
+errors, please remove them from the exclude list.
+
+## Mypy is there to find bugs in Zulip before they impact users
+
+For the purposes of Zulip development, you can treat `mypy` like a
+much more powerful linter that can catch a wide range of bugs.  If,
+after running tools/run-mypy on your Zulip branch, you get mypy
+errors, it's important to get to the bottom of the issue, not just do
+something quick to silence the warnings.  Possible explanations include:
+
+* A bug in any new type annotations you added.
+* A bug in the existing type annotations.
+* A bug in Zulip!
+* Some Zulip code is correct but confusingly reuses variables with
+  different types.
+* A bug in mypy (though this is increasingly rare as mypy is now
+  fairly mature as a project).
+
+Each explanation has its own solution, but in every case the result
+should be solving the mypy warning in a way that makes the Zulip
+codebase better.  If you need help understanding an issue, please feel
+free to, mention @sharmaeklavya2 or @timabbott on the relevant pull
+request or issue on GitHub.
+
+If you think you have found a bug in Zulip or mypy, inform the zulip
+developers by opening an issue on [Zulip's github
+repository](https://github.com/zulip/zulip/issues) or posting on
+[zulip-devel](https://groups.google.com/d/forum/zulip-devel).  If it's
+indeed a mypy bug, we can help with reporting it upstream.
+
+## Annotating strings
+
+In python 3, strings can have non-ASCII characters without any problems.
+Such characters are required to support languages which use non-latin
+scripts like Japanese and Hindi.  They are also needed to support special
+characters like mathematical symbols, musical symbols, etc.
+In python 2, however, `str` generally doesn't work well with non-ASCII
+characters.  That's why `unicode` was introduced in python 2.
+
+But there are problems with the `unicode` and `str` system.  Implicit
+conversions between `str` and `unicode` use the `ascii` codec, which
+fails on strings containing non-ASCII characters.  Such errors are hard
+to detect by people who always write in English.  To minimize such
+implicit conversions, we should have a strict separation between `str`
+and `unicode` in python 2.  It might seem that using `unicode` everywhere
+will solve all problems, but unfortunately it doesn't.  This is because
+some parts of the standard library and the python language (like keyword
+argument unpacking) insist that parameters passed to them are `str`.
+
+To make our code work correctly on python 2, we have to identify strings
+which contain data which could come from non-ASCII sources like stream
+names, people's names, domain names, content of messages, emails, etc.
+These strings should be `unicode`.  We also have to identify strings
+which should be `str` like Exception names, attribute names, parameter
+names, etc.
+
+Mypy can help with this.  We just have to annotate each string as either
+`str` or `unicode` and mypy's static type checking will tell us if we
+are incorrectly mixing the two.  However, `unicode` is not defined in
+python 3.  We want our code to be python 3 compatible in the future.
+This can be achieved using 'six', a Python 2 and 3 compatibility library.
+
+`six.text_type` is defined as `str` on python 3 and as `unicode` on
+python 2.  We'll be using `text_type` (instead of `unicode`) and `str`
+to annotate strings in Zulip's code.  We follow the style of doing
+`from six import text_type` and using `text_type` for annotation instead
+of doing `import six` and using `six.text_type` for annotation, because
+`text_type` is used so extensively for type annotations that we don't
+need to be that verbose.
+
+Sometimes you'll find that you have to convert strings from one type to
+another.  `zerver/lib/str_utils.py` has utility functions to help with that.
+It also has documentation (in docstrings) which explains the right way
+to use them.

docs/new-feature-tutorial.md

@@ -0,0 +1,218 @@
+How to write a new application feature
+======================================
+
+The changes needed to add a new feature will vary, of course, but this
+document provides a general outline of what you may need to do, as well
+as an example of the specific steps needed to add a new feature: adding
+a new option to the application that is dynamically synced through the
+data system in real-time to all browsers the user may have open.
+
+General Process
+---------------
+
+### Adding a field to the database
+
+**Update the model:** The server accesses the underlying database in
+`zerver/ models.py`. Add a new field in the appropriate class.
+
+**Create and run the migration:** To create and apply a migration, run:
+:
+
+./manage.py makemigrations ./manage.py migrate
+
+**Test your changes:** Once you've run the migration, restart memcached
+on your development server (`/etc/init.d/memcached restart`) and then
+restart `run-dev.py` to avoid interacting with cached objects.
+
+### Backend changes
+
+**Database interaction:** Add any necessary code for updating and
+interacting with the database in `zerver/lib/actions.py`. It should
+update the database and send an event announcing the change.
+
+**Application state:** Modify the `fetch_initial_state_data` and
+`apply_events` functions in `zerver/lib/actions.py` to update the state
+based on the event you just created.
+
+**Backend implementation:** Make any other modifications to the backend
+required for your change.
+
+**Testing:** At the very least, add a test of your event data flowing
+through the system in `test_events.py`.
+
+### Frontend changes
+
+**JavaScript:** Zulip's JavaScript is located in the directory
+`static/js/`. The exact files you may need to change depend on your
+feature. If you've added a new event that is sent to clients, be sure to
+add a handler for it to `static/js/server_events.js`.
+
+**CSS:** The primary CSS file is `static/styles/zulip.css`. If your new
+feature requires UI changes, you may need to add additional CSS to this
+file.
+
+**Templates:** The initial page structure is rendered via Jinja2
+templates located in `templates/zerver`. For JavaScript, Zulip uses
+Handlebars templates located in `static/templates`. Templates are
+precompiled as part of the build/deploy process.
+
+**Testing:** There are two types of frontend tests: node-based unit
+tests and blackbox end-to-end tests. The blackbox tests are run in a
+headless browser using Casper.js and are located in
+`frontend_tests/casper_tests/`. The unit tests use Node's `assert`
+module are located in `frontend_tests/node_tests/`. For more information
+on writing and running tests see the testing documentation \<testing\>.
+
+Example Feature
+---------------
+
+This example describes the process of adding a new setting to Zulip: a
+flag that restricts inviting new users to admins only (the default
+behavior is that any user can invite other users). It is based on an
+actual Zulip feature, and you can review [the original commit in the
+Zulip git
+repo](https://github.com/zulip/zulip/commit/5b7f3466baee565b8e5099bcbd3e1ccdbdb0a408).
+(Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the
+migration format has changed.)
+
+First, update the database and model to store the new setting. Add a new
+boolean field, `realm_invite_by_admins_only`, to the Realm model in
+`zerver/models.py`.
+
+Then create a Django migration that adds a new field,
+`invite_by_admins_only`, to the `zerver_realm` table.
+
+In `zerver/lib/actions.py`, create a new function named
+`do_set_realm_invite_by_admins_only`. This function will update the
+database and trigger an event to notify clients when this setting
+changes. In this case there was an existing `realm|update` event type
+which was used for setting similar flags on the Realm model, so it was
+possible to add a new property to that event rather than creating a new
+one. The property name matches the database field to make it easy to
+understand what it indicates.
+
+The second argument to `send_event` is the list of users whose browser
+sessions should be notified. Depending on the setting, this can be a
+single user (if the setting is a personal one, like time display
+format), only members in a particular stream or all active users in a
+realm. :
+
+    # zerver/lib/actions.py
+
+    def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only):
+      realm.invite_by_admins_only = invite_by_admins_only
+      realm.save(update_fields=['invite_by_admins_only'])
+      event = dict(
+        type="realm",
+        op="update",
+        property='invite_by_admins_only',
+        value=invite_by_admins_only,
+      )
+      send_event(event, active_user_ids(realm))
+      return {}
+
+You then need to add code that will handle the event and update the
+application state. In `zerver/lib/actions.py` update the
+`fetch_initial_state` and `apply_events` functions. :
+
+    def fetch_initial_state_data(user_profile, event_types, queue_id):
+      # ...
+      state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`
+
+In this case you don't need to change `apply_events` because there is
+already code that will correctly handle the realm update event type: :
+
+    def apply_events(state, events, user_profile):
+      for event in events:
+        # ...
+        elif event['type'] == 'realm':
+           field = 'realm_' + event['property']
+           state[field] = event['value']
+
+You then need to add a view for clients to access that will call the
+newly-added `actions.py` code to update the database. This example
+feature adds a new parameter that should be sent to clients when the
+application loads and be accessible via JavaScript, and there is already
+a view that does this for related flags: `update_realm`. So in this
+case, we can add out code to the existing view instead of creating a
+new one. :
+
+    # zerver/views/__init__.py
+
+    def home(request):
+      # ...
+      page_params = dict(
+        # ...
+        realm_invite_by_admins_only = register_ret['realm_invite_by_admins_only'],
+        # ...
+      )
+
+Since this feature also adds a checkbox to the admin page, and adds a
+new property the Realm model that can be modified from there, you also
+need to make changes to the `update_realm` function in the same file: :
+
+    # zerver/views/__init__.py
+
+    def update_realm(request, user_profile,
+      name=REQ(validator=check_string, default=None),
+      restricted_to_domain=REQ(validator=check_bool, default=None),
+      invite_by_admins_only=REQ(validator=check_bool,default=None)):
+
+      # ...
+
+      if invite_by_admins_only is not None and
+        realm.invite_by_admins_only != invite_by_admins_only:
+          do_set_realm_invite_by_admins_only(realm, invite_by_admins_only)
+          data['invite_by_admins_only'] = invite_by_admins_only
+
+Then make the required front end changes: in this case a checkbox needs
+to be added to the admin page (and its value added to the data sent back
+to server when a realm is updated) and the change event needs to be
+handled on the client.
+
+To add the checkbox to the admin page, modify the relevant template,
+`static/templates/admin_tab.handlebars` (omitted here since it is
+relatively straightforward). Then add code to handle changes to the new
+form control in `static/js/admin.js`. :
+
+    var url = "/json/realm";
+    var new_invite_by_admins_only =
+      $("#id_realm_invite_by_admins_only").prop("checked");
+    data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);
+
+    channel.patch({
+      url: url,
+      data: data,
+      success: function (data) {
+        # ...
+        if (data.invite_by_admins_only) {
+          ui.report_success("New users must be invited by an admin!", invite_by_admins_only_status);
+        } else {
+          ui.report_success("Any user may now invite new users!", invite_by_admins_only_status);
+        }
+        # ...
+      }
+    });
+
+Finally, update `server_events.js` to handle related events coming from
+the server. :
+
+    # static/js/server_events.js
+
+    function get_events_success(events) {
+      # ...
+      var dispatch_event = function dispatch_event(event) {
+          switch (event.type) {
+          # ...
+          case 'realm':
+            if (event.op === 'update' && event.property === 'invite_by_admins_only') {
+              page_params.realm_invite_by_admins_only = event.value;
+          }
+      }
+    }
+
+Any code needed to update the UI should be placed in `dispatch_event`
+callback (rather than the `channel.patch`) function. This ensures the
+appropriate code will run even if the changes are made in another
+browser window. In this example most of the changes are on the backend,
+so no UI updates are required.

docs/new-feature-tutorial.rst

@@ -1,85 +0,0 @@
-====================
-New Feature Tutorial
-====================
-
-.. attention::
-   This tutorial is an unfinished work -- contributions welcome!
-
-The changes needed to add a new feature will vary, of course.  We give an
-example here that illustrates some of the common steps needed.  We describe
-the process of adding a new setting for admins that restricts inviting new
-users to admins only.
-
-Backend Changes
-===============
-
-Adding a field to the database
-------------------------------
-
-The server accesses the underlying database in `zerver/models.py`.  Add
-a new field in the appropriate class, `realm_invite_by_admins_only`
-in the `Realm` class in this case.
-
-Once you do so, you need to create the migration and run it; the
-process is documented at:
-https://docs.djangoproject.com/en/1.8/topics/migrations/
-
-Once you've run the migration, to test your changes, you'll want to
-restart memcached on your development server (``/etc/init.d/memcached restart``) and
-then restart ``run-dev.py`` to avoid interacting with cached objects.
-
-Backend changes
----------------
-
-You should add code in `zerver/lib/actions.py` to interact with the database,
-that actually updates the relevant field.  In this case, `do_set_realm_invite_by_admins_only`
-is a function that actually updates the field in the database, and sends
-an event announcing that this change has been made.
-
-You then need update the `fetch_initial_state_data` and `apply_events` functions
-in `zerver/lib/actions.py` to update the state based on the event you just created.
-In this case, we add a line
-
-::
-
-  state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`
-
-to the `fetch_initial_state_data` function.  The `apply_events` function
-doesn't need to be updated since
-
-::
-
-   elif event['type'] == 'realm':
-       field = 'realm_' + event['property']
-       state[field] = event['value']
-
-already took care of our event.
-
-Then update `zerver/views/__init__.py` to actually call your function.
-In the dictionary which sets the javascript `page_params` dictionary,
-add a value for your feature.
-
-::
-
-   realm_invite_by_admins_only = register_ret['realm_invite_by_admins_only']
-
-Perhaps your new option controls some other backend rendering: in our case
-we test for this option in the `home` method for adding a variable to the response.
-The functions in this file control the generation of various pages served
-(along with the Django templates).
-Our new feature also shows up in the administration tab (as a checkbox),
-so we need to update the `update_realm` function.
-
-
-Finally, add tests for your backend changes; at the very least you
-should add a test of your event data flowing through the system in
-``test_events.py``.
-
-
-Frontend changes
-----------------
-
-You need to change various things on the front end.  In this case, the relevant files
-are `static/js/server_events.js`, `static/js/admin.js`, `static/styles/zulip.css
-and `static/templates/admin_tab.handlebars`.
-

docs/queuing.md

@@ -0,0 +1,81 @@
+# RabbitMQ queues
+
+Zulip uses RabbitMQ to manage a system of internal queues.  These are
+used for a variety of purposes:
+
+* Asynchronously doing expensive operations like sending email
+  notifications which can take seconds per email and thus would
+  otherwise timeout when 100s are triggered at once (E.g. inviting a
+  lot of new users to a realm).
+
+* Asynchronously doing non-time-critical somewhat expensive operations
+  like updating analytics tables (e.g. UserActivityInternal) which
+  don't have any immediate runtime effect.
+
+* Communicating events to push to clients (browsers, etc.) from the
+  main Zulip Django application process to the Tornado-based events
+  system.  Example events might be that a new message was sent, a user
+  has changed their subscriptions, etc.
+
+* Processing mobile push notifications and email mirroring system
+  messages.
+
+* Processing various errors, frontend tracebacks, and slow database
+  queries in a batched fashion.
+
+* Doing markdown rendering for messages delivered to the Tornado via
+  websockets.
+
+Needless to say, the RabbitMQ-based queuing system is an important
+part of the overall Zulip architecture, since it's in critical code
+paths for everything from signing up for account, to rendering
+messages, to delivering updates to clients.
+
+We use the `pika` library to interface with RabbitMQ, using a simple
+custom integration defined in `zerver/lib/queue.py`.
+
+### Adding a new queue processor
+
+To add a new queue processor:
+
+* Define the processor in `zerver/worker/queue_processors.py` using
+  the `@assign_queue` decorator; it's pretty easy to get the template
+  for an existing similar queue processor.  This suffices to test your
+  queue worker in the Zulip development environment, though you'll
+  need to restart `tools/run-dev.py` in order to run your new queue
+  processor.  You can also run a single queue processor manually using
+  e.g. `./manage.py process_queue --queue=user_activity`.
+
+* So that supervisord will known to run the queue processor in
+  production, you will need to define a program entry for it in
+  `servers/puppet/modules/zulip/files/supervisor/conf.d/zulip.conf`
+  and add it to the `zulip-workers` group further down in the file.
+
+* For monitoring, you need to add a check that your worker is running
+  to puppet/zulip/files/cron.d/rabbitmq-numconsumers if it's a
+  one-at-a-time consumer like `user_activity_internal` or a custom
+  nagios check if it is a bulk processor like `slow_queries`.
+
+### Publishing events into a queue
+
+You can publish events to a RabbitMQ queue using the
+`queue_json_publish` function defined in `zerver/lib/queue.py`.
+
+### Clearing a RabbitMQ queue
+
+If you need to clear a queue (delete all the events in it), run
+`./manage.py purge_queue <queue_name>`, for example:
+
+```
+./manage.py purge_queue user_activity
+```
+
+You can also use the amqp tools directly.  Install `amqp-tools` from
+apt and then run:
+
+```
+amqp-delete-queue --username=zulip --password='...' --server=localhost \
+   --queue=user_presence
+```
+
+with the RabbitMQ password from `/etc/zulip/zulip-secrets.conf`.

docs/readme-symlink.md

@@ -0,0 +1 @@
+../README.md

docs/requirements.readthedocs.txt

@@ -1 +0,0 @@
-# Empty requirements.txt to avoid readthedocs installing all our dependencies.

docs/requirements.readthedocs.txt

@@ -0,0 +1 @@
+../requirements/docs.txt

docs/roadmap.md

@@ -0,0 +1,262 @@
+Zulip 2016 Roadmap
+==================
+
+## Introduction
+
+Zulip has received a great deal of interest and attention since it was
+released as free and open source software by Dropbox.  That attention
+has come with a lot of active development work from members of the
+Zulip community.  From when Zulip was released as open source in late
+September 2015 through today (mid-April, 2016), over 300 pull requests
+have been submitted to the various Zulip repositories (and over 250
+have been merged!), the vast majority of which are submitted by
+Zulip's users around the world (as opposed to the small core team who
+review and merge the pull requests).
+
+In any project, there can be a lot of value in periodically putting
+together a roadmap detailing the major areas where the project is
+hoping to improve.  This can be especially important in an open source
+project like Zulip where development is distributed across many people
+around the world.  This roadmap is intended to organize a list of the
+most important improvements that should be made to Zulip in the
+relatively near future.  Our aim is to complete most of these
+improvements in 2016.
+
+This document is not meant to constrain in any way what contributions
+to Zulip will be accepted; instead, it will be used by the Zulip core
+team to prioritize our efforts, measure progress on improving the
+Zulip product, hold ourselves accountable for making Zulip improve
+rapidly, and celebrate members of the community who contribute to
+projects on the roadmap.
+
+If you're someone interested in making a larger contribution to Zulip
+and looking for somewhere to start, this roadmap is the best place to
+look for substantial projects that will definitely be of value to the
+community (if you're looking for a starter project, see the [guide to
+getting involved with
+Zulip](https://github.com/zulip/zulip#how-to-get-involved-with-contributing-to-zulip)).
+
+Without further ado, below is the Zulip 2016 roadmap.
+
+## Burning problems
+
+The top problem for the Zulip project is the state of the mobile apps.
+The Android app has started seeing rapid progress thanks to a series
+of contributions by Lisa Neigut of Recurse Center, and we believe to
+be on a good path.  The iOS app has fewer features than Android and
+has more bugs, but more importantly is in need of an experienced iOS
+developer who has time to drive the project.
+
+## Core User Experience
+
+This category includes important improvements to the core user
+experience that will benefit all users.
+
+* [Improve missed message notifications to make "reply" work nicely](https://github.com/zulip/zulip/issues/612)
+* [Add support for showing "user is typing" notifications](https://github.com/zulip/zulip/issues/150)
+* [Add pretty bubbles for recipients in the compose box](https://github.com/zulip/zulip/issues/595)
+* [Finish and merge support for pinning a few important streams](https://github.com/zulip/zulip/issues/285)
+* [Display stream descriptions more prominently](https://github.com/zulip/zulip/issues/164)
+* [Integration inline URL previews](https://github.com/zulip/zulip/issues/406)
+* [Add support for managing uploaded files](https://github.com/zulip/zulip/issues/454)
+* [Make Zulip onboarding experience smoother for teams not used to topics](https://github.com/zulip/zulip/issues/647).  That specific proposal might not be right but the issue is worth investing time in.
+
+## Ease of setup and onboarding issues
+
+This category focuses on issues users experience when installing a new
+Zulip server or setting up a new Zulip realm.
+
+* [Create a web flow for setting up a new realm / the first realm on a new server (currently, it's a command-line process)](https://github.com/zulip/zulip/issues/260)
+* [Document or better script solution to rabbitmq startup issues](https://github.com/zulip/zulip/issues/465)
+* [Add a mechanism for deleting early test messages](https://github.com/zulip/zulip/issues/135)
+* [Merge a supported way to use Zulip in Docker in production
+  implementation](https://github.com/zulip/zulip/pull/450).
+
+## Internationalization
+
+The core Zulip UI has been mostly translated into 5 languages;
+however, more work is required to make those translations actually
+displayed in the Zulip UI for the users who would benefit from them.
+
+* [Merge support for using translations in Django templates](https://github.com/zulip/zulip/pull/607)
+* [Add text in handlebars templates to translatable string database](https://github.com/zulip/zulip/issues/726)
+* [Merge support for translating text in handlebars](https://github.com/zulip/zulip/issues/726)
+* [Add text in error messages to translatable strings](https://github.com/zulip/zulip/issues/727)
+
+## User Experience at scale
+
+There are a few parts of the Zulip UI which could benefit from
+overhauls designed around making the user experience nice for large
+teams.
+
+* [Make the buddy list work better for large teams](https://github.com/zulip/zulip/issues/236)
+* [Improve @-mentioning syntax based on stronger unique identifiers](https://github.com/zulip/zulip/issues/374)
+* [Show subscriber counts on streams](https://github.com/zulip/zulip/pull/525)
+* [Make the streams page easier to navigate with 100s of streams](https://github.com/zulip/zulip/issues/563)
+* [Add support for filtering long lists of streams](https://github.com/zulip/zulip/issues/565)
+
+## Administration and management
+
+Currently, Zulip has a number of administration features that can be
+controlled only via the command line.
+
+* [Make default streams web-configurable](https://github.com/zulip/zulip/issues/665)
+* [Make realm emoji web-configurable](https://github.com/zulip/zulip/pull/543)
+* [Make realm filters web-configurable](https://github.com/zulip/zulip/pull/544)
+* [Make realm aliases web-configurable](https://github.com/zulip/zulip/pull/651)
+* [Enhance the LDAP integration and make it web-configurable](https://github.com/zulip/zulip/issues/715)
+* [Add a SAML integration for Zulip](https://github.com/zulip/zulip/issues/716)
+* [Improve administrative controls for managing streams](https://github.com/zulip/zulip/issues/425)
+
+## Scalability
+
+Zulip should support 10000 users in a realm and also support smaller
+realms in more resource-constrained environments (probably a good
+initial goal is working well with only 2GB of RAM).
+
+* [Make the Zulip Tornado service support horizontal scaling](https://github.com/zulip/zulip/issues/445)
+* [Make presence system scale well to 10000 users in a realm.](https://github.com/zulip/zulip/issues/728)
+* [Support running queue workers multithreaded in production to
+  decrease minimum memory footprint](https://github.com/zulip/zulip/issues/34)
+
+## Performance
+
+Performance is essential for a communication tool.  While some things
+are already quite good (E.g. narrowing and message sending is speedy),
+this is an area where one can always improve.  There are a few known
+performance opportunities:
+
+* [Migrate to faster jinja2 templating engine](https://github.com/zulip/zulip/issues/620)
+* [Don't load zxcvbn when it isn't needed](https://github.com/zulip/zulip/issues/263)
+* [Optimize the frontend performance of loading the Zulip webapp using profiling](https://github.com/zulip/zulip/issues/714)
+
+## Technology improvements
+
+Zulip should be making use of the best Python/Django tools available.
+
+* [Add support for Zulip running on Python 3](https://github.com/zulip/zulip/issues/256)
+* [Add support for changing users' email addresses](https://github.com/zulip/zulip/issues/734)
+* [Automatic thumbnailing of uploaded images](https://github.com/zulip/zulip/issues/432)
+* [Upgrade Zulip to use Django 1.10 once it is released.  The patches
+  needed to run Zulip were merged into mainline Django in Django 1.10,
+  so this will mean we don't need to use a fork of Django anymore.](https://github.com/zulip/zulip/issues/3)
+
+## Technical Debt
+
+While the Zulip server has a great codebase compared to most projects
+of its size, it takes work to keep it that way.
+
+* [Migrate most web routes to REST API](https://github.com/zulip/zulip/issues/611)
+* [Finish purging global variables from the Zulip javascript](https://github.com/zulip/zulip/issues/610)
+* [Finish deprecating and remove the pre-REST Zulip /send_message API](https://github.com/zulip/zulip/issues/730)
+* [Split Tornado subsystem into a separate Django app](https://github.com/zulip/zulip/issues/729)
+* [Clean up clutter in the root of the zulip.git repository](https://github.com/zulip/zulip/issues/707)
+* [Refactor zulip.css to be broken into components](https://github.com/zulip/zulip/issues/731)
+
+## Deployment and upgrade process
+
+* [Support backwards-incompatible upgrades to Python libraries](https://github.com/zulip/zulip/issues/717)
+* [Minimize the downtime required in Zulip upgrade process](https://github.com/zulip/zulip/issues/646)
+
+## Security
+
+* [Add support for 2-factor authentication on all platforms](https://github.com/zulip/zulip/pull/451)
+* [Add a retention policy feature that automatically deletes old messages](https://github.com/zulip/zulip/issues/106)
+* [Upgrade every Zulip dependency to a modern version](https://github.com/zulip/zulip/issues/717)
+* [The LOCAL_UPLOADS_DIR file uploads backend only supports world-readable uploads](https://github.com/zulip/zulip/issues/320)
+* [Add support for stronger security controls for uploaded files](https://github.com/zulip/zulip/issues/320)
+
+## Testing
+
+* [Extend Zulip's automated test coverage to include all API endpoints](https://github.com/zulip/zulip/issues/732)
+* [Build automated tests for the client API bindings](https://github.com/zulip/zulip/issues/713)
+* [Add Python static type-checking to Zulip using mypy](https://github.com/zulip/zulip/issues/733)
+* [Improve the runtime of Zulip's backend test suite](https://github.com/zulip/zulip/issues/441)
+* [Use caching to make Travis CI runtimes faster](https://github.com/zulip/zulip/issues/712)
+* [Add automated tests for the production upgrade process](https://github.com/zulip/zulip/issues/306)
+* [Improve Travis CI "production" test suite to catch more regressions](https://github.com/zulip/zulip/issues/598)
+
+## Development environment
+
+* [Migrate from jslint to eslint](https://github.com/zulip/zulip/issues/535)
+* [Figure out a nice upgrade process for Zulip Vagrant VMs](https://github.com/zulip/zulip/issues/264)
+* [Overhaul new contributor documentation](https://github.com/zulip/zulip/issues/677)
+* [Replace closure-compiler with a faster minifier toolchain](https://github.com/zulip/zulip/issues/693)
+* [Add support for building frontend features in React](https://github.com/zulip/zulip/issues/694)
+* [Use a javascript bundler like webpack](https://github.com/zulip/zulip/issues/695)
+
+## Documentation
+
+* [Significantly expand documentation of the Zulip API and integrating
+  with Zulip.](https://github.com/zulip/zulip/issues/672)
+* [Expand library of documentation on Zulip's feature set.  Currently
+  most documentation is for either developers or system administrators.](https://github.com/zulip/zulip/issues/675)
+* [Expand developer documentation with more tutorials explaining how to do
+  various types of projects.](https://github.com/zulip/zulip/issues/676)
+* [Overhaul new contributor documentation, especially on coding style,
+  to better highlight and teach the important pieces.](https://github.com/zulip/zulip/issues/677)
+* [Update all screenshots to show the current Zulip UI](https://github.com/zulip/zulip/issues/599)
+
+## Integrations
+
+Integrations are essential to Zulip.  While we currently have a
+reasonably good framework for writing new webhook integrations for
+getting notifications into Zulip, it'd be great to streamline that
+process and make bots that receive messages just as easy to build.
+
+* [Make it super easy to take screenshots for new webhook integrations](https://github.com/zulip/zulip/issues/658)
+* [Add an outgoing webhook integration system](https://github.com/zulip/zulip/issues/735)
+* [Build a framework to cut duplicated code in new webhook integrations](https://github.com/zulip/zulip/issues/660)
+* [Make setting up a new integration a smooth flow](https://github.com/zulip/zulip/issues/692)
+* [Optimize the integration writing documentation to make writing new
+   ones really easy.](https://github.com/zulip/zulip/issues/70)
+
+## Android app
+
+The Zulip Android app is ahead of the iOS app in terms of feature set,
+so this section serves to document the goals for Zulip on mobile.
+
+* [Support using a non-zulip.com server](https://github.com/zulip/zulip-android/issues/1)
+* [Support Google authentication with a non-Zulip.com server](https://github.com/zulip/zulip-android/issues/49)
+* [Add support for narrowing to @-mentions](https://github.com/zulip/zulip-android/issues/39)
+* [Support having multiple Zulip realms open simultaneously](https://github.com/zulip/zulip-android/issues/47)
+* [Build a slick development login page to simplify testing (similar to
+  the development homepage on web)](https://github.com/zulip/zulip-android/issues/48)
+* [Improve the compose box to let you see what you're replying to](https://github.com/zulip/zulip-android/issues/8)
+* [Make it easy to compose messages with mentions, emoji, etc.](https://github.com/zulip/zulip-android/issues/11)
+* [Display unread counts and improve navigation](https://github.com/zulip/zulip-android/issues/57)
+* [Hide messages sent to muted topics](https://github.com/zulip/zulip-android/issues/9)
+* [Fill out documentation to make it easy to get started](https://github.com/zulip/zulip-android/issues/58)
+
+## iOS app
+
+Most of the projects listed under Android apply here as well, but it's
+worth highlighting some areas where iOS is substantially behind
+Android.  The top priority here is recruiting a lead developer for the
+iOS app.  Once we have that resolved, we'll expand our ambitions for
+the app with more specific improvements.
+
+* [iOS app needs maintainer](https://github.com/zulip/zulip-ios/issues/12)
+* [APNS notifications are broken](https://github.com/zulip/zulip/issues/538)
+
+## Desktop apps
+
+The top goal for the desktop apps is to rebuild it in modern toolchain
+(probably Electron) so that it's easy for a wide range of developers
+to contribute to the apps.
+
+* Migrate platform from QT/webkit to Electron
+* Desktop app doesn't recover well from entering the wrong Zulip server
+* Support having multiple Zulip realms open simultaneously
+* Build an efficient process for testing and releasing new versions of
+  the desktop apps
+
+## Community
+
+These don't get GitHub issues since they're not technical projects,
+but they are important goals for the project.
+
+* Setup a Zulip server for the Zulip development community
+* Expand the number of core developers able to do code reviews
+* Expand the number of contributors regularly adding features to Zulip
+* Have a successful summer with Zulip's 3 GSOC students

docs/schema-migrations.md

@@ -0,0 +1,23 @@
+# Schema Migrations
+
+Zulip uses the [standard Django system for doing schema
+migrations](https://docs.djangoproject.com/en/1.8/topics/migrations/).
+There is some example usage in the Zulip new feature tutorial on
+readthedocs.
+
+This page documents some important issues related to writing schema
+migrations.
+
+* **Large tables**: For large tables like Message and UserMessage, you
+  want to take precautions when adding columns to the table,
+  performing data backfills, or building indexes. We have a
+  `zerver/lib/migrate.py` library to help with adding columns and
+  backfilling data. For building indexes on these tables, we should do
+  this using SQL with postgres's CONCURRENTLY keyword.
+
+* **Numbering conflicts across branches**: If you've done your schema
+  change in a branch, and meanwhile another schema change has taken
+  place, Django will now have two migrations with the same number. To
+  fix this, you can just rename the file, as long as no other
+  migrations depend on it (in which case you also need to update the
+  dependencies).

docs/settings.md

@@ -0,0 +1,127 @@
+# Zulip settings
+
+The page documents the Zulip settings system, and hopefully should
+help you decide how to correctly implement new settings you're adding
+to Zulip.  We have two types of administrative settings in Zulip:
+server settings (which are set via configuration files are apply to
+the whole Zulip installation), and realm settings (which are usually
+set via the /#administration page in the Zulip web application) and
+apply to a single Zulip realm/organization (which for most Zulip
+servers is the only realm on the server).
+
+## Server settings
+
+Zulip uses the [Django settings
+system](https://docs.djangoproject.com/en/1.9/topics/settings/), which
+means that the settings files are Python programs that set a lot of
+variables with all-capital names like EMAIL_GATEWAY_PATTERN.  You can
+access these anywhere in the Zulip Django code using e.g.:
+
+```
+from django.conf import settings
+print(settings.EMAIL_GATEWAY_PATTERN)
+```
+
+Additionally, if you need to access a Django setting in a shell
+script (or just on the command line for debugging), you can use e.g.:
+
+```
+$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN
+%s@localhost:9991
+```
+
+Zulip has separated those settings that we expect a system
+administrator to change (with nice documentation) from the ~1000 lines
+of settings needed by the Zulip Django app.  As a result, there are a
+few files involved in the Zulip settings for server administrations.
+In a production environment, we have:
+
+* `/etc/zulip/settings.py` (generated from
+  `zproject/local_settings_template.py`) is the main system
+  administration facing settings file for Zulip.  It contains all the
+  server-specific settings, such as how to send outgoing email, the
+  hostname of the Postgres database, etc., but does not contain any
+  secrets (e.g. passwords, secret API keys, cryptographic keys, etc.).
+  The way we generally do settings that can be controlled with shell
+  access to a Zulip server is to put a default in
+  `zproject/settings.py`, and then override it here.
+
+* `/etc/zulip/zulip-secrets.conf` (generated by
+  `scripts/setup/generate-secrets.py` as part of installation)
+  contains secrets used by the Zulip installation.  These are read
+  using the standard Python `ConfigParser`, and accessed in
+  `zproject/settings.py` by the `get_secret` function.
+
+* `zproject/settings.py` is the main Django settings file for Zulip.
+  It contains all the settings that are constant for all Zulip
+  installations (e.g. configuration for logging, static assets,
+  middleware, etc.), as well as default values for the settings the
+  user would set in `/etc/zulip/settings.py` (you can look at the
+  `DEFAULT_SETTINGS` dictionary to easily review the settings
+  available).  `zproject/settings.py` has a line `from local_settings
+  import *`, which has the effect of importing
+  `/etc/zulip/settings.py`.
+
+In a development environment, we have `zproject/settings.py`, and
+additionally:
+
+* `zproject/dev_settings.py` has the settings for the Zulip development
+  environment; it mostly just imports local_settings_template.py.
+
+* `zproject/dev-secrets.conf` replaces `/etc/zulip/zulip-secrets.conf`.
+
+* `zproject/test_settings.py` has the (default) settings used for the
+  Zulip tests (both backend and Casper), which are applied on top of
+  the development environment settings.
+
+When adding a new server setting to Zulip, you will typically add it
+in two or three places:
+
+* In DEFAULT_SETTINGS in `zproject/settings.py`, with a default value
+  for production environments.
+
+* In an appropriate section of `zproject/local_settings_template.py`,
+  with documentation in the comments explaining the settings's
+  purpose and effect.
+
+* Possibly also `zproject/dev_settings.py`, if the desired value of
+  the setting for Zulip development environments is different from the
+  default for production (and similarly for `zproject/test_settings.py`).
+
+Most settings should be enabled in the development environment, to
+maximize convenience of testing all of Zulip's features; they should
+be enabled by default in production if we expect most Zulip sites to
+want those settings.
+
+#### Testing non-default settings
+
+You can write tests for settings using e.g. `with
+self.settings(GOOGLE_CLIENT_ID=None)`.  However, this only works for
+settings which are checked at runtime, not settings which are only
+accessed in initialization of Django (or Zulip) internals
+(e.g. `DATABASES`).  See the [Django docs on overriding settings in
+tests][django-test-settings] for more details.
+
+[django-test-settings]: https://docs.djangoproject.com/en/1.9/topics/testing/tools/#overriding-settings
+
+## Realm settings
+
+Realm settings are preferred for any configuration that is a matter of
+organizational policy (as opposed to technical capabilities of the
+server).  As a result, configuration options for user-facing
+functionality is almost always added as a new realm setting, not a
+server setting.  The [new feature tutorial][doc-newfeat] documents the
+process for adding a new realm setting to Zulip.
+
+So for example, the following server settings will eventually be
+replaced with realm settings:
+
+* NAME_CHANGES_DISABLED
+* INLINE_IMAGE_PREVIEW
+* ENABLE_GRAVATAR
+* Which authentication methods are allowed should probably appear in
+  both places; in server settings indicating the capabilities of the
+  server, and in the realm settings indicating which methods the realm
+  administrator wants to allow users to login with.
+
+[doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html

docs/testing.md

@@ -0,0 +1,329 @@
+Testing and writing tests
+=========================
+
+Running tests
+-------------
+
+To run everything, just use `./tools/test-all`. This runs lint checks,
+web frontend / whole-system blackbox tests, and backend Django tests.
+
+If you want to run individual parts, see the various commands inside
+that script.
+
+### Schema and initial data changes
+
+If you change the database schema or change the initial test data, you
+have to regenerate the pristine test database by running
+`tools/do-destroy-rebuild-test-database`.
+
+### Wiping the test databases
+
+You should first try running: `tools/do-destroy-rebuild-test-database`
+
+If that fails you should try to do:
+
+    sudo -u postgres psql
+    > DROP DATABASE zulip_test;
+    > DROP DATABASE zulip_test_template;
+
+and then run `tools/do-destroy-rebuild-test-database`
+
+#### Recreating the postgres cluster
+
+> **warning**
+>
+> **This is irreversible, so do it with care, and never do this anywhere
+> in production.**
+
+If your postgres cluster (collection of databases) gets totally trashed
+permissions-wise, and you can't otherwise repair it, you can recreate
+it. On Ubuntu:
+
+    sudo pg_dropcluster --stop 9.1 main
+    sudo pg_createcluster --locale=en_US.utf8 --start 9.1 main
+
+### Backend Django tests
+
+These live in `zerver/tests/tests.py` and `zerver/tests/test_*.py`. Run
+them with `tools/test-backend`.
+
+### Web frontend black-box casperjs tests
+
+These live in `frontend_tests/casper_tests/`. This is a "black box"
+test; we load the frontend in a real (headless) browser, from a real dev
+server, and simulate UI interactions like sending messages, narrowing,
+etc.
+
+Since this is interacting with a real dev server, it can catch backend
+bugs as well.
+
+You can run this with `./tools/test-js-with-casper` or as
+`./tools/test-js-with-casper 05-settings.js` to run a single test file
+from `frontend_tests/casper_tests/`.
+
+#### Debugging Casper.JS
+
+Casper.js (via PhantomJS) has support for remote debugging. However, it
+is not perfect. Here are some steps for using it and gotchas you might
+want to know.
+
+To turn on remote debugging, pass `--remote-debug` to the
+`./frontend_tests/run-casper` script. This will run the tests with port
+`7777` open for remote debugging. You can now connect to
+`localhost:7777` in a Webkit browser. Somewhat recent versions of Chrome
+or Safari might be required.
+
+-   When connecting to the remote debugger, you will see a list of
+    pages, probably 2. One page called `about:blank` is the headless
+    page in which the CasperJS test itself is actually running in. This
+    is where your test code is.
+-   The other page, probably `localhost:9981`, is the Zulip page that
+    the test is testing---that is, the page running our app that our
+    test is exercising.
+
+Since the tests are now running, you can open the `about:blank` page,
+switch to the Scripts tab, and open the running `0x-foo.js` test. If you
+set a breakpoint and it is hit, the inspector will pause and you can do
+your normal JS debugging. You can also put breakpoints in the Zulip
+webpage itself if you wish to inspect the state of the Zulip frontend.
+
+You can also check the screenshots of failed tests at `/tmp/casper-failure*.png`.
+
+If you need to use print debugging in casper, you can do using
+`casper.log`; see <http://docs.casperjs.org/en/latest/logging.html> for
+details.
+
+An additional debugging technique is to enable verbose mode in the
+Casper tests; you can do this by adding to the top of the relevant test
+file the following:
+
+>     var casper = require('casper').create({
+>        verbose: true,
+>        logLevel: "debug"
+>     });
+
+This can sometimes give insight into exactly what's happening.
+
+### Web frontend unit tests
+
+As an alternative to the black-box whole-app testing, you can unit test
+individual JavaScript files that use the module pattern. For example, to
+test the `foobar.js` file, you would first add the following to the
+bottom of `foobar.js`:
+
+>     if (typeof module !== 'undefined') {
+>         module.exports = foobar;
+>     }
+
+This makes `foobar.js` follow the CommonJS module pattern, so it can be
+required in Node.js, which runs our tests.
+
+Now create `frontend_tests/node_tests/foobar.js`. At the top, require
+the [Node.js assert module](http://nodejs.org/api/assert.html), and the
+module you're testing, like so:
+
+>     var assert = require('assert');
+>     var foobar = require('js/foobar.js');
+
+(If the module you're testing depends on other modules, or modifies
+global state, you need to also read [the next
+section](handling-dependencies_).)
+
+Define and call some tests using the [assert
+module](http://nodejs.org/api/assert.html). Note that for "equal"
+asserts, the *actual* value comes first, the *expected* value second.
+
+>     (function test_somefeature() {
+>         assert.strictEqual(foobar.somefeature('baz'), 'quux');
+>         assert.throws(foobar.somefeature('Invalid Input'));
+>     }());
+
+The test runner (index.js) automatically runs all .js files in the
+frontend\_tests/node directory.
+
+#### Coverage reports
+
+You can automatically generate coverage reports for the JavaScript unit
+tests. To do so, install istanbul:
+
+>     sudo npm install -g istanbul
+
+And run test-js-with-node with the 'cover' parameter:
+
+>     tools/test-js-with-node cover
+
+Then open `coverage/lcov-report/js/index.html` in your browser. Modules
+we don't test *at all* aren't listed in the report, so this tends to
+overstate how good our overall coverage is, but it's accurate for
+individual files. You can also click a filename to see the specific
+statements and branches not tested. 100% branch coverage isn't
+necessarily possible, but getting to at least 80% branch coverage is a
+good goal.
+
+Writing tests
+-------------
+
+### Writing Casper tests
+
+Probably the easiest way to learn how to write Casper tests is to study
+some of the existing test files. There are a few tips that can be useful
+for writing Casper tests in addition to the debugging notes below:
+
+-   Run just the file containing your new tests as described above to
+    have a fast debugging cycle.
+-   With frontend tests in general, it's very important to write your
+    code to wait for the right events. Before essentially every action
+    you take on the page, you'll want to use `waitForSelector`,
+    `waitUntilVisible`, or a similar function to make sure the page or
+    elemant is ready before you interact with it. For instance, if you
+    want to click a button that you can select via `#btn-submit`, and
+    then check that it causes `success-elt` to appear, you'll want to
+    write something like:
+
+        casper.waitForSelector("#btn-submit", function () {
+           casper.click('#btn-submit')
+           casper.test.assertExists("#success-elt");
+         });
+
+    This will ensure that the element is present before the interaction
+    is attempted. The various wait functions supported in Casper are
+    documented in the Casper here:
+    <http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector>
+    and the various assert statements available are documented here:
+    <http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype>
+-   Casper uses CSS3 selectors; you can often save time by testing and
+    debugging your selectors on the relevant page of the Zulip
+    development app in the Chrome javascript console by using e.g.
+    `$$("#settings-dropdown")`.
+-   The test suite uses a smaller set of default user accounts and other
+    data initialized in the database than the development environment;
+    to see what differs check out the section related to
+    `options["test_suite"]` in
+    `zilencer/management/commands/populate_db.py`.
+-   Casper effectively runs your test file in two phases -- first it
+    runs the code in the test file, which for most test files will just
+    collect a series of steps (each being a `casper.then` or
+    `casper.wait...` call). Then, usually at the end of the test file,
+    you'll have a `casper.run` call which actually runs that series of
+    steps. This means that if you write code in your test file outside a
+    `casper.then` or `casper.wait...` method, it will actually run
+    before all the Casper test steps that are declared in the file,
+    which can lead to confusing failures where the new code you write in
+    between two `casper.then` blocks actually runs before either of
+    them. See this for more details about how Casper works:
+    <http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work>
+
+### Handling dependencies in unit tests
+
+The following scheme helps avoid tests leaking globals between each
+other.
+
+First, if you can avoid globals, do it, and the code that is directly
+under test can simply be handled like this:
+
+>     var search = require('js/search_suggestion.js');
+
+For deeper dependencies, you want to categorize each module as follows:
+
+-   Exercise the module's real code for deeper, more realistic testing?
+-   Stub out the module's interface for more control, speed, and
+    isolation?
+-   Do some combination of the above?
+
+For all the modules where you want to run actual code, add a statement
+like the following to the top of your test file:
+
+>     add_dependencies({
+>         _: 'third/underscore/underscore.js',
+>         util: 'js/util.js',
+>         Dict: 'js/dict.js',
+>         Handlebars: 'handlebars',
+>         Filter: 'js/filter.js',
+>         typeahead_helper: 'js/typeahead_helper.js',
+>         stream_data: 'js/stream_data.js',
+>         narrow: 'js/narrow.js'
+>     });
+
+For modules that you want to completely stub out, please use a pattern
+like this:
+
+>     set_global('page_params', {
+>         email: 'bob@zulip.com'
+>     });
+>
+>     // then maybe further down
+>     global.page_params.email = 'alice@zulip.com';
+
+Finally, there's the hybrid situation, where you want to borrow some of
+a module's real functionality but stub out other pieces. Obviously, this
+is a pretty strong smell that the other module might be lacking in
+cohesion, but that code might be outside your jurisdiction. The pattern
+here is this:
+
+>     // Use real versions of parse/unparse
+>     var narrow = require('js/narrow.js');
+>     set_global('narrow', {
+>         parse: narrow.parse,
+>         unparse: narrow.unparse
+>     });
+>
+>     // But later, I want to stub the stream without having to call super-expensive
+>     // real code like narrow.activate().
+>     global.narrow.stream = function () {
+>         return 'office';
+>     };
+
+Manual testing (local app + web browser)
+----------------------------------------
+
+### Clearing the manual testing database
+
+You can use:
+
+    ./tools/do-destroy-rebuild-database
+
+to drop the database on your development environment and repopulate
+your it with the Shakespeare characters and some test messages between
+them.  This is run automatically as part of the development
+environment setup process, but is occasionally useful when you want to
+return to a clean state for testing.
+
+### JavaScript manual testing
+
+debug.js has some tools for profiling Javascript code, including:
+
+-   \`print\_elapsed\_time\`: Wrap a function with it to print the time
+    that function takes to the javascript console.
+-   \`IterationProfiler\`: Profile part of looping constructs (like a
+    for loop or \$.each). You mark sections of the iteration body and
+    the IterationProfiler will sum the costs of those sections over all
+    iterations.
+
+Chrome has a very good debugger and inspector in its developer tools.
+Firebug for Firefox is also pretty good. They both have profilers, but
+Chrome's is a sampling profiler while Firebug's is an instrumenting
+profiler. Using them both can be helpful because they provide different
+information.
+
+Python 3 Compatibility
+----------------------
+
+Zulip is working on supporting Python 3, and all new code in Zulip
+should be Python 2+3 compatible. We have converted most of the codebase
+to be compatible with Python 3 using a suite of 2to3 conversion tools
+and some manual work. In order to avoid regressions in that
+compatibility as we continue to develop new features in zulip, we have a
+special tool, tools/check-py3, which checks all code for Python 3
+syntactic compatibility by running a subset of the automated migration
+tools and checking if they trigger any changes. tools/check-py3 is run
+automatically in Zulip's Travis CI tests to avoid any regressions, but
+is not included in test-all since it is quite slow.
+
+To run tooks/check-py3, you need to install the modernize and future
+python packages (which are included in requirements/py3k.txt, which
+itself is included in requirements/dev.txt, so you probably already
+have these packages installed).
+
+To run check-py3 on just the python files in a particular directory, you
+can change the current working directory (e.g. cd zerver/) and run
+check-py3 from there.

docs/testing.rst

@@ -1,251 +0,0 @@
-=======
-Testing
-=======
-
-Running tests
-=============
-
-To run everything, just use ``./tools/test-all``. This runs lint checks,
-web frontend / whole-system blackbox tests, and backend Django tests.
-
-If you want to run individual parts, see the various commands inside
-that script.
-
-Schema and initial data changes
--------------------------------
-
-If you change the database schema or change the initial test data, you
-have have to regenerate the pristine test database by running
-``tools/do-destroy-rebuild-test-database``.
-
-Wiping the test databases
--------------------------
-
-You should first try running: ``tools/do-destroy-rebuild-test-database``
-
-If that fails you should try to do:
-
-::
-
-    sudo -u postgres psql
-    > DROP DATABASE zulip_test;
-    > DROP DATABASE zulip_test_template;
-
-and then run ``tools/do-destroy-rebuild-test-database``
-
-Recreating the postgres cluster
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. warning::
-
-   **This is irreversible, so do it with care, and never do this anywhere
-   in production.**
-
-If your postgres cluster (collection of databases) gets totally trashed
-permissions-wise, and you can't otherwise repair it, you can recreate
-it. On Ubuntu:
-
-::
-
-    sudo pg_dropcluster --stop 9.1 main
-    sudo pg_createcluster --locale=en_US.utf8 --start 9.1 main
-
-Backend Django tests
---------------------
-
-These live in ``zerver/tests.py`` and ``zerver/test_*.py``. Run them
-with ``tools/test-backend``.
-
-Web frontend black-box tests
-----------------------------
-
-These live in ``zerver/tests/frontend/tests/``. This is a "black box"
-test; we load the frontend in a real (headless) browser, from a real dev
-server, and simulate UI interactions like sending messages, narrowing,
-etc.
-
-Since this is interacting with a real dev server, it can catch backend
-bugs as well.
-
-You can run this with ``./zerver/tests/frontend/run``. You will need
-`PhantomJS <http://phantomjs.org/>`__ 1.7.0 or later.
-
-Debugging Casper.JS
-~~~~~~~~~~~~~~~~~~~
-
-Casper.js (via PhantomJS) has support for remote debugging. However, it
-is not perfect. Here are some steps for using it and gotchas you might
-want to know.
-
-To turn on remote debugging, pass ``--remote-debug`` to the
-``./zerver/frontend/tests/run`` script. This will run the tests with
-port ``7777`` open for remote debugging. You can now connect to
-``localhost:7777`` in a Webkit browser. Somewhat recent versions of
-Chrome or Safari might be required.
-
--  When connecting to the remote debugger, you will see a list of pages,
-   probably 2. One page called ``about:blank`` is the headless page in
-   which the CasperJS test itself is actually running in. This is where
-   your test code is.
--  The other page, probably ``localhost:9981``, is the Zulip page that
-   the test is testing---that is, the page running our app that our test
-   is exercising.
-
-Since the tests are now running, you can open the ``about:blank`` page,
-switch to the Scripts tab, and open the running ``0x-foo.js`` test. If
-you set a breakpoint and it is hit, the inspector will pause and you can
-do your normal JS debugging. You can also put breakpoints in the Zulip
-webpage itself if you wish to inspect the state of the Zulip frontend.
-
-Web frontend unit tests
------------------------
-
-As an alternative to the black-box whole-app testing, you can unit test
-individual JavaScript files that use the module pattern. For example, to
-test the ``foobar.js`` file, you would first add the following to the
-bottom of ``foobar.js``:
-
-  ::
-
-     if (typeof module !== 'undefined') {
-         module.exports = foobar;
-     }
-
-This makes ``foobar.js`` follow the CommonJS module pattern, so it can
-be required in Node.js, which runs our tests.
-
-Now create ``zerver/tests/frontend/node/foobar.js``. At the top, require
-the `Node.js assert module <http://nodejs.org/api/assert.html>`__, and
-the module you're testing, like so:
-
-  ::
-
-     var assert = require('assert');
-     var foobar = require('js/foobar.js');
-
-(If the module you're testing depends on other modules, or modifies
-global state, you need to also read `the next section`__.)
-
-__ handling-dependencies_
-
-Define and call some tests using the `assert
-module <http://nodejs.org/api/assert.html>`__. Note that for "equal"
-asserts, the *actual* value comes first, the *expected* value second.
-
-  ::
-
-     (function test_somefeature() {
-         assert.strictEqual(foobar.somefeature('baz'), 'quux');
-         assert.throws(foobar.somefeature('Invalid Input'));
-     }());
-
-The test runner (index.js) automatically runs all .js files in the
-zerver/tests/frontend/node directory.
-
-.. _handling-dependencies:
-
-Handling dependencies in tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following scheme helps avoid tests leaking globals between each
-other.
-
-First, if you can avoid globals, do it, and the code that is directly
-under test can simply be handled like this:
-
-  ::
-
-        var search = require('js/search_suggestion.js');
-
-For deeper dependencies, you want to categorize each module as follows:
-
--  Exercise the module's real code for deeper, more realistic testing?
--  Stub out the module's interface for more control, speed, and
-   isolation?
--  Do some combination of the above?
-
-For all the modules where you want to run actual code, add a statement
-like the following to the top of your test file:
-
-  ::
-
-     add_dependencies({
-         _: 'third/underscore/underscore.js',
-         util: 'js/util.js',
-         Dict: 'js/dict.js',
-         Handlebars: 'handlebars',
-         Filter: 'js/filter.js',
-         typeahead_helper: 'js/typeahead_helper.js',
-         stream_data: 'js/stream_data.js',
-         narrow: 'js/narrow.js'
-     });
-
-For modules that you want to completely stub out, please use a pattern
-like this:
-
-  ::
-
-     set_global('page_params', {
-         email: 'bob@zulip.com'
-     });
-
-     // then maybe further down
-     global.page_params.email = 'alice@zulip.com';
-
-Finally, there's the hybrid situation, where you want to borrow some of
-a module's real functionality but stub out other pieces. Obviously, this
-is a pretty strong smell that the other module might be lacking in
-cohesion, but that code might be outside your jurisdiction. The pattern
-here is this:
-
-  ::
-
-     // Use real versions of parse/unparse
-     var narrow = require('js/narrow.js');
-     set_global('narrow', {
-         parse: narrow.parse,
-         unparse: narrow.unparse
-     });
-
-     // But later, I want to stub the stream without having to call super-expensive
-     // real code like narrow.activate().
-     global.narrow.stream = function () {
-         return 'office';
-     };
-
-Coverage reports
-~~~~~~~~~~~~~~~~
-
-You can automatically generate coverage reports for the JavaScript unit
-tests. To do so, install istanbul:
-
-  ::
-
-     sudo npm install -g istanbul
-
-And run test-js-with-node with the 'cover' parameter:
-
-  ::
-
-     tools/test-js-with-node cover
-
-Then open ``coverage/lcov-report/js/index.html`` in your browser.
-Modules we don't test *at all* aren't listed in the report, so this
-tends to overstate how good our overall coverage is, but it's accurate
-for individual files. You can also click a filename to see the specific
-statements and branches not tested. 100% branch coverage isn't
-necessarily possible, but getting to at least 80% branch coverage is a
-good goal.
-
-Manual testing (local app + web browser)
-========================================
-
-Setting up the manual testing database
---------------------------------------
-
-::
-
-    ./tools/do-destroy-rebuild-database
-
-Will populate your local database with all the usual accounts plus some
-test messages involving Shakespeare characters.

docs/translating.md

@@ -0,0 +1,123 @@
+# Translating Zulip
+
+Zulip has full support for unicode, so you can already use your
+preferred language everywhere in Zulip.
+
+To make Zulip even better for users around the world, the Zulip UI is
+being translated into a number of major languages, including Spanish,
+German, French, Chinese, Russian, and Japanese, with varying levels of
+progress.  If you speak a language other than English, your help with
+translating Zulip would be greatly appreciated!
+
+If you're interested in contributing translations to Zulip, join the
+[Zulip project on Transifex](https://www.transifex.com/zulip/zulip/)
+and ask to join any languages you'd like to contribute to (or add new
+ones).  Transifex's notification system sometimes fails to notify the
+maintainers when you ask to join a project, so please send a quick
+email to zulip-core@googlegroups.com when you request to join the
+project or add a language so that we can be sure to accept your
+request to contribute.
+
+## Translation Tags
+
+All user-facing text in the Zulip UI should be generated by a HTML
+template so that it can be translated.
+
+Zulip uses two types of templates: backend templates (powered by the
+[Jinja2][] template engine, though the original [Django][] template
+engine is still supported) and frontend templates (powered by
+[Handlebars][]).  At present, the frontend templates don't support
+translation (though we're working on fixing this!), so the rest of
+this discussion will be about the backend templates.
+
+To mark a string for translation in the Jinja2 and Django template
+engines, you can use the `_()` function in the templates like this:
+
+```
+{{ _("English text") }}
+```
+
+If a string contains both a literal string component and variables,
+you can use a block translation, which makes use of placeholders to
+help translators to translated an entire sentence.  To translate a
+block, Jinja2 uses the [trans][] tag while Django uses the
+[blocktrans][] tag.  So rather than writing something ugly and
+confusing for translators like this:
+
+```
+# Don't do this!
+{{ _("This string will have") }} {{ value }} {{ _("inside") }}
+```
+
+You can instead use:
+
+```
+# Jinja2 style
+{% trans %}This string will have {{ value }} inside.{% endtrans %}
+# Django style
+{% blocktrans %}This string will have {{ value }} inside.{% endblocktrans %}
+```
+
+Zulip expects all the error messages to be translatable.  To ensure
+this, the error message passed to `json_error` and `JsonableError`
+should always be a literal string enclosed by `_()` function, e.g:
+
+```
+json_error(_('English Text'))
+JsonableError(_('English Text'))
+```
+
+To ensure we always internationalize our JSON errors messages, the
+Zulip linter (`tools/lint-all`) checks for correct usage.
+
+## Frontend Translations
+The first step in translating the frontend is to create the translation
+files using `python manage makemessages`. This command will create
+translation files under `static/locale`, the location can be changed by
+passing an argument to the command, however make sure that the location is
+publicly accessible since these files are loaded through XHR in the
+frontend which will only work with publicly accessible resources.
+
+The second step is to upload the translatable strings to Transifex using
+`tx push -s -a`.
+
+The final step is to get the translated files from Transifex using
+`tx pull -a`.
+
+
+[Django]: https://docs.djangoproject.com/en/1.9/topics/templates/#the-django-template-language
+[Jinja2]: http://jinja.pocoo.org/
+[Handlebars]: http://handlebarsjs.com/
+[trans]: http://jinja.pocoo.org/docs/dev/templates/#i18n
+[blocktrans]: https://docs.djangoproject.com/en/1.8/topics/i18n/translation/#std:templatetag-blocktrans
+
+## Testing Translations
+
+First of all make sure that you have compiled the translation strings
+using `python manage.py compilemessages`.
+
+Django figures out the effective language by going through the
+following steps:
+
+1. It looks for the language code in the url.
+2. It loooks for the LANGUGE_SESSION_KEY key in the current user's
+session.
+3. It looks for the cookie named 'django_language'. You can set a
+different name through LANGUAGE_COOKIE_NAME setting.
+4. It looks for the `Accept-Language` HTTP header in the HTTP request.
+Normally your browser will take care of this.
+
+The easiest way to test translations is through the i18n urls e.g. if
+you have German translations available you can access the German
+version of a page by going to `/de/path_to_page`.
+
+To test translations using other methods you will need an HTTP client
+library like `requests`, `cURL` or `urllib`. Here is a sample code to
+test `Accept-Language` header using requests:
+
+```
+import requests
+headers = {"Accept-Language": "de"}
+response = requests.get("http://localhost:9991/login/", headers=headers)
+print(response.content)
+```

frontend_tests/casper_lib/.gitignore

@@ -0,0 +1,2 @@
+/server.log
+/test_credentials.js