Add changelog for Zulip 1.3.13 release.

If a user's session cookie expired, the next REST API request their
browser did would go into the json_unauthorized code path.  This
returned a response with a WWW-Authenticate tag for HTTP Basic Auth
(since that's what the REST API uses), even for /json requests which
should only be authenticated using session auth.

We fix this by explicitly passing the desired WWW-Authenticate state.

Fixes: #800.

.coveralls.yml

@@ -0,0 +1,2 @@
+service_name: travis-pro
+repo_token: hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG

.gitattributes

@@ -14,7 +14,8 @@
 /zproject/local_settings.py export-ignore
 /zproject/test_settings.py export-ignore
 /zerver/fixtures export-ignore
-/zerver/tests.py export-ignore
+/zerver/tests export-ignore
 /frontend_tests export-ignore
 /node_modules export-ignore
 /humbug export-ignore
+/locale export-ignore

.gitignore

@@ -13,6 +13,7 @@
 /update-prod-static.log
 frontend_tests/casper_tests/server.log
 frontend_tests/casper_lib/test_credentials.js
+memcached_prefix
 /prod-static
 /errors/*
 *.sw[po]
@@ -24,7 +25,7 @@ zerver/fixtures/migration-status
 zerver/fixtures/test_data1.json
 .kdev4
 zulip.kdev4
-memcached_prefix
+remote_cache_prefix
 coverage/
 /queue_error
 .test-js-with-node.html
@@ -37,7 +38,11 @@ event_queues.json
 static/js/bundle.js
 static/third/gemoji/
 static/third/zxcvbn/
-tools/emoji_dump/bitmaps/
-tools/emoji_dump/*.ttx
+tools/setup/emoji_dump/bitmaps/
+tools/setup/emoji_dump/*.ttx
 tools/phantomjs
 node_modules
+npm-debug.log
+uploads/
+test_uploads/
+*.mo

.travis.yml

@@ -1,19 +1,33 @@
 before_install:
    - nvm install 0.10
 install:
+  - pip install coveralls
   - tools/travis/setup-$TEST_SUITE
+  - tools/clean-venv-cache --travis
 cache:
   - apt: false
   - directories:
-    - /srv/phantomjs
+    - $HOME/phantomjs
+    - $HOME/zulip-venv-cache
+    - node_modules
+    - $HOME/node
 env:
-  - TEST_SUITE=frontend
-  - TEST_SUITE=backend
-  - TEST_SUITE=production
-  - TEST_SUITE=py3k
+  global:
+    - COVERALLS_PARALLEL=true
+    - COVERALLS_SERVICE_NAME=travis-pro
+    - COVERALLS_REPO_TOKEN=hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG
+  matrix:
+    - TEST_SUITE=frontend
+    - TEST_SUITE=backend
+    - TEST_SUITE=production
+    - TEST_SUITE=py3k
 language: python
 python:
   - "2.7"
+matrix:
+  include:
+    - python: "3.4"
+      env: TEST_SUITE=mypy
 # command to run tests
 script:
   - ./tools/travis/$TEST_SUITE
@@ -22,3 +36,7 @@ services:
 - docker
 addons:
   postgresql: "9.3"
+after_success:
+  coveralls
+notifications:
+  webhooks: https://coveralls.io/webhook?repo_token=$COVERALLS_REPO_TOKEN

.tx/config

@@ -0,0 +1,17 @@
+[main]
+host = https://www.transifex.com
+
+[zulip.djangopo]
+source_file = locale/en/LC_MESSAGES/django.po
+source_lang = en
+type = PO
+file_filter = locale/<lang>/LC_MESSAGES/django.po
+lang_map = zh-Hans: zh_CN
+
+[zulip.translationsjson]
+source_file = static/locale/en/translations.json
+source_lang = en
+type = KEYVALUEJSON
+file_filter = static/locale/<lang>/translations.json
+lang_map = zh-Hans: zh-CN
+

Dockerfile

@@ -9,4 +9,7 @@ RUN apt-get update && apt-get install -y \
 RUN useradd -d /home/zulip -m zulip && echo 'zulip ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
 USER zulip
+
+RUN ln -nsf /srv/zulip ~/zulip
+
 WORKDIR /srv/zulip

README.md

@@ -1,5 +1,11 @@
-Zulip
-=====
+**[Zulip overview](#zulip-overview)** |
+**[Installing for dev](#installing-the-zulip-development-environment)** |
+**[Installing for production](#running-zulip-in-production)** |
+**[Ways to contribute](#ways-to-contribute)** |
+**[How to get involved](#how-to-get-involved-with-contributing-to-zulip)** |
+**[License](#license)**
+
+# Zulip overview
 
 Zulip is a powerful, open source group chat application. Written in
 Python and using the Django framework, Zulip supports both private
@@ -12,85 +18,128 @@ missed-message emails, desktop apps, and much more.
 Further information on the Zulip project and its features can be found
 at https://www.zulip.org.
 
-Installing the Zulip Development environment
-============================================
+[![Build Status](https://travis-ci.org/zulip/zulip.svg?branch=master)](https://travis-ci.org/zulip/zulip) [![Coverage Status](https://coveralls.io/repos/github/zulip/zulip/badge.svg?branch=master)](https://coveralls.io/github/zulip/zulip?branch=master)
 
-The Zulip development environment is the recommened option for folks
+## Installing the Zulip Development environment
+
+The Zulip development environment is the recommended option for folks
 interested in trying out Zulip.  This is documented in
-[README.dev.md](README.dev.md).
+[README.dev.md](https://github.com/zulip/zulip/blob/master/README.dev.md).
 
-Running Zulip in production
-===========================
+## Running Zulip in production
 
 Zulip in production only supports Ubuntu 14.04 right now, but work is
 ongoing on adding support for additional platforms. The installation
-process is documented in https://zulip.org/server.html and in more
-detail in [README.prod.md](README.prod.md).
+process is documented at https://zulip.org/server.html and in more
+detail in [README.prod.md](https://github.com/zulip/zulip/blob/master/README.prod.md).
 
-Contributing to Zulip
-=====================
+## Ways to contribute
 
 Zulip welcomes all forms of contributions!  The page documents the
 Zulip development process.
 
-* **Pull requests**. Before a pull request can be merged, you need to to sign the [Dropbox
-Contributor License Agreement](https://opensource.dropbox.com/cla/).
-Also, please skim our [commit message style
-guidelines](http://zulip.readthedocs.org/en/latest/code-style.html#commit-messages).
+* **Pull requests**. Before a pull request can be merged, you need to
+to sign the [Dropbox Contributor License Agreement][cla].  Also,
+please skim our [commit message style guidelines][doc-commit-style].
 
 * **Testing**. The Zulip automated tests all run automatically when
 you submit a pull request, but you can also run them all in your
 development environment following the instructions in the [testing
-section](https://github.com/zulip/zulip#running-the-test-suite) below.
+docs][doc-test].
 
 * **Developer Documentation**.  Zulip has a growing collection of
-developer documentation on [Read The Docs](https://zulip.readthedocs.org/).
-Recommended reading for new contributors includes the
-[directory structure](http://zulip.readthedocs.org/en/latest/directory-structure.html) and
-[new feature tutorial](http://zulip.readthedocs.org/en/latest/new-feature-tutorial.html).
+developer documentation on [Read The Docs][doc].  Recommended reading
+for new contributors includes the [directory structure][doc-dirstruct]
+and [new feature tutorial][doc-newfeat]. You can also improve
+[Zulip.org][z-org].
 
-* **Mailing list and bug tracker** Zulip has a [development discussion
-mailing list](https://groups.google.com/forum/#!forum/zulip-devel) and
-uses [GitHub issues](https://github.com/zulip/zulip/issues).  Feel
-free to send any questions or suggestions of areas where you'd love to
-see more documentation to the list!  Please report any security issues
-you discover to support@zulip.com.
+* **Mailing lists and bug tracker**. Zulip has a [development
+discussion mailing list][gg-devel] and uses [GitHub issues
+][gh-issues].  There are also lists for the [Android][email-android]
+and [iOS][email-ios] apps.  Feel free to send any questions or
+suggestions of areas where you'd love to see more documentation to the
+relevant list!  Please report any security issues you discover to
+zulip-security@googlegroups.com.
 
-* **App codebases** This repository is for the Zulip server and web app; the
-[desktop](https://github.com/zulip/zulip-desktop),
-[Android](https://github.com/zulip/zulip-android), and
-[iOS](https://github.com/zulip/zulip-ios) apps are separate
-repositories.
+* **App codebases**. This repository is for the Zulip server and web
+app (including most integrations); the [desktop][], [Android][], and
+[iOS][] apps, are separate repositories, as is [our experimental React
+Native iOS app][ios-exp].
 
-How to get involved with contributing to Zulip
-==============================================
+* **Glue code**. We maintain a [Hubot adapter][hubot-adapter] and several
+integrations ([Phabricator][phab], [Jenkins][], [Puppet][], [Redmine][],
+and [Trello][]), plus [node.js API bindings][node], and a [full-text search
+PostgreSQL extension][tsearch], as separate repos.
 
-First, subscribe to the Zulip [development discussion mailing list](https://groups.google.com/forum/#!forum/zulip-devel).
+* **Translations**.  Zulip is in the process of being translated into
+10+ languages, and we love contributions to our translations.  See our
+[translating documentation](transifex) if you're interested in
+contributing!
+
+[cla]: https://opensource.dropbox.com/cla/
+[doc]: https://zulip.readthedocs.io/
+[doc-commit-style]: http://zulip.readthedocs.io/en/latest/code-style.html#commit-messages
+[doc-dirstruct]: http://zulip.readthedocs.io/en/latest/directory-structure.html
+[doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
+[doc-test]: https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+[gg-devel]: https://groups.google.com/forum/#!forum/zulip-devel
+[gh-issues]: https://github.com/zulip/zulip/issues
+[desktop]: https://github.com/zulip/zulip-desktop
+[android]: https://github.com/zulip/zulip-android
+[ios]: https://github.com/zulip/zulip-ios
+[ios-exp]: https://github.com/zulip/zulip-mobile
+[email-android]: https://groups.google.com/forum/#!forum/zulip-android
+[email-ios]: https://groups.google.com/forum/#!forum/zulip-ios
+[hubot-adapter]: https://github.com/zulip/hubot-zulip
+[jenkins]: https://github.com/zulip/zulip-jenkins-plugin
+[node]: https://github.com/zulip/zulip-node
+[phab]: https://github.com/zulip/phabricator-to-zulip
+[puppet]: https://github.com/matthewbarr/puppet-zulip
+[redmine]: https://github.com/zulip/zulip-redmine-plugin
+[trello]: https://github.com/zulip/trello-to-zulip
+[tsearch]: https://github.com/zulip/tsearch_extras
+[transifex]: https://www.transifex.com/zulip/zulip/
+[z-org]: https://github.com/zulip/zulip.github.io
+
+## How to get involved with contributing to Zulip
+
+First, subscribe to the Zulip [development discussion mailing
+list][gg-devel].
 
 The Zulip project uses a system of labels in our [issue
-tracker](https://github.com/zulip/zulip/issues) to make it easy to
-find a project if you don't have your own project idea in mind or want
-to get some experience with working on Zulip before embarking on a
-larger project you have in mind:
+tracker][gh-issues] to make it easy to find a project if you don't
+have your own project idea in mind or want to get some experience with
+working on Zulip before embarking on a larger project you have in
+mind:
 
-* [Bite Size](https://github.com/zulip/zulip/labels/bite%20size):
-  Smaller projects that could be a great first contribution.
 * [Integrations](https://github.com/zulip/zulip/labels/integrations).
   Integrate Zulip with another piece of software and contribute it
-  back to the community!  Writing an integration can be a great
-  started project.  There's some brief documentation on the best way
-  to write integrations at https://github.com/zulip/zulip/issues/70.
-* [Documentation](https://github.com/zulip/zulip/labels/documentation).
+  back to the community!  Writing an integration can be a great first
+  contribution.  There's detailed documentation on how to write
+  integrations in [the Zulip integration writing
+  guide](https://zulip.readthedocs.io/en/latest/integration-guide.html).
+
+* [Bite Size](https://github.com/zulip/zulip/labels/bite%20size):
+  Smaller projects that might be a great first contribution.
+
+* [Documentation](https://github.com/zulip/zulip/labels/documentation):
   The Zulip project loves contributions of new documentation.
+
 * [Help Wanted](https://github.com/zulip/zulip/labels/help%20wanted):
   A broader list of projects that nobody is currently working on.
-* [Platform support](https://github.com/zulip/zulip/labels/Platform%20support).
-  These are open issues about making it possible to install Zulip on a wider
-  range of platforms.
-* [Bugs](https://github.com/zulip/zulip/labels/bug). Open bugs.
-* [Feature requests](https://github.com/zulip/zulip/labels/enhancement).
-  Browsing this list can be a great way to find feature ideas to implement that
-  other Zulip users are excited about.
+
+* [Platform support](https://github.com/zulip/zulip/labels/Platform%20support):
+  These are open issues about making it possible to install Zulip on a
+  wider range of platforms.
+
+* [Bugs](https://github.com/zulip/zulip/labels/bug): Open bugs.
+
+* [Feature requests](https://github.com/zulip/zulip/labels/enhancement):
+  Browsing this list can be a great way to find feature ideas to
+  implement that other Zulip users are excited about.
+
+* [2016 roadmap milestone](http://zulip.readthedocs.io/en/latest/roadmap.html): The
+  projects that are [priorities for the Zulip project](https://zulip.readthedocs.io/en/latest/roadmap.html).  These are great projects if you're looking to make an impact.
 
 If you're excited about helping with an open issue, just post on the
 conversation thread that you're working on it.  You're encouraged to
@@ -123,11 +172,10 @@ looking at the new feature tutorial and coding style guidelines on
 ReadTheDocs.
 
 Feedback on how to make this development process more efficient, fun,
-and friendly to new contributors is very welcome!  Just shoot an email
+and friendly to new contributors is very welcome!  Just send an email
 to the Zulip Developers list with your thoughts.
 
-License
-=======
+## License
 
 Copyright 2011-2015 Dropbox, Inc.
 

README.prod.md

@@ -38,7 +38,8 @@ These instructions should be followed as root.
 
 (1) Install the SSL certificates for your machine to
   `/etc/ssl/private/zulip.key` and `/etc/ssl/certs/zulip.combined-chain.crt`.
-  If you don't know how to generate an SSL certificate, you, you can
+
+  If you don't know how to generate an SSL certificate, you can
   do the following to generate a self-signed certificate:
 
   ```
@@ -53,17 +54,45 @@ These instructions should be followed as root.
   cp zulip.combined-chain.crt /etc/ssl/certs/zulip.combined-chain.crt
   ```
 
-  You will eventually want to get a properly signed certificate (and
-  note that at present the Zulip desktop app doesn't support
+  You will eventually want to get a properly signed SSL certificate
+  (and note that at present the Zulip desktop app doesn't support
   self-signed certificates), but this will let you finish the
-  installation process.
+  installation process.  When you do get an actual certificate, you
+  will need to install as /etc/ssl/certs/zulip.combined-chain.crt the
+  full certificate authority chain, not just the certificate; see the
+  section on "SSL certificate chains" [in the nginx
+  docs](http://nginx.org/en/docs/http/configuring_https_servers.html)
+  for how to do this:
+
+  You can get a free, properly signed certificate from the [Let's
+  Encrypt service](https://letsencrypt.org/); here are the simplified
+  instructions for using it with Zulip (run it all as root):
+
+  ```
+  sudo apt-get install git bc nginx
+  git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
+  cd /opt/letsencrypt
+  letsencrypt-auto certonly --standalone
+
+  # Now symlink the certificates to make them available where Zulip expects them.
+  ln -s /etc/letsencrypt/live/your_domain/privkey.pem /etc/ssl/private/zulip.key
+  ln -s /etc/letsencrypt/live/your_domain/fullchain.pem /etc/ssl/certs/zulip.combined-chain.crt
+  ```
+
+  If you already had a webserver installed on the system (e.g. you
+  previously installed Zulip and are now getting a cert), you will
+  need to stop the webserver (e.g. `service nginx stop`) and start it
+  again after (e.g. `service nginx start`) running the above.
+
+  Finally, if you want to proceed with just an IP address, it is
+  possible to finish a Zulip installation that way; just set
+  EXTERNAL_HOST to be the IP address.
 
 (2) Download [the latest built server tarball](https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz)
   and unpack it to `/root/zulip`, e.g.
   ```
   wget https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz
-  tar -xf zulip-server-latest.tar.gz
-  mv zulip-server-1.3.6 /root/zulip
+  mkdir -p /root/zulip && tar -xf zulip-server-latest.tar.gz --directory=/root/zulip --strip-components=1
   ```
 
 (3) Run
@@ -138,8 +167,11 @@ need to do some additional setup documented in the `settings.py` template:
 
 * For Google authentication, you need to follow the configuration
   instructions around `GOOGLE_OAUTH2_CLIENT_ID` and `GOOGLE_CLIENT_ID`.
+
 * For Email authentication, you will need to follow the configuration
-  instructions around outgoing SMTP from Django.
+  instructions for outgoing SMTP from Django.  You can use `manage.py
+  send_test_email username@example.com` to test whether you've
+  successfully configured outgoing SMTP.
 
 You should be able to login now.  If you get an error, check
 `/var/log/zulip/errors.log` for a traceback, and consult the next
@@ -313,15 +345,17 @@ only supports talking to servers with a properly signed SSL
 certificate, so you may find that you get a blank screen when you
 connect to a Zulip server using a self-signed certificate.
 
-The Zulip iOS and Android apps in their respective stores don't yet
-support talking to non-zulip.com servers; the iOS app is waiting on
-Apple's app store review, while the Android app is waiting on someone
-to do the small project of adding a field to specify what Zulip server
-to talk to.
+The Zulip Android app in the Google Play store doesn't yet support
+talking to non-zulip.com servers (and the iOS one doesn't support
+Google auth SSO against non-zulip.com servers; there's a design for
+how to fix that which wouldn't be a ton of work to implement).  If you
+are interested in helping out with the Zulip mobile apps, shoot an
+email to zulip-devel@googlegroups.com and the maintainers can guide
+you on how to help.
 
-These issues will likely all be addressed in the coming weeks; make
-sure to join the zulip-announce@googlegroups.com list so that you can
-receive the announcements when these become available.
+For announcements about improvements to the apps, make sure to join
+the zulip-announce@googlegroups.com list so that you can receive the
+announcements when these become available.
 
 (5) All the other features: Hotkeys, emoji, search filters,
 @-mentions, etc.  Zulip has lots of great features, make sure your
@@ -377,7 +411,7 @@ upgrade.
 * The Zulip upgrade process works by creating a new deployment under
   /home/zulip/deployments/ containing a complete copy of the Zulip
   server code, and then moving the symlinks at
-  `/home/zulip/deployments/current` and /root/zulip` as part of the
+  `/home/zulip/deployments/current` and `/root/zulip` as part of the
   upgrade process.  This means that if the new version isn't working,
   you can quickly downgrade to the old version by using
   `/home/zulip/deployments/<date>/scripts/restart-server` to return to
@@ -443,7 +477,7 @@ computed using a hash of avatar_salt and user's email), etc.
 they do get large on a busy server, and it's definitely
 lower-priority.
 
-### Restoration
+#### Restoration
 
 To restore from backups, the process is basically the reverse of the above:
 
@@ -475,10 +509,11 @@ that they are up to date using the Nagios plugin at:
 Contributions to more fully automate this process or make this section
 of the guide much more explicit and detailed are very welcome!
 
-### Postgres streaming replication
 
-Zulip has database configuration for doing with Postgres streaming
-replication ; you can see the configuration in these files:
+#### Postgres streaming replication
+
+Zulip has database configuration for using Postgres streaming
+replication; you can see the configuration in these files:
 
 * puppet/zulip_internal/manifests/postgres_slave.pp
 * puppet/zulip_internal/manifests/postgres_master.pp
@@ -488,65 +523,64 @@ Contribution of a step-by-step guide for setting this up (and moving
 this configuration to be available in the main `puppet/zulip/` tree)
 would be very welcome!
 
-### Using a remote postgres host
-
-This is a bit annoying to setup, but you can configure Zulip to use a
-dedicated postgres server by setting the `REMOTE_POSTGRES_HOST`
-variable in /etc/zulip/settings.py, and configuring Postgres
-certificate authentication (see
-http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and
-http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
-documentation on how to set this up and deploy the certificates) to
-make the DATABASES configuration in `zproject/settings.py` work (or
-override that configuration).
 
 ### Monitoring Zulip
 
-The complete Nagios configuration (sans secret keys) we used to
+The complete Nagios configuration (sans secret keys) used to
 monitor zulip.com is available under `puppet/zulip_internal` in the
 Zulip Git repository (those files are not installed in the release
-tarballs); there are a number of useful Nagios plugins available
-there, including:
+tarballs).
 
-Frontend server monitoring:
+The Nagios plugins used by that configuration are installed
+automatically by the Zulip installation process in subdirectories
+under `/usr/lib/nagios/plugins/`.  The following is a summary of the
+various Nagios plugins included with Zulip and what they check:
+
+Application server and queue worker monitoring:
 
 * check_send_receive_time (sends a test message through the system
   between two bot users to check that end-to-end message sending works)
-* check_website_response.sh (standard HTTP check)
-
-Queue worker monitoring:
 
 * check_rabbitmq_consumers and check_rabbitmq_queues (checks for
   rabbitmq being down or the queue workers being behind)
+
 * check_queue_worker_errors (checks for errors reported by the queue workers)
+
 * check_worker_memory (monitors for memory leaks in queue workers)
+
 * check_email_deliverer_backlog and check_email_deliverer_process
   (monitors for whether outgoing emails are being sent)
 
 Database monitoring:
 
-* check_pg_replication_lag
+* check_postgres_replication_lag (checks streaming replication is up
+  to date).
+
 * check_postgres (checks the health of the postgres database)
+
 * check_postgres_backup (checks backups are up to date; see above)
+
 * check_fts_update_log (monitors for whether full-text search updates
   are being processed)
 
 Standard server monitoring:
 
-* check_debian_packages
+* check_website_response.sh (standard HTTP check)
 
-Contributions on making it easier to monitor Zulip and maintain it in
-production, e.g.  https://github.com/zulip/zulip/issues/371, are very
-welcome!
+* check_debian_packages (checks apt repository is up to date)
+
+If you're using these plugins, bug reports and pull requests to make
+it easier to monitor Zulip and maintain it in production are
+encouraged!
 
 ### Scalability of Zulip
 
 This section attempts to address the considerations involved with
 running Zulip with a large team (>1000 users).
 
-* We recommend using a remote postgres database (see
-  REMOTE_POSTGRES_HOST docs above) for isolation, though it is not
-  required.  In the following, we discuss a relatively simple
+* We recommend using a [remote postgres
+  database](#postgres-database-details) for isolation, though it is
+  not required.  In the following, we discuss a relatively simple
   configuration with two types of servers: application servers
   (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and
   database servers.
@@ -593,12 +627,6 @@ running Zulip with a large team (>1000 users).
   likely the first part of any project to support exchanging events
   amongst multiple servers.
 
-* The first scalability issue encountered by a very large realm (more
-  than a few thousand users), will be with the [frontend buddy list
-  perf and UI](https://github.com/zulip/zulip/issues/262).  Fixing
-  this should be a small project; the code for that part of the UI
-  layer doesn't do proper incremental updates.
-
 Questions, concerns, and bug reports about this area of Zulip are very
 welcome!  This is an area we are hoping to improve.
 
@@ -683,15 +711,23 @@ we can do a responsible security announcement).
 #### Users and Bots
 
 * There are three types of users in a Zulip realm: Administrators,
-  normal users, and botsq.  Administrators have the ability to
+  normal users, and bots.  Administrators have the ability to
   deactivate and reactivate other human and bot users, delete streams,
   add/remove administrator privileges, as well as change configuration
   for the overall realm (e.g. whether an invitation is required to
   join the realm).  Being a Zulip administrator does not provide the
   ability to interact with other users' private messages or the
-  messages sent private streams to which the administrator is not
+  messages sent to private streams to which the administrator is not
   subscribed.  However, a Zulip administrator subscribed to a stream
-  can toggle whether that stream is public or private.
+  can toggle whether that stream is public or private.  Also, Zulip
+  realm administrators have administrative access to the API keys of
+  all bots in the realm, so a Zulip administrator may be able to
+  access messages sent to private streams that have bots subscribed,
+  by using the bot's credentials.
+
+  In the future, Zulip's security model may change to allow realm
+  administrators to access private messages (e.g. to support auditing
+  functionality).
 
 * Every Zulip user has an API key, available on the settings page.
   This API key can be used to do essentially everything the user can
@@ -857,10 +893,25 @@ hostname/DNS side of the configuration.  Suggestions for how to
 improve this SSO setup documentation are very welcome!
 
 
-Remote Postgresql database
-==========================
+Postgres database details
+=========================
 
-If you want to use a remote Postgresql database, you should configure the information about the connection with the server. You need a user called "zulip" in your database server. You can configure these options in /etc/zulip/settings.py
+#### Remote Postgres database
+
+This is a bit annoying to setup, but you can configure Zulip to use a
+dedicated postgres server by setting the `REMOTE_POSTGRES_HOST`
+variable in /etc/zulip/settings.py, and configuring Postgres
+certificate authentication (see
+http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and
+http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
+documentation on how to set this up and deploy the certificates) to
+make the DATABASES configuration in `zproject/settings.py` work (or
+override that configuration).
+
+If you want to use a remote Postgresql database, you should configure
+the information about the connection with the server. You need a user
+called "zulip" in your database server. You can configure these
+options in /etc/zulip/settings.py:
 
 * REMOTE_POSTGRES_HOST: Name or IP address of the remote host
 * REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are:
@@ -877,10 +928,101 @@ Then you should specify the password of the user zulip for the database in /etc/
 postgres_password = xxxx
 ```
 
-Finally you can stop your database in the zulip server to save some memory, you can do it with:
+Finally, you can stop your database on the Zulip server via:
 
 ```
 sudo service postgresql stop
 sudo update-rc.d postgresql disable
 ```
 
+In future versions of this feature, we'd like to implement and
+document how to the remote postgres database server itself
+automatically by using the Zulip install script with a different set
+of puppet manifests than the all-in-one feature; if you're interested
+in working on this, post to the Zulip development mailing list and we
+can give you some tips.
+
+#### Debugging postgres database issues
+
+When debugging postgres issues, in addition to the standard `pg_top`
+tool, often it can be useful to use this query:
+
+```
+SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY procpid;
+```
+
+which shows the currently running backends and their activity. This is
+similar to the pg_top output, with the added advantage of showing the
+complete query, which can be valuable in debugging.
+
+To stop a runaway query, you can run `SELECT pg_cancel_backend(pid
+int)` or `SELECT pg_terminate_backend(pid int)` as the 'postgres'
+user. The former cancels the backend's current query and the latter
+terminates the backend process. They are implemented by sending SIGINT
+and SIGTERM to the processes, respectively.  We recommend against
+sending a Postgres process SIGKILL. Doing so will cause the database
+to kill all current connections, roll back any pending transactions,
+and enter recovery mode.
+
+#### Stopping the Zulip postgres database
+
+To start or stop postgres manually, use the pg_ctlcluster command:
+
+```
+pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
+```
+
+By default, using stop uses "smart" mode, which waits for all clients
+to disconnect before shutting down the database. This can take
+prohibitively long. If you use the --force option with stop,
+pg_ctlcluster will try to use the "fast" mode for shutting
+down. "Fast" mode is described by the manpage thusly:
+
+  With the --force option the "fast" mode is used which rolls back all
+  active transactions, disconnects clients immediately and thus shuts
+  down cleanly. If that does not work, shutdown is attempted again in
+  "immediate" mode, which can leave the cluster in an inconsistent state
+  and thus will lead to a recovery run at the next start. If this still
+  does not help, the postmaster process is killed. Exits with 0 on
+  success, with 2 if the server is not running, and with 1 on other
+  failure conditions. This mode should only be used when the machine is
+  about to be shut down.
+
+Many database parameters can be adjusted while the database is
+running. Just modify /etc/postgresql/9.1/main/postgresql.conf and
+issue a reload. The logs will note the change.
+
+#### Debugging issues starting postgres
+
+pg_ctlcluster often doesn't give you any information on why the
+database failed to start. It may tell you to check the logs, but you
+won't find any information there. pg_ctlcluster runs the following
+command underneath when it actually goes to start Postgres:
+
+```
+/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o  '-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
+```
+
+Since pg_ctl doesn't redirect stdout or stderr, running the above can
+give you better diagnostic information. However, you might want to
+stop Postgres and restart it using pg_ctlcluster after you've debugged
+with this approach, since it does bypass some of the work that
+pg_ctlcluster does.
+
+
+#### Postgres Vacuuming alerts
+
+The `autovac_freeze` postgres alert from `check_postgres` is
+particularly important.  This alert indicates that the age (in terms
+of number of transactions) of the oldest transaction id (XID) is
+getting close to the `autovacuum_freeze_max_age` setting.  When the
+oldest XID hits that age, Postgres will force a VACUUM operation,
+which can often lead to sudden downtime until the operation finishes.
+If it did not do this and the age of the oldest XID reached 2 billion,
+transaction id wraparound would occur and there would be data loss.
+To clear the nagios alert, perform a `VACUUM` in each indicated
+database as a database superuser (`postgres`).
+
+See
+http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
+for more details on postgres vacuuming.

THIRDPARTY

@@ -63,16 +63,16 @@ Copyright: 2006 Otheus Shelling
 License: GPL-2.0
 Comment: Not linked.
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_debian_packages
+Files: puppet/zulip/files/nagios_plugins/zulip_base/check_debian_packages
 Copyright: 2005 Francesc Guasch
 License: GPL-2.0
 Comment: Not linked.
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_postgres.pl
+Files: puppet/zulip/files/nagios_plugins/zulip_postgres_appdb/check_postgres.pl
 Copyright: 2007-2015 Greg Sabino Mullane
 License: BSD-2-Clause
 
-Files: puppet/zulip_internal/files/nagios_plugins/check_website_response.sh
+Files: puppet/zulip/files/nagios_plugins/zulip_nagios_server/check_website_response.sh
 Copyright: 2011 Chris Freeman
 License: GPL-2.0
 
@@ -122,7 +122,7 @@ Copyright: 2013 Nijiko Yonskai
 License: Apache-2.0
 Comment: The software has been modified.
 
-Files: static/third/gemoji/images/emoji/unicode/* tools/emoji_dump/*.ttf
+Files: static/third/gemoji/images/emoji/unicode/* tools/setup/emoji_dump/*.ttf
 Copyright: Google, Inc.
 License: Apache-2.0
 Comment: These are actually Noto Emoji, not gemoji.
@@ -142,7 +142,7 @@ Copyright: 2013 Jack Moore
 License: Expat
 
 Files: static/third/jquery-caret/*
-Copyright: 2010 C.F., Wong
+Copyright: 2012, 2013 Andrew C. Dvorak
 License: Expat
 
 Files: static/third/jquery-filedrop/jquery.filedrop.js
@@ -204,7 +204,7 @@ Copyright: 2011-2013 Felix Gnass
 License: Expat
 
 Files: static/third/underscore/underscore.js
-Copyright: 2009-2013 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
+Copyright: 2009-2015 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 License: Expat
 Comment: https://github.com/jashkenas/underscore/blob/master/LICENSE
 
@@ -229,10 +229,6 @@ Files: tools/jslint/jslint.js
 Copyright: 2002 Douglas Crockford
 License: XXX-good-not-evil
 
-Files: tools/python-proxy
-Copyright: 2009 F.bio Domingues
-License: Expat
-
 Files: tools/review
 Copyright: 2010 Ksplice, Inc.
 License: Apache-2.0
@@ -246,6 +242,10 @@ Files: zerver/lib/ccache.py
 Copyright: 2013 David Benjamin and Alan Huang
 License: Expat
 
+Files: zerver/lib/decorator.py zerver/management/commands/runtornado.py scripts/setup/generate_secrets.py
+Copyright: Django Software Foundation and individual contributors
+License: BSD-3-Clause
+
 Files: frontend_tests/casperjs/*
 Copyright: 2011-2012 Nicolas Perriault
  Joyent, Inc. and other Node contributors

Vagrantfile

@@ -2,6 +2,11 @@
 
 VAGRANTFILE_API_VERSION = "2"
 
+def command?(name)
+  `which #{name}`
+  $?.success?
+end
+
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
   # For LXC. VirtualBox hosts use a different box, described below.
@@ -13,17 +18,59 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.synced_folder ".", "/vagrant", disabled: true
   config.vm.synced_folder ".", "/srv/zulip"
 
+  proxy_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
+  if File.file?(proxy_config_file)
+    http_proxy = https_proxy = no_proxy = ""
+
+    IO.foreach(proxy_config_file) do |line|
+      line.chomp!
+      key, value = line.split(nil, 2)
+      case key
+      when /^([#;]|$)/; # ignore comments
+      when "HTTP_PROXY"; http_proxy = value
+      when "HTTPS_PROXY"; https_proxy = value
+      when "NO_PROXY"; no_proxy = value
+      end
+    end
+
+    if Vagrant.has_plugin?("vagrant-proxyconf")
+      if http_proxy != ""
+        config.proxy.http = http_proxy
+      end
+      if https_proxy != ""
+        config.proxy.https = https_proxy
+      end
+      if https_proxy != ""
+        config.proxy.no_proxy = no_proxy
+      end
+    end
+  end
+
+  # Specify LXC provider before VirtualBox provider so it's preferred.
+  config.vm.provider "lxc" do |lxc|
+    if command? "lxc-ls"
+      LXC_VERSION = `lxc-ls --version`.strip unless defined? LXC_VERSION
+      if LXC_VERSION >= "1.1.0"
+        # Allow start without AppArmor, otherwise Box will not Start on Ubuntu 14.10
+        # see https://github.com/fgrehm/vagrant-lxc/issues/333
+        lxc.customize 'aa_allow_incomplete', 1
+      end
+      if LXC_VERSION >= "2.0.0"
+        lxc.backingstore = 'dir'
+      end
+    end
+  end
+
   config.vm.provider "virtualbox" do |vb, override|
     override.vm.box = "ubuntu/trusty64"
-    # 2GiB seemed reasonable here. The VM OOMs with only 1024MiB.
-    vb.memory = 2048
+    # It's possible we can get away with just 1GB; more testing needed
+    vb.memory = 1280
   end
 
 $provision_script = <<SCRIPT
 set -x
 set -e
-sudo apt-get update
-sudo apt-get install -y python-pbs
+ln -nsf /srv/zulip ~/zulip
 /usr/bin/python /srv/zulip/provision.py
 SCRIPT
 

analytics/management/commands/active_user_stats.py

@@ -2,6 +2,7 @@ from __future__ import absolute_import
 from __future__ import print_function
 
 from django.core.management.base import BaseCommand
+from typing import Any
 
 from zerver.models import UserPresence, UserActivity
 from zerver.lib.utils import statsd, statsd_key
@@ -15,6 +16,7 @@ class Command(BaseCommand):
     Run as a cron job that runs every 10 minutes."""
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         # Get list of all active users in the last 1 week
         cutoff = datetime.now() - timedelta(minutes=30, hours=168)
 
@@ -22,7 +24,7 @@ class Command(BaseCommand):
 
         # Calculate 10min, 2hrs, 12hrs, 1day, 2 business days (TODO business days), 1 week bucket of stats
         hour_buckets = [0.16, 2, 12, 24, 48, 168]
-        user_info = defaultdict(dict)
+        user_info = defaultdict(dict) # type: Dict[str, Dict[float, List[str]]]
 
         for last_presence in users:
             if last_presence.status == UserPresence.IDLE:
@@ -31,7 +33,7 @@ class Command(BaseCommand):
                 known_active = last_presence.timestamp
 
             for bucket in hour_buckets:
-                if not bucket in user_info[last_presence.user_profile.realm.domain]:
+                if bucket not in user_info[last_presence.user_profile.realm.domain]:
                     user_info[last_presence.user_profile.realm.domain][bucket] = []
                 if datetime.now(known_active.tzinfo) - known_active < timedelta(hours=bucket):
                     user_info[last_presence.user_profile.realm.domain][bucket].append(last_presence.user_profile.email)
@@ -40,14 +42,14 @@ class Command(BaseCommand):
             print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.active.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.active.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))
 
         # Also do stats for how many users have been reading the app.
-        users_reading = UserActivity.objects.select_related().filter(query="/json/update_message_flags")
+        users_reading = UserActivity.objects.select_related().filter(query="/json/messages/flags")
         user_info = defaultdict(dict)
         for activity in users_reading:
             for bucket in hour_buckets:
-                if not bucket in user_info[activity.user_profile.realm.domain]:
+                if bucket not in user_info[activity.user_profile.realm.domain]:
                     user_info[activity.user_profile.realm.domain][bucket] = []
                 if datetime.now(activity.last_visit.tzinfo) - activity.last_visit < timedelta(hours=bucket):
                     user_info[activity.user_profile.realm.domain][bucket].append(activity.user_profile.email)
@@ -55,4 +57,4 @@ class Command(BaseCommand):
             print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers reading for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.reading.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.reading.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))

analytics/management/commands/active_user_stats_by_day.py

@@ -5,6 +5,7 @@ import datetime
 import pytz
 
 from optparse import make_option
+from typing import Any
 from django.core.management.base import BaseCommand
 from zerver.lib.statistics import activity_averages_during_day
 
@@ -16,6 +17,7 @@ class Command(BaseCommand):
                      help="Day to query in format 2013-12-05.  Default is yesterday"),)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options["date"] is None:
             date = datetime.datetime.now() - datetime.timedelta(days=1)
         else:

analytics/management/commands/analyze_mit.py

@@ -1,6 +1,8 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
 from optparse import make_option
 from django.core.management.base import BaseCommand
 from zerver.models import Recipient, Message
@@ -10,6 +12,7 @@ import time
 import logging
 
 def compute_stats(log_level):
+    # type: (int) -> None
     logger = logging.getLogger()
     logger.setLevel(log_level)
 
@@ -27,15 +30,15 @@ def compute_stats(log_level):
                            "bitcoin@mit.edu", "lp@mit.edu", "clocks@mit.edu",
                            "root@mit.edu", "nagios@mit.edu",
                            "www-data|local-realm@mit.edu"])
-    user_counts = {}
+    user_counts = {} # type: Dict[str, Dict[str, int]]
     for m in mit_query.select_related("sending_client", "sender"):
         email = m.sender.email
         user_counts.setdefault(email, {})
         user_counts[email].setdefault(m.sending_client.name, 0)
         user_counts[email][m.sending_client.name] += 1
 
-    total_counts = {}
-    total_user_counts = {}
+    total_counts = {} # type: Dict[str, int]
+    total_user_counts = {} # type: Dict[str, int]
     for email, counts in user_counts.items():
         total_user_counts.setdefault(email, 0)
         for client_name, count in counts.items():
@@ -44,9 +47,9 @@ def compute_stats(log_level):
             total_user_counts[email] += count
 
     logging.debug("%40s | %10s | %s" % ("User", "Messages", "Percentage Zulip"))
-    top_percents = {}
+    top_percents = {} # type: Dict[int, float]
     for size in [10, 25, 50, 100, 200, len(total_user_counts.keys())]:
-        top_percents[size] = 0
+        top_percents[size] = 0.0
     for i, email in enumerate(sorted(total_user_counts.keys(),
                                      key=lambda x: -total_user_counts[x])):
         percent_zulip = round(100 - (user_counts[email].get("zephyr_mirror", 0)) * 100. /
@@ -76,6 +79,7 @@ class Command(BaseCommand):
     help = "Compute statistics on MIT Zephyr usage."
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         level = logging.INFO
         if options["verbose"]:
             level = logging.DEBUG

analytics/management/commands/analyze_user_activity.py

@@ -1,6 +1,9 @@
 from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
+from typing import Any, Dict
+
 from zerver.lib.statistics import seconds_usage_between
 
 from optparse import make_option
@@ -10,6 +13,7 @@ import datetime
 from django.utils.timezone import utc
 
 def analyze_activity(options):
+    # type: (Dict[str, Any]) -> None
     day_start = datetime.datetime.strptime(options["date"], "%Y-%m-%d").replace(tzinfo=utc)
     day_end = day_start + datetime.timedelta(days=options["duration"])
 
@@ -26,7 +30,7 @@ def analyze_activity(options):
             continue
 
         total_duration += duration
-        print("%-*s%s" % (37, user_profile.email, duration, ))
+        print("%-*s%s" % (37, user_profile.email, duration,))
 
     print("\nTotal Duration:                      %s" % (total_duration,))
     print("\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,))
@@ -43,7 +47,7 @@ It will correctly not count server-initiated reloads in the activity statistics.
 
 The duration flag can be used to control how many days to show usage duration for
 
-Usage: python2.7 manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
+Usage: python manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
 
 By default, if no date is selected 2013-09-10 is used. If no realm is provided, information
 is shown for all realms"""
@@ -55,4 +59,5 @@ is shown for all realms"""
         )
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         analyze_activity(options)

analytics/management/commands/client_activity.py

@@ -1,8 +1,11 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
-from django.db.models import Count
+from django.db.models import Count, QuerySet
 
 from zerver.models import UserActivity, UserProfile, Realm, \
     get_realm, get_user_profile_by_email
@@ -14,15 +17,17 @@ class Command(BaseCommand):
 
 Usage examples:
 
-python2.7 manage.py client_activity
-python2.7 manage.py client_activity zulip.com
-python2.7 manage.py client_activity jesstess@zulip.com"""
+python manage.py client_activity
+python manage.py client_activity zulip.com
+python manage.py client_activity jesstess@zulip.com"""
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('arg', metavar='<arg>', type=str, nargs='?', default=None,
                             help="realm or user to estimate client activity for")
 
     def compute_activity(self, user_activity_objects):
+        # type: (QuerySet) -> None
         # Report data from the past week.
         #
         # This is a rough report of client activity because we inconsistently
@@ -54,6 +59,7 @@ python2.7 manage.py client_activity jesstess@zulip.com"""
 
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['arg'] is None:
             # Report global activity.
             self.compute_activity(UserActivity.objects.all())

analytics/management/commands/realm_stats.py

@@ -1,6 +1,10 @@
 from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 import datetime
 import pytz
 
@@ -18,50 +22,60 @@ class Command(BaseCommand):
     help = "Generate statistics on realm activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def active_users(self, realm):
+        # type: (Realm) -> List[UserProfile]
         # Has been active (on the website, for now) in the last 7 days.
         activity_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=7)
         return [activity.user_profile for activity in \
                     UserActivity.objects.filter(user_profile__realm=realm,
                                                 user_profile__is_active=True,
                                                 last_visit__gt=activity_cutoff,
-                                                query="/json/update_pointer",
+                                                query="/json/users/me/pointer",
                                                 client__name="website")]
 
     def messages_sent_by(self, user, days_ago):
+        # type: (UserProfile, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender=user, pub_date__gt=sent_time_cutoff).count()
 
     def total_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return Message.objects.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def human_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def api_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         return (self.total_messages(realm, days_ago) - self.human_messages(realm, days_ago))
 
     def stream_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff,
                                      recipient__type=Recipient.STREAM).count()
 
     def private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.HUDDLE).count()
 
     def group_private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.PERSONAL).count()
 
     def report_percentage(self, numerator, denominator, text):
+        # type: (float, float, str) -> None
         if not denominator:
             fraction = 0.0
         else:
@@ -69,6 +83,7 @@ class Command(BaseCommand):
         print("%.2f%% of" % (fraction * 100,), text)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/management/commands/stream_stats.py

@@ -1,6 +1,9 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
 from django.db.models import Q
 from zerver.models import Realm, Stream, Message, Subscription, Recipient, get_realm
@@ -9,10 +12,12 @@ class Command(BaseCommand):
     help = "Generate statistics on the streams for a realm."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/management/commands/user_stats.py

@@ -1,8 +1,10 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from argparse import ArgumentParser
 import datetime
 import pytz
+from typing import Any
 
 from django.core.management.base import BaseCommand
 from zerver.models import UserProfile, Realm, Stream, Message, get_realm
@@ -12,15 +14,18 @@ class Command(BaseCommand):
     help = "Generate statistics on user activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def messages_sent_by(self, user, week):
+        # type: (UserProfile, int) -> int
         start = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=(week + 1)*7)
         end = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=week*7)
         return Message.objects.filter(sender=user, pub_date__gt=start, pub_date__lte=end).count()
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/urls.py

@@ -1,8 +1,9 @@
 from django.conf.urls import patterns, url
 
-urlpatterns = patterns('analytics.views',
-    url(r'^activity$', 'get_activity'),
-    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'get_realm_activity'),
-    url(r'^user_activity/(?P<email>[\S]+)/$', 'get_user_activity'),
-)
+i18n_urlpatterns = [
+    url(r'^activity$', 'analytics.views.get_activity'),
+    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'analytics.views.get_realm_activity'),
+    url(r'^user_activity/(?P<email>[\S]+)/$', 'analytics.views.get_user_activity'),
+]
 
+urlpatterns = patterns('', *i18n_urlpatterns)

analytics/views.py

@@ -1,10 +1,14 @@
 from __future__ import absolute_import
+from __future__ import division
+from six import text_type
+from typing import Any, Dict, List, Tuple, Optional, Sequence, Callable, Union
+
 from django.db import connection
+from django.db.models.query import QuerySet
 from django.template import RequestContext, loader
-from django.utils.html import mark_safe
-from django.shortcuts import render_to_response
 from django.core import urlresolvers
-from django.http import HttpResponseNotFound
+from django.http import HttpResponseNotFound, HttpRequest, HttpResponse
+from jinja2 import Markup as mark_safe
 
 from zerver.decorator import has_request_variables, REQ, zulip_internal
 from zerver.models import get_realm, UserActivity, UserActivityInterval, Realm
@@ -22,10 +26,14 @@ from six.moves import range
 from six.moves import zip
 eastern_tz = pytz.timezone('US/Eastern')
 
+from zproject.jinja2 import render_to_response
+
 def make_table(title, cols, rows, has_row_class=False):
+    # type: (str, List[str], List[Any], bool) -> str
 
     if not has_row_class:
         def fix_row(row):
+            # type: (Any) -> Dict[str, Any]
             return dict(cells=row, row_class=None)
         rows = list(map(fix_row, rows))
 
@@ -39,6 +47,7 @@ def make_table(title, cols, rows, has_row_class=False):
     return content
 
 def dictfetchall(cursor):
+    # type: (connection.cursor) -> List[Dict[str, Any]]
     "Returns all rows from a cursor as a dict"
     desc = cursor.description
     return [
@@ -48,6 +57,7 @@ def dictfetchall(cursor):
 
 
 def get_realm_day_counts():
+    # type: () -> Dict[str, Dict[str, str]]
     query = '''
         select
             r.domain,
@@ -75,18 +85,19 @@ def get_realm_day_counts():
     rows = dictfetchall(cursor)
     cursor.close()
 
-    counts = defaultdict(dict)
+    counts = defaultdict(dict) # type: Dict[str, Dict[int, int]]
     for row in rows:
         counts[row['domain']][row['age']] = row['cnt']
 
 
     result = {}
     for domain in counts:
-        cnts = [counts[domain].get(age, 0) for age in range(8)]
-        min_cnt = min(cnts)
-        max_cnt = max(cnts)
+        raw_cnts = [counts[domain].get(age, 0) for age in range(8)]
+        min_cnt = min(raw_cnts)
+        max_cnt = max(raw_cnts)
 
         def format_count(cnt):
+            # type: (int) -> str
             if cnt == min_cnt:
                 good_bad = 'bad'
             elif cnt == max_cnt:
@@ -96,12 +107,13 @@ def get_realm_day_counts():
 
             return '<td class="number %s">%s</td>' % (good_bad, cnt)
 
-        cnts = ''.join(map(format_count, cnts))
+        cnts = ''.join(map(format_count, raw_cnts))
         result[domain] = dict(cnts=cnts)
 
     return result
 
 def realm_summary_table(realm_minutes):
+    # type: (Dict[str, float]) -> str
     query = '''
         SELECT
             realm.domain,
@@ -137,7 +149,8 @@ def realm_summary_table(realm_minutes):
                         '/json/send_message',
                         'send_message_backend',
                         '/api/v1/send_message',
-                        '/json/update_pointer'
+                        '/json/update_pointer',
+                        '/json/users/me/pointer'
                     )
                 AND
                     last_visit > now() - interval '1 day'
@@ -166,8 +179,9 @@ def realm_summary_table(realm_minutes):
                         ua.query in (
                             '/json/send_message',
                             'send_message_backend',
-                           '/api/v1/send_message',
-                            '/json/update_pointer'
+                            '/api/v1/send_message',
+                            '/json/update_pointer',
+                            '/json/users/me/pointer'
                         )
                     GROUP by realm.id, up.email
                     HAVING max(last_visit) between
@@ -187,7 +201,8 @@ def realm_summary_table(realm_minutes):
                         '/json/send_message',
                         '/api/v1/send_message',
                         'send_message_backend',
-                        '/json/update_pointer'
+                        '/json/update_pointer',
+                        '/json/users/me/pointer'
                     )
                 AND
                     up.realm_id = realm.id
@@ -211,10 +226,10 @@ def realm_summary_table(realm_minutes):
             row['history'] = ''
 
     # augment data with realm_minutes
-    total_hours = 0
+    total_hours = 0.0
     for row in rows:
         domain = row['domain']
-        minutes = realm_minutes.get(domain, 0)
+        minutes = realm_minutes.get(domain, 0.0)
         hours = minutes / 60.0
         total_hours += hours
         row['hours'] = str(int(hours))
@@ -229,6 +244,7 @@ def realm_summary_table(realm_minutes):
 
     # Count active sites
     def meets_goal(row):
+        # type: (Dict[str, int]) -> bool
         return row['active_user_count'] >= 5
 
     num_active_sites = len(list(filter(meets_goal, rows)))
@@ -237,10 +253,12 @@ def realm_summary_table(realm_minutes):
     total_active_user_count = 0
     total_user_profile_count = 0
     total_bot_count = 0
+    total_at_risk_count = 0
     for row in rows:
         total_active_user_count += int(row['active_user_count'])
         total_user_profile_count += int(row['user_profile_count'])
         total_bot_count += int(row['bot_count'])
+        total_at_risk_count += int(row['at_risk_count'])
 
 
     rows.append(dict(
@@ -248,7 +266,8 @@ def realm_summary_table(realm_minutes):
         active_user_count=total_active_user_count,
         user_profile_count=total_user_profile_count,
         bot_count=total_bot_count,
-        hours=int(total_hours)
+        hours=int(total_hours),
+        at_risk_count=total_at_risk_count,
     ))
 
     content = loader.render_to_string(
@@ -259,6 +278,7 @@ def realm_summary_table(realm_minutes):
 
 
 def user_activity_intervals():
+    # type: () -> Tuple[mark_safe, Dict[str, float]]
     day_end = timestamp_to_datetime(time.time())
     day_start = day_end - timedelta(hours=24)
 
@@ -298,7 +318,7 @@ def user_activity_intervals():
 
             total_duration += duration
             realm_duration += duration
-            output += "  %-*s%s\n" % (37, email, duration, )
+            output += "  %-*s%s\n" % (37, email, duration)
 
         realm_minutes[domain] = realm_duration.total_seconds() / 60
 
@@ -309,6 +329,7 @@ def user_activity_intervals():
     return content, realm_minutes
 
 def sent_messages_report(realm):
+    # type: (str) -> str
     title = 'Recently sent messages for ' + realm
 
     cols = [
@@ -376,7 +397,9 @@ def sent_messages_report(realm):
     return make_table(title, cols, rows)
 
 def ad_hoc_queries():
+    # type: () -> List[Dict[str, str]]
     def get_page(query, cols, title):
+        # type: (str, List[str], str) -> Dict[str, str]
         cursor = connection.cursor()
         cursor.execute(query)
         rows = cursor.fetchall()
@@ -384,6 +407,7 @@ def ad_hoc_queries():
         cursor.close()
 
         def fix_rows(i, fixup_func):
+            # type: (int, Union[Callable[[Realm], mark_safe], Callable[[datetime], str]]) -> None
             for row in rows:
                 row[i] = fixup_func(row[i])
 
@@ -546,8 +570,9 @@ def ad_hoc_queries():
 @zulip_internal
 @has_request_variables
 def get_activity(request):
-    duration_content, realm_minutes = user_activity_intervals()
-    counts_content = realm_summary_table(realm_minutes)
+    # type: (HttpRequest) -> HttpResponse
+    duration_content, realm_minutes = user_activity_intervals() # type: Tuple[mark_safe, Dict[str, float]]
+    counts_content = realm_summary_table(realm_minutes) # type: str
     data = [
         ('Counts', counts_content),
         ('Durations', duration_content),
@@ -560,10 +585,11 @@ def get_activity(request):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title, is_home=True),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 def get_user_activity_records_for_realm(realm, is_bot):
+    # type: (str, bool) -> QuerySet
     fields = [
         'user_profile__full_name',
         'user_profile__email',
@@ -583,6 +609,7 @@ def get_user_activity_records_for_realm(realm, is_bot):
     return records
 
 def get_user_activity_records_for_email(email):
+    # type: (str) -> List[QuerySet]
     fields = [
         'user_profile__full_name',
         'query',
@@ -599,6 +626,7 @@ def get_user_activity_records_for_email(email):
     return records
 
 def raw_user_activity_table(records):
+    # type: (List[QuerySet]) -> str
     cols = [
         'query',
         'client',
@@ -607,6 +635,7 @@ def raw_user_activity_table(records):
     ]
 
     def row(record):
+        # type: (QuerySet) -> List[Any]
         return [
                 record.query,
                 record.client.name,
@@ -619,8 +648,15 @@ def raw_user_activity_table(records):
     return make_table(title, cols, rows)
 
 def get_user_activity_summary(records):
-    summary = {}
+    # type: (List[QuerySet]) -> Dict[str, Dict[str, Any]]
+    #: `Any` used above should be `Union(int, datetime)`.
+    #: However current version of `Union` does not work inside other function.
+    #: We could use something like:
+    # `Union[Dict[str, Dict[str, int]], Dict[str, Dict[str, datetime]]]`
+    #: but that would require this long `Union` to carry on throughout inner functions.
+    summary = {} # type: Dict[str, Dict[str, Any]]
     def update(action, record):
+        # type: (str, QuerySet) -> None
         if action not in summary:
             summary[action] = dict(
                     count=record.count,
@@ -654,7 +690,7 @@ def get_user_activity_summary(records):
             update('website', record)
         if ('send_message' in query) or re.search('/api/.*/external/.*', query):
             update('send', record)
-        if query in ['/json/update_pointer', '/api/v1/update_pointer']:
+        if query in ['/json/update_pointer', '/json/users/me/pointer', '/api/v1/update_pointer']:
             update('pointer', record)
         update(client, record)
 
@@ -662,24 +698,28 @@ def get_user_activity_summary(records):
     return summary
 
 def format_date_for_activity_reports(date):
+    # type: (Optional[datetime]) -> str
     if date:
         return date.astimezone(eastern_tz).strftime('%Y-%m-%d %H:%M')
     else:
         return ''
 
 def user_activity_link(email):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_user_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(email=email))
     email_link = '<a href="%s">%s</a>' % (url, email)
     return mark_safe(email_link)
 
 def realm_activity_link(realm):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_realm_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(realm=realm))
     realm_link = '<a href="%s">%s</a>' % (url, realm)
     return mark_safe(realm_link)
 
 def realm_client_table(user_summaries):
+    # type: (Dict[str, Dict[str, Dict[str, Any]]]) -> str
     exclude_keys = [
             'internal',
             'name',
@@ -724,6 +764,7 @@ def realm_client_table(user_summaries):
     return make_table(title, cols, rows)
 
 def user_activity_summary_table(user_summary):
+    # type: (Dict[str, Dict[str, Any]]) -> str
     rows = []
     for k, v in user_summary.items():
         if k == 'name':
@@ -750,28 +791,33 @@ def user_activity_summary_table(user_summary):
     return make_table(title, cols, rows)
 
 def realm_user_summary_table(all_records, admin_emails):
+    # type: (List[QuerySet], Set[text_type]) -> Tuple[Dict[str, Dict[str, Any]], str]
     user_records = {}
 
     def by_email(record):
+        # type: (QuerySet) -> str
         return record.user_profile.email
 
     for email, records in itertools.groupby(all_records, by_email):
         user_records[email] = get_user_activity_summary(list(records))
 
     def get_last_visit(user_summary, k):
+        # type: (Dict[str, Dict[str, datetime]], str) -> Optional[datetime]
         if k in user_summary:
             return user_summary[k]['last_visit']
         else:
             return None
 
     def get_count(user_summary, k):
+        # type: (Dict[str, Dict[str, str]], str) -> str
         if k in user_summary:
             return user_summary[k]['count']
         else:
             return ''
 
     def is_recent(val):
-        age = datetime.now(val.tzinfo) - val
+        # type: (Optional[datetime]) -> bool
+        age = datetime.now(val.tzinfo) - val # type: ignore # datetie.now tzinfo bug.
         return age.total_seconds() < 5 * 60
 
     rows = []
@@ -781,18 +827,19 @@ def realm_user_summary_table(all_records, admin_emails):
         cells = [user_summary['name'], email_link, sent_count]
         row_class = ''
         for field in ['use', 'send', 'pointer', 'desktop', 'ZulipiOS', 'Android']:
-            val = get_last_visit(user_summary, field)
+            visit = get_last_visit(user_summary, field)
             if field == 'use':
-                if val and is_recent(val):
+                if visit and is_recent(visit):
                     row_class += ' recently_active'
                 if email in admin_emails:
                     row_class += ' admin'
-            val = format_date_for_activity_reports(val)
+            val = format_date_for_activity_reports(visit)
             cells.append(val)
         row = dict(cells=cells, row_class=row_class)
         rows.append(row)
 
     def by_used_time(row):
+        # type: (Dict[str, Sequence[str]]) -> str
         return row['cells'][3]
 
     rows = sorted(rows, key=by_used_time, reverse=True)
@@ -816,9 +863,9 @@ def realm_user_summary_table(all_records, admin_emails):
 
 @zulip_internal
 def get_realm_activity(request, realm):
-    data = []
-    all_records = {}
-    all_user_records = {}
+    # type: (HttpRequest, str) -> HttpResponse
+    data = [] # type: List[Tuple[str, str]]
+    all_user_records = {} # type: Dict[str, Any]
 
     try:
         admins = get_realm(realm).get_admin_users()
@@ -828,8 +875,7 @@ def get_realm_activity(request, realm):
     admin_emails = {admin.email for admin in admins}
 
     for is_bot, page_title in [(False,  'Humans'), (True, 'Bots')]:
-        all_records = get_user_activity_records_for_realm(realm, is_bot)
-        all_records = list(all_records)
+        all_records = list(get_user_activity_records_for_realm(realm, is_bot))
 
         user_records, content = realm_user_summary_table(all_records, admin_emails)
         all_user_records.update(user_records)
@@ -854,14 +900,15 @@ def get_realm_activity(request, realm):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, realm_link=realm_link, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 @zulip_internal
 def get_user_activity(request, email):
+    # type: (HttpRequest, str) -> HttpResponse
     records = get_user_activity_records_for_email(email)
 
-    data = []
+    data = [] # type: List[Tuple[str, str]]
     user_summary = get_user_activity_summary(records)
     content = user_activity_summary_table(user_summary)
 
@@ -874,5 +921,5 @@ def get_user_activity(request, email):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )

api/MANIFEST.in

@@ -8,3 +8,4 @@ include examples/unsubscribe
 include examples/list-members
 include examples/list-subscriptions
 include examples/print-messages
+include examples/recent-messages

api/bin/zulip-send

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 # zulip-send -- Sends a message to the specified recipients.
 

api/examples/create-user

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012-2014 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 from os import path
 import optparse
@@ -46,9 +47,9 @@ parser.add_option('--new-short-name')
 
 client = zulip.init_from_options(options)
 
-print client.create_user({
+print(client.create_user({
         'email': options.new_email,
         'password': options.new_password,
         'full_name': options.new_full_name,
         'short_name': options.new_short_name
-        })
+        }))

api/examples/edit-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -53,4 +54,4 @@ if options.subject != "":
     message_data["subject"] = options.subject
 if options.content != "":
     message_data["content"] = options.content
-print client.update_message(message_data)
+print(client.update_message(message_data))

api/examples/get-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,4 +44,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.get_streams(include_public=True, include_subscribed=False)
+print(client.get_streams(include_public=True, include_subscribed=False))

api/examples/list-members

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2014 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 for user in client.get_members()["members"]:
-    print user["full_name"], user["email"]
+    print(user["full_name"], user["email"])

api/examples/list-subscriptions

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.list_subscriptions()
+print(client.list_subscriptions())

api/examples/print-events

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,7 +44,7 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 def print_event(event):
-    print event
+    print(event)
 
 # This is a blocking call, and will continuously poll for new events
 # Note also the filter here is messages to the stream Denmark; if you

api/examples/print-messages

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -43,7 +44,7 @@ parser.add_option_group(zulip.generate_option_group(parser))
 client = zulip.init_from_options(options)
 
 def print_message(message):
-    print message
+    print(message)
 
 # This is a blocking call, and will continuously poll for new messages
 client.call_on_each_message(print_message)

api/examples/print-next-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -42,4 +43,4 @@ parser.add_option_group(zulip.generate_option_group(parser))
 
 client = zulip.init_from_options(options)
 
-print client.get_messages({})
+print(client.get_messages({}))

api/examples/recent-messages

@@ -0,0 +1,61 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright © 2012 Zulip, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from __future__ import print_function
+import sys
+import os
+import json
+import optparse
+
+usage = """recent-messages [options] --count=<no. of previous messages> --user=<sender's email address> --api-key=<sender's api key>
+
+Prints out last count messages recieved by the indicated bot or user
+
+Example: recent-messages --count=101 --user=username@example.com --api-key=a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5
+
+You can omit --user and --api-key arguments if you have a properly set up ~/.zuliprc
+"""
+sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
+import zulip
+
+parser = optparse.OptionParser(usage=usage)
+parser.add_option('--count', default=100)
+parser.add_option_group(zulip.generate_option_group(parser))
+(options, args) = parser.parse_args()
+
+client = zulip.init_from_options(options)
+
+req = {
+    'narrow': [["stream", "Denmark"]],
+    'num_before': options.count,
+    'num_after': 0,
+    'anchor': 1000000000,
+    'apply_markdown': False
+}
+
+old_messages = client.do_api_query(req, zulip.API_VERSTRING + 'messages', method='GET')
+if 'messages' in old_messages:
+    for message in old_messages['messages']:
+        print(json.dumps(message, indent=4))
+else:
+    print([])

api/examples/send-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -54,4 +55,4 @@ message_data = {
     "subject": options.subject,
     "to": args,
 }
-print client.send_message(message_data)
+print(client.send_message(message_data))

api/examples/subscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -45,8 +46,8 @@ parser.add_option('--streams', default='')
 client = zulip.init_from_options(options)
 
 if options.streams == "":
-    print >>sys.stderr, "Usage:", parser.usage
+    print("Usage:", parser.usage, file=sys.stderr)
     sys.exit(1)
 
-print client.add_subscriptions([{"name": stream_name} for stream_name in
-                                options.streams.split()])
+print(client.add_subscriptions([{"name": stream_name} for stream_name in
+                                options.streams.split()]))

api/examples/unsubscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.
@@ -21,6 +21,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import sys
 import os
 import optparse
@@ -45,7 +46,7 @@ parser.add_option('--streams', default='')
 client = zulip.init_from_options(options)
 
 if options.streams == "":
-    print >>sys.stderr, "Usage:", parser.usage
+    print("Usage:", parser.usage, file=sys.stderr)
     sys.exit(1)
 
-print client.remove_subscriptions(options.streams.split())
+print(client.remove_subscriptions(options.streams.split()))

api/integrations/asana/zulip_asana_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/asana/zulip_asana_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Asana integration for Zulip
@@ -30,6 +30,7 @@
 #
 # python-dateutil is a dependency for this script.
 
+from __future__ import print_function
 import base64
 from datetime import datetime, timedelta
 
@@ -37,16 +38,16 @@ import json
 import logging
 import os
 import time
-import urllib2
+from six.moves import urllib
 
 import sys
 
 try:
     import dateutil.parser
     import dateutil.tz
-except ImportError, e:
-    print >>sys.stderr, e
-    print >>sys.stderr, "Please install the python-dateutil package."
+except ImportError as e:
+    print(e, file=sys.stderr)
+    print("Please install the python-dateutil package.", file=sys.stderr)
     exit(1)
 
 sys.path.insert(0, os.path.dirname(__file__))
@@ -74,8 +75,8 @@ def fetch_from_asana(path):
     headers = {"Authorization": "Basic %s" % auth}
 
     url = "https://app.asana.com/api/1.0" + path
-    request = urllib2.Request(url, None, headers)
-    result = urllib2.urlopen(request)
+    request = urllib.request.Request(url, None, headers)
+    result = urllib.request.urlopen(request)
 
     return json.load(result)
 
@@ -189,7 +190,7 @@ def since():
                 timestamp = float(datestring)
                 max_timestamp_processed = datetime.fromtimestamp(timestamp)
                 logging.info("Reading from resume file: " + datestring)
-        except (ValueError,IOError) as e:
+        except (ValueError, IOError) as e:
             logging.warn("Could not open resume file: %s" % (
                     e.message or e.strerror,))
             max_timestamp_processed = default_since()

api/integrations/basecamp/zulip_basecamp_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/basecamp/zulip_basecamp_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Basecamp activity
@@ -26,6 +26,7 @@
 # or preferably on a server.
 # You may need to install the python-requests library.
 
+from __future__ import absolute_import
 import requests
 import logging
 import time
@@ -33,7 +34,8 @@ import re
 import sys
 import os
 from datetime import datetime, timedelta
-from HTMLParser import HTMLParser
+from six.moves.html_parser import HTMLParser
+import six
 
 sys.path.insert(0, os.path.dirname(__file__))
 import zulip_basecamp_config as config
@@ -80,7 +82,7 @@ def check_permissions():
 
 # builds the message dict for sending a message with the Zulip API
 def build_message(event):
-    if not (event.has_key('bucket') and event.has_key('creator') and event.has_key('html_url')):
+    if not ('bucket' in event and 'creator' in event and 'html_url' in event):
         logging.error("Perhaps the Basecamp API changed behavior? "
                       "This event doesn't have the expected format:\n%s" %(event,))
         return None
@@ -92,7 +94,7 @@ def build_message(event):
     action = htmlParser.unescape(re.sub(r"<[^<>]+>", "", event.get('action', '')))
     target = htmlParser.unescape(event.get('target', ''))
     # Some events have "excerpts", which we blockquote
-    excerpt = htmlParser.unescape(event.get('excerpt',''))
+    excerpt = htmlParser.unescape(event.get('excerpt', ''))
     if excerpt.strip() == "":
         message = '**%s** %s [%s](%s).' % (event['creator']['name'], action, target, event['html_url'])
     else:
@@ -116,13 +118,13 @@ def run_mirror():
         since = re.search(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}-\d{2}:\d{2}", since)
         assert since, "resume file does not meet expected format"
         since = since.string
-    except (AssertionError,IOError) as e:
+    except (AssertionError, IOError) as e:
         logging.warn("Could not open resume file: %s" % (e.message or e.strerror,))
         since = (datetime.utcnow() - timedelta(hours=config.BASECAMP_INITIAL_HISTORY_HOURS)).isoformat() + "-00:00"
     try:
         # we use an exponential backoff approach when we get 429 (Too Many Requests).
         sleepInterval = 1
-        while 1:
+        while True:
             time.sleep(sleepInterval)
             response = requests.get("https://basecamp.com/%s/api/v1/events.json" % (config.BASECAMP_ACCOUNT_ID),
                                     params={'since': since},
@@ -170,7 +172,7 @@ def run_mirror():
 
 
 if __name__ == "__main__":
-    if not isinstance(config.RESUME_FILE, basestring):
+    if not isinstance(config.RESUME_FILE, six.string_types):
         sys.stderr("RESUME_FILE path not given; refusing to continue")
     check_permissions()
     if config.LOG_FILE:

api/integrations/codebase/zulip_codebase_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/codebase/zulip_codebase_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Codebase HQ activity
@@ -29,6 +29,8 @@
 #
 # python-dateutil is a dependency for this script.
 
+from __future__ import print_function
+from __future__ import absolute_import
 import requests
 import logging
 import time
@@ -36,13 +38,14 @@ import sys
 import os
 
 from datetime import datetime, timedelta
+import six
 
 
 try:
     import dateutil.parser
-except ImportError, e:
-    print >>sys.stderr, e
-    print >>sys.stderr, "Please install the python-dateutil package."
+except ImportError as e:
+    print(e, file=sys.stderr)
+    print("Please install the python-dateutil package.", file=sys.stderr)
     exit(1)
 
 sys.path.insert(0, os.path.dirname(__file__))
@@ -132,7 +135,7 @@ def handle_event(event):
             content = "%s deleted branch %s from %s" % (actor_name, branch, project)
         else:
             if new_ref:
-                branch = "new branch %s" % (branch, )
+                branch = "new branch %s" % (branch,)
             content = "%s pushed %s commit(s) to %s in project %s:\n\n" % \
                         (actor_name, num_commits, branch, project)
             for commit in raw_props.get('commits'):
@@ -271,13 +274,13 @@ def run_mirror():
         else:
             timestamp = int(timestamp, 10)
             since = datetime.fromtimestamp(timestamp)
-    except (ValueError,IOError) as e:
+    except (ValueError, IOError) as e:
         logging.warn("Could not open resume file: %s" % (e.message or e.strerror,))
         since = default_since()
 
     try:
         sleepInterval = 1
-        while 1:
+        while True:
             events = make_api_call("activity")[::-1]
             if events is not None:
                 sleepInterval = 1
@@ -314,7 +317,7 @@ def check_permissions():
         sys.stderr(e)
 
 if __name__ == "__main__":
-    if not isinstance(config.RESUME_FILE, basestring):
+    if not isinstance(config.RESUME_FILE, six.string_types):
         sys.stderr("RESUME_FILE path not given; refusing to continue")
     check_permissions()
     if config.LOG_FILE:

api/integrations/git/post-receive

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-receive hook.
@@ -29,13 +29,14 @@
 # For example:
 #  aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master
 
+from __future__ import absolute_import
 import os
 import sys
 import subprocess
 import os.path
 
 sys.path.insert(0, os.path.dirname(__file__))
-import zulip_git_config as config
+from . import zulip_git_config as config
 VERSION = "0.9"
 
 if config.ZULIP_API_PATH is not None:

api/integrations/git/zulip_git_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/hg/zulip-changegroup.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip hook for Mercurial changeset pushes.

api/integrations/jira/org/humbug/jira/ZulipListener.groovy

@@ -67,7 +67,7 @@ class ZulipListener extends AbstractIssueEventListener {
                                   author, issueUrlMd, comment)
           break
         case ISSUE_CREATED_ID:
-          content = String.format("%s **created** %s priority %s, assigned to **%s**: \n\n> %s",
+          content = String.format("%s **created** %s priority %s, assigned to @**%s**: \n\n> %s",
                                   author, issueUrlMd, event.issue.priorityObject.name,
                                   assignee, title)
           break

api/integrations/nagios/nagios-notify-zulip

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import optparse
 import zulip
 

api/integrations/perforce/git_p4.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # git-p4.py -- A tool for bidirectional operation between a Perforce depot and git.
 #
@@ -9,6 +9,7 @@
 #
 from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
 import sys
 import six
 from six.moves import input
@@ -32,7 +33,7 @@ import stat
 try:
     from subprocess import CalledProcessError
 except ImportError:
-    # from python2.7:subprocess.py
+    # from python:subprocess.py
     # Exception classes used by this module.
     class CalledProcessError(Exception):
         """This exception is raised when a process run by check_call() returns
@@ -2346,7 +2347,7 @@ class P4Sync(Command, P4UserMap):
             self.labels[newestChange] = [output, revisions]
 
         if self.verbose:
-            print("Label changes: %s" % self.labels.keys())
+            print("Label changes: %s" % (list(self.labels.keys()),))
 
     # Import p4 labels as git tags. A direct mapping does not
     # exist, so assume that if all the files are at the same revision
@@ -2779,7 +2780,7 @@ class P4Sync(Command, P4UserMap):
                 if short in branches:
                     self.p4BranchesInGit = [ short ]
             else:
-                self.p4BranchesInGit = branches.keys()
+                self.p4BranchesInGit = list(branches.keys())
 
             if len(self.p4BranchesInGit) > 1:
                 if not self.silent:
@@ -2921,7 +2922,7 @@ class P4Sync(Command, P4UserMap):
                     b = b[len(self.projectName):]
                 self.createdBranches.add(b)
 
-        self.tz = "%+03d%02d" % (- time.timezone / 3600, ((- time.timezone % 3600) / 60))
+        self.tz = "%+03d%02d" % (- time.timezone / 3600, ((- time.timezone % 3600) // 60))
 
         self.importProcess = subprocess.Popen(["git", "fast-import"],
                                               stdin=subprocess.PIPE,
@@ -3214,7 +3215,7 @@ commands = {
 
 def main():
     if len(sys.argv[1:]) == 0:
-        printUsage(commands.keys())
+        printUsage(list(commands.keys()))
         sys.exit(2)
 
     cmdName = sys.argv[1]
@@ -3224,7 +3225,7 @@ def main():
     except KeyError:
         print("unknown command %s" % cmdName)
         print("")
-        printUsage(commands.keys())
+        printUsage(list(commands.keys()))
         sys.exit(2)
 
     options = cmd.options

api/integrations/perforce/zulip_change-commit.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2012-2014 Zulip, Inc.

api/integrations/perforce/zulip_perforce_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/rss/rss-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # RSS integration for Zulip
@@ -23,16 +23,17 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import calendar
 import errno
 import hashlib
-from HTMLParser import HTMLParser
+from six.moves.html_parser import HTMLParser
 import logging
 import optparse
 import os
 import sys
 import time
-import urlparse
+from six.moves import urllib
 
 import feedparser
 import zulip
@@ -87,7 +88,7 @@ def mkdir_p(path):
     # Python doesn't have an analog to `mkdir -p` < Python 3.2.
     try:
         os.makedirs(path)
-    except OSError, e:
+    except OSError as e:
         if e.errno == errno.EEXIST and os.path.isdir(path):
             pass
         else:
@@ -97,7 +98,7 @@ try:
     mkdir_p(opts.data_dir)
 except OSError:
     # We can't write to the logfile, so just print and give up.
-    print >>sys.stderr, "Unable to store RSS data at %s." % (opts.data_dir,)
+    print("Unable to store RSS data at %s." % (opts.data_dir,), file=sys.stderr)
     exit(1)
 
 log_file = os.path.join(opts.data_dir, "rss-bot.log")
@@ -169,7 +170,7 @@ client = zulip.Client(email=opts.email, api_key=opts.api_key,
 first_message = True
 
 for feed_url in feed_urls:
-    feed_file = os.path.join(opts.data_dir, urlparse.urlparse(feed_url).netloc)
+    feed_file = os.path.join(opts.data_dir, urllib.parse.urlparse(feed_url).netloc)
 
     try:
         with open(feed_file, "r") as f:

api/integrations/svn/post-commit

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-commit hook.

api/integrations/svn/zulip_svn_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/twitter/twitter-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Twitter integration for Zulip
@@ -23,10 +23,11 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import os
 import sys
 import optparse
-import ConfigParser
+import six.moves.configparser
 
 import zulip
 VERSION = "0.9"
@@ -85,14 +86,14 @@ if not options.twitter_id:
     parser.error('You must specify --twitter-id')
 
 try:
-    config = ConfigParser.ConfigParser()
+    config = six.moves.configparser.ConfigParser()
     config.read(CONFIGFILE)
 
     consumer_key = config.get('twitter', 'consumer_key')
     consumer_secret = config.get('twitter', 'consumer_secret')
     access_token_key = config.get('twitter', 'access_token_key')
     access_token_secret = config.get('twitter', 'access_token_secret')
-except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+except (six.moves.configparser.NoSectionError, six.moves.configparser.NoOptionError):
    parser.error("Please provide a ~/.zulip_twitterrc")
 
 if not consumer_key or not consumer_secret or not access_token_key or not access_token_secret:
@@ -112,17 +113,17 @@ api = twitter.Api(consumer_key=consumer_key,
 user = api.VerifyCredentials()
 
 if not user.GetId():
-    print "Unable to log in to twitter with supplied credentials. Please double-check and try again"
+    print("Unable to log in to twitter with supplied credentials. Please double-check and try again")
     sys.exit()
 
 try:
     since_id = config.getint('twitter', 'since_id')
-except ConfigParser.NoOptionError:
+except six.moves.configparser.NoOptionError:
     since_id = -1
 
 try:
     user_id = config.get('twitter', 'user_id')
-except ConfigParser.NoOptionError:
+except six.moves.configparser.NoOptionError:
     user_id = options.twitter_id
 
 client = zulip.Client(
@@ -154,7 +155,7 @@ for status in statuses[::-1][:options.limit_tweets]:
 
     if ret['result'] == 'error':
         # If sending failed (e.g. no such stream), abort and retry next time
-        print "Error sending message to zulip: %s" % ret['msg']
+        print("Error sending message to zulip: %s" % ret['msg'])
         break
     else:
         since_id = status.GetId()

api/integrations/twitter/twitter-search-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Twitter search integration for Zulip
@@ -23,10 +23,11 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
 import os
 import sys
 import optparse
-import ConfigParser
+import six.moves.configparser
 
 import zulip
 VERSION = "0.9"
@@ -107,14 +108,14 @@ if not opts.search_terms:
     parser.error('You must specify a search term.')
 
 try:
-    config = ConfigParser.ConfigParser()
+    config = six.moves.configparser.ConfigParser()
     config.read(CONFIGFILE)
 
     consumer_key = config.get('twitter', 'consumer_key')
     consumer_secret = config.get('twitter', 'consumer_secret')
     access_token_key = config.get('twitter', 'access_token_key')
     access_token_secret = config.get('twitter', 'access_token_secret')
-except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+except (six.moves.configparser.NoSectionError, six.moves.configparser.NoOptionError):
    parser.error("Please provide a ~/.zulip_twitterrc")
 
 if not (consumer_key and consumer_secret and access_token_key and access_token_secret):
@@ -122,7 +123,7 @@ if not (consumer_key and consumer_secret and access_token_key and access_token_s
 
 try:
     since_id = config.getint('search', 'since_id')
-except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+except (six.moves.configparser.NoOptionError, six.moves.configparser.NoSectionError):
     since_id = 0
 
 try:
@@ -138,8 +139,8 @@ api = twitter.Api(consumer_key=consumer_key,
 user = api.VerifyCredentials()
 
 if not user.GetId():
-    print "Unable to log in to Twitter with supplied credentials.\
-Please double-check and try again."
+    print("Unable to log in to Twitter with supplied credentials.\
+Please double-check and try again.")
     sys.exit()
 
 client = zulip.Client(
@@ -182,7 +183,7 @@ for status in statuses[::-1][:opts.limit_tweets]:
 
     if ret['result'] == 'error':
         # If sending failed (e.g. no such stream), abort and retry next time
-        print "Error sending message to zulip: %s" % ret['msg']
+        print("Error sending message to zulip: %s" % ret['msg'])
         break
     else:
         since_id = status.GetId()

api/setup.py

@@ -1,7 +1,9 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 from __future__ import print_function
+from typing import Any, Generator, List, Tuple
+
 import os
 import sys
 
@@ -16,6 +18,7 @@ def version():
     return version
 
 def recur_expand(target_root, dir):
+    # type: (Any, Any) -> Generator[Tuple[str, List[str]], None, None]
     for root, _, files in os.walk(dir):
         paths = [os.path.join(root, f) for f in files]
         if len(paths):
@@ -40,7 +43,7 @@ package_info = dict(
     data_files=[('share/zulip/examples', ["examples/zuliprc", "examples/send-message", "examples/subscribe",
                                            "examples/get-public-streams", "examples/unsubscribe",
                                            "examples/list-members", "examples/list-subscriptions",
-                                           "examples/print-messages"])] + \
+                                           "examples/print-messages", "examples/recent-messages"])] + \
         list(recur_expand('share/zulip', 'integrations/')),
     scripts=["bin/zulip-send"],
 )
@@ -48,6 +51,8 @@ package_info = dict(
 setuptools_info = dict(
     install_requires=['requests>=0.12.1',
                       'simplejson',
+                      'six',
+                      'typing',
     ],
 )
 
@@ -65,7 +70,7 @@ except ImportError:
         sys.exit(1)
     try:
         import requests
-        assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
+        assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1')) # type: ignore # https://github.com/JukkaL/mypy/issues/1165
     except (ImportError, AssertionError):
         print("requests >=0.12.1 is not installed", file=sys.stderr)
         sys.exit(1)

api/zulip/__init__.py

@@ -22,31 +22,32 @@
 
 from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
 import simplejson
 import requests
 import time
 import traceback
-import urlparse
 import sys
 import os
 import optparse
 import platform
-import urllib
 import random
 from distutils.version import LooseVersion
 
 from six.moves.configparser import SafeConfigParser
+from six.moves import urllib
 import logging
 import six
+from typing import Any, Dict
 
 
-__version__ = "0.2.4"
+__version__ = "0.2.5"
 
 logger = logging.getLogger(__name__)
 
 # Check that we have a recent enough version
 # Older versions don't provide the 'json' attribute on responses.
-assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
+assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1')) # type: ignore # https://github.com/python/mypy/issues/1165 and https://github.com/python/typeshed/pull/206
 # In newer versions, the 'json' attribute is a function, not a property
 requests_json_is_function = callable(requests.Response.json)
 
@@ -164,7 +165,7 @@ class Client(object):
             config_file = get_default_config_filename()
         if os.path.exists(config_file):
             config = SafeConfigParser()
-            with file(config_file, 'r') as f:
+            with open(config_file, 'r') as f:
                 config.readfp(f, config_file)
             if api_key is None:
                 api_key = config.get("api", "key")
@@ -245,7 +246,7 @@ class Client(object):
     def do_api_query(self, orig_request, url, method="POST", longpolling = False):
         request = {}
 
-        for (key, val) in orig_request.iteritems():
+        for (key, val) in six.iteritems(orig_request):
             if not (isinstance(val, str) or isinstance(val, six.text_type)):
                 request[key] = simplejson.dumps(val)
             else:
@@ -255,7 +256,7 @@ class Client(object):
             'had_error_retry': False,
             'request': request,
             'failures': 0,
-        }
+        } # type: Dict[str, Any]
 
         def error_retry(error_string):
             if not self.retry_on_errors or query_state["failures"] >= 10:
@@ -289,7 +290,7 @@ class Client(object):
                 kwargs = {kwarg: query_state["request"]}
                 res = requests.request(
                         method,
-                        urlparse.urljoin(self.base_url, url),
+                        urllib.parse.urljoin(self.base_url, url),
                         auth=requests.auth.HTTPBasicAuth(self.email,
                                                          self.api_key),
                         verify=self.tls_verification, timeout=90,
@@ -343,10 +344,15 @@ class Client(object):
                     "status_code": res.status_code}
 
     @classmethod
-    def _register(cls, name, url=None, make_request=(lambda request={}: request),
+    def _register(cls, name, url=None, make_request=None,
                   method="POST", computed_url=None, **query_kwargs):
         if url is None:
             url = name
+        if make_request is None:
+            def make_request(request=None):
+                if request is None:
+                    request = {}
+                return request
         def call(self, *args, **kwargs):
             request = make_request(*args, **kwargs)
             if computed_url is not None:
@@ -357,7 +363,9 @@ class Client(object):
         call.__name__ = name
         setattr(cls, name, call)
 
-    def call_on_each_event(self, callback, event_types=None, narrow=[]):
+    def call_on_each_event(self, callback, event_types=None, narrow=None):
+        if narrow is None:
+            narrow = []
         def do_register():
             while True:
                 if event_types is None:
@@ -425,9 +433,11 @@ def _mk_rm_subs(streams):
 def _mk_deregister(queue_id):
     return {'queue_id': queue_id}
 
-def _mk_events(event_types=None, narrow=[]):
+def _mk_events(event_types=None, narrow=None):
     if event_types is None:
         return dict()
+    if narrow is None:
+        narrow = []
     return dict(event_types=event_types, narrow=narrow)
 
 def _kwargs_to_dict(**kwargs):
@@ -468,7 +478,7 @@ Client._register('list_subscriptions', method='GET', url='users/me/subscriptions
 Client._register('add_subscriptions', url='users/me/subscriptions', make_request=_mk_subs)
 Client._register('remove_subscriptions', method='PATCH', url='users/me/subscriptions', make_request=_mk_rm_subs)
 Client._register('get_subscribers', method='GET',
-                 computed_url=lambda request: 'streams/%s/members' % (urllib.quote(request['stream'], safe=''),),
+                 computed_url=lambda request: 'streams/%s/members' % (urllib.parse.quote(request['stream'], safe=''),),
                  make_request=_kwargs_to_dict)
 Client._register('render_message', method='GET', url='messages/render')
 Client._register('create_user', method='POST', url='users')

assets/favicon/generate

@@ -1,6 +1,8 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
+from __future__ import absolute_import
 import xml.etree.ElementTree as ET
 import subprocess
+from six.moves import range
 
 # Generates the favicon images containing unread message counts.
 
@@ -10,7 +12,7 @@ elems = [tree.getroot().findall(
     ".//*[@id='%s']/{http://www.w3.org/2000/svg}tspan" % (name,))[0]
      for name in ('number_back', 'number_front')]
 
-for i in xrange(1,100):
+for i in range(1, 100):
     # Prepare a modified SVG
     s = '%2d' % (i,)
     for e in elems:

bots/check-mirroring

@@ -1,4 +1,6 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
+from __future__ import print_function
+from __future__ import absolute_import
 import sys
 import time
 import optparse
@@ -7,6 +9,7 @@ import random
 import logging
 import subprocess
 import hashlib
+from six.moves import range
 
 parser = optparse.OptionParser()
 parser.add_option('--verbose',
@@ -100,7 +103,7 @@ def print_status_and_exit(status):
     # e.g. true success and punting due to a SERVNAK, result in a
     # non-alert case, so to give us something unambiguous to check in
     # Nagios, print the exit status.
-    print status
+    print(status)
     sys.exit(status)
 
 def send_zulip(message):
@@ -149,7 +152,7 @@ for (stream, test) in test_streams:
         zephyr_subs_to_add.append((stream, '*', '*'))
 
 actually_subscribed = False
-for tries in xrange(10):
+for tries in range(10):
     try:
         zephyr.init()
         zephyr._z.subAll(zephyr_subs_to_add)
@@ -163,7 +166,7 @@ for tries in xrange(10):
         if missing == 0:
             actually_subscribed = True
             break
-    except IOError, e:
+    except IOError as e:
         if "SERVNAK received" in e:
             logger.error("SERVNAK repeatedly received, punting rest of test")
         else:
@@ -276,7 +279,7 @@ logger.info("Finished receiving Zulip messages!")
 receive_zephyrs()
 logger.info("Finished receiving Zephyr messages!")
 
-all_keys = set(zhkeys.keys() + hzkeys.keys())
+all_keys = set(list(zhkeys.keys()) + list(hzkeys.keys()))
 def process_keys(content_list):
     # Start by filtering out any keys that might have come from
     # concurrent check-mirroring processes

bots/gcal-bot

@@ -1,9 +1,10 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
+from __future__ import print_function
 import sys
 import time
 import datetime
 import optparse
-import urlparse
+from six.moves import urllib
 import itertools
 import traceback
 import os
@@ -56,13 +57,13 @@ except ImportError:
     parser.error('Install python-gdata')
 
 def get_calendar_url():
-    parts = urlparse.urlparse(options.calendar)
+    parts = urllib.parse.urlparse(options.calendar)
     pat = os.path.split(parts.path)
     if pat[1] != 'basic':
         parser.error('The --calendar URL should be the XML "Private Address" ' +
                      'from your calendar settings')
-    return urlparse.urlunparse((parts.scheme, parts.netloc, pat[0] + '/full',
-                                '', 'futureevents=true&orderby=startdate', ''))
+    return urllib.parse.urlunparse((parts.scheme, parts.netloc, pat[0] + '/full',
+                                   '', 'futureevents=true&orderby=startdate', ''))
 
 calendar_url = get_calendar_url()
 
@@ -99,7 +100,7 @@ def send_reminders():
             key = (uid, start)
             if key not in sent:
                 line = '%s starts at %s' % (title, start.strftime('%H:%M'))
-                print 'Sending reminder:', line
+                print('Sending reminder:', line)
                 messages.append(line)
                 keys.add(key)
 

bots/irc-mirror.py

@@ -1,4 +1,4 @@
-#! /usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # EXPERIMENTAL
 # IRC <=> Zulip mirroring bot
@@ -93,11 +93,11 @@ class IRCBot(irc.bot.SingleServerIRCBot):
                 return
             self.dcc_connect(address, port)
 
-usage = """python2.7 irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
+usage = """python irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
 
 Example:
 
-python2.7 irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
+python irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
   --site=https://zulip.example.com --user=irc-bot@example.com
   --api-key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 

bots/jabber_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2014 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person

bots/jabber_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # Copyright (C) 2013 Permabit, Inc.
 # Copyright (C) 2013--2014 Zulip, Inc.
@@ -37,6 +37,7 @@
 #               | other sender|  x  |    |        |
 # public mode   +-------------+-----+----+--------+----
 #               | self sender |     |    |        |
+from typing import Set
 
 import logging
 import threading
@@ -78,11 +79,11 @@ class JabberToZulipBot(ClientXMPP):
             self.nick = jid.username
             jid.resource = "zulip"
         ClientXMPP.__init__(self, jid, password)
-        self.rooms = set()
+        self.rooms = set() # type: Set[str]
         self.rooms_to_join = rooms
         self.add_event_handler("session_start", self.session_start)
         self.add_event_handler("message", self.message)
-        self.zulip = None
+        self.zulip = None # type: zulip.Client
         self.use_ipv6 = False
 
         self.register_plugin('xep_0045') # Jabber chatrooms
@@ -195,7 +196,7 @@ class JabberToZulipBot(ClientXMPP):
 class ZulipToJabberBot(object):
     def __init__(self, zulip_client):
         self.client = zulip_client
-        self.jabber = None
+        self.jabber = None # type: JabberToZulipBot
 
     def set_jabber_client(self, client):
         self.jabber = client
@@ -376,7 +377,7 @@ option does not affect login credentials.'''.replace("\n", " "))
 
     config = SafeConfigParser()
     try:
-        with file(config_file, 'r') as f:
+        with open(config_file, 'r') as f:
             config.readfp(f, config_file)
     except IOError:
         pass

bots/log2zulip

@@ -1,4 +1,5 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
+from __future__ import print_function
 import subprocess
 import os
 import sys
@@ -20,7 +21,7 @@ def mkdir_p(path):
     # Python doesn't have an analog to `mkdir -p` < Python 3.2.
     try:
         os.makedirs(path)
-    except OSError, e:
+    except OSError as e:
         if e.errno == errno.EEXIST and os.path.isdir(path):
             pass
         else:
@@ -55,14 +56,14 @@ def process_logs():
         data_file_path = "/var/tmp/log2zulip.state"
     mkdir_p(os.path.dirname(data_file_path))
     if not os.path.exists(data_file_path):
-        file(data_file_path, "w").write("{}")
-    last_data = ujson.loads(file(data_file_path).read())
+        open(data_file_path, "w").write("{}")
+    last_data = ujson.loads(open(data_file_path).read())
     new_data = {}
     for log_file in log_files:
         file_data = last_data.get(log_file, {})
         if not os.path.exists(log_file):
             # If the file doesn't exist, log an error and then move on to the next file
-            print "Log file %s does not exist!" % (log_file,)
+            print("Log file does not exist or could not stat log file: %s" % (log_file,))
             continue
         length = int(subprocess.check_output(["wc", "-l", log_file]).split()[0])
         if file_data.get("last") is None:
@@ -78,26 +79,26 @@ def process_logs():
             process_lines(new_lines, filename)
             file_data["last"] += len(new_lines)
         new_data[log_file] = file_data
-    file(data_file_path, "w").write(ujson.dumps(new_data))
+    open(data_file_path, "w").write(ujson.dumps(new_data))
 
 if __name__ == "__main__":
     if os.path.exists(lock_path):
-        print "Log2zulip lock held; not doing anything"
+        print("Log2zulip lock held; not doing anything")
         sys.exit(0)
 
     try:
-        file(lock_path, "w").write("1")
+        open(lock_path, "w").write("1")
         zulip_client = zulip.Client(config_file="/etc/log2zulip.zuliprc")
         try:
-            log_files = ujson.loads(file(control_path, "r").read())
+            log_files = ujson.loads(open(control_path, "r").read())
         except Exception:
-            print "Could not load control data from %s" % (control_path,)
+            print("Could not load control data from %s" % (control_path,))
             traceback.print_exc()
             sys.exit(1)
         process_logs()
     finally:
         try:
             os.remove(lock_path)
-        except OSError, IOError:
+        except OSError as IOError:
             pass
 

bots/process_ccache

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import sys
 import subprocess
 import base64
@@ -9,27 +9,27 @@ ccache_data_encoded = sys.argv[3]
 
 # Update the Kerberos ticket cache file
 program_name = "zmirror-%s" % (short_user,)
-with file("/home/zulip/ccache/%s" % (program_name,), "w") as f:
+with open("/home/zulip/ccache/%s" % (program_name,), "w") as f:
     f.write(base64.b64decode(ccache_data_encoded))
 
 # Setup API key
 api_key_path = "/home/zulip/api-keys/%s" % (program_name,)
-file(api_key_path, "w").write(api_key + "\n")
+open(api_key_path, "w").write(api_key + "\n")
 
 # Setup supervisord configuration
 supervisor_path = "/etc/supervisor/conf.d/%s.conf" % (program_name,)
 template = "/home/zulip/zulip/bots/zmirror_private.conf.template"
-template_data = file(template).read()
+template_data = open(template).read()
 session_path = "/home/zulip/zephyr_sessions/%s" % (program_name,)
 
 # Preserve mail zephyrs forwarding setting across rewriting the config file
 
 try:
-    if "--forward-mail-zephyrs" in file(supervisor_path, "r").read():
+    if "--forward-mail-zephyrs" in open(supervisor_path, "r").read():
         template_data = template_data.replace("--use-sessions", "--use-sessions --forward-mail-zephyrs")
 except Exception:
     pass
-file(supervisor_path, "w").write(template_data.replace("USERNAME", short_user))
+open(supervisor_path, "w").write(template_data.replace("USERNAME", short_user))
 
 # Delete your session
 subprocess.check_call(["rm", "-f", session_path])

bots/summarize_stream.py

@@ -1,4 +1,5 @@
 from __future__ import print_function
+from typing import Any, Dict, List
 # This is hacky code to analyze data on our support stream.  The main
 # reusable bits are get_recent_messages and get_words.
 
@@ -50,13 +51,13 @@ def generate_support_stats():
     narrow = 'stream:support'
     count = 2000
     msgs = get_recent_messages(client, narrow, count)
-    msgs_by_topic = collections.defaultdict(list)
+    msgs_by_topic = collections.defaultdict(list) # type: Dict[str, List[Dict[str, Any]]]
     for msg in msgs:
         topic = msg['subject']
         msgs_by_topic[topic].append(msg)
 
-    word_count = collections.defaultdict(int)
-    email_count = collections.defaultdict(int)
+    word_count = collections.defaultdict(int) # type: Dict[str, int]
+    email_count = collections.defaultdict(int) # type: Dict[str, int]
 
     if False:
         for topic in msgs_by_topic:
@@ -64,16 +65,14 @@ def generate_support_stats():
     analyze_messages(msgs, word_count, email_count)
 
     if True:
-        words = word_count.keys()
-        words = [w for w in words if word_count[w] >= 10]
-        words = [w for w in words if len(w) >= 5]
+        words = [w for w in word_count.keys() if word_count[w] >= 10 and len(w) >= 5]
         words = sorted(words, key=lambda w: word_count[w], reverse=True)
         for word in words:
             print(word, word_count[word])
 
     if False:
-        emails = email_count.keys()
-        emails = sorted(emails, key=lambda w: email_count[w], reverse=True)
+        emails = sorted(list(email_count.keys()),
+                        key=lambda w: email_count[w], reverse=True)
         for email in emails:
             print(email, email_count[email])
 

bots/sync-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import sys
 import os
 import logging
@@ -59,7 +59,7 @@ if __name__ == "__main__":
         if public_streams is None:
             continue
 
-        f = file("/home/zulip/public_streams.tmp", "w")
+        f = open("/home/zulip/public_streams.tmp", "w")
         f.write(simplejson.dumps(list(public_streams)) + "\n")
         f.close()
 

bots/tddium-notify-humbug

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person

bots/zephyr_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person

bots/zephyr_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -21,6 +21,7 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 from __future__ import absolute_import
+from typing import Any, List
 
 import sys
 from six.moves import map
@@ -28,7 +29,7 @@ from six.moves import range
 try:
     import simplejson
 except ImportError:
-    import json as simplejson
+    import json as simplejson # type: ignore
 import re
 import time
 import subprocess
@@ -48,6 +49,8 @@ class States(object):
     Startup, ZulipToZephyr, ZephyrToZulip, ChildSending = list(range(4))
 CURRENT_STATE = States.Startup
 
+logger = None # type: logging.Logger
+
 def to_zulip_username(zephyr_username):
     if "@" in zephyr_username:
         (user, realm) = zephyr_username.split("@")
@@ -191,7 +194,7 @@ def zephyr_bulk_subscribe(subs):
 
 def update_subscriptions():
     try:
-        f = file(options.stream_file_path, "r")
+        f = open(options.stream_file_path, "r")
         public_streams = simplejson.loads(f.read())
         f.close()
     except:
@@ -287,7 +290,7 @@ def parse_zephyr_body(zephyr_data):
 
 def parse_crypt_table(zephyr_class, instance):
     try:
-        crypt_table = file(os.path.join(os.environ["HOME"], ".crypt-table"))
+        crypt_table = open(os.path.join(os.environ["HOME"], ".crypt-table"))
     except IOError:
         return None
 
@@ -349,7 +352,7 @@ def process_notice(notice, log):
 
     if zephyr_class == options.nagios_class:
         # Mark that we got the message and proceed
-        with file(options.nagios_path, "w") as f:
+        with open(options.nagios_path, "w") as f:
             f.write("0\n")
         return
 
@@ -468,7 +471,7 @@ def zephyr_load_session_autoretry(session_path):
     backoff = zulip.RandomExponentialBackoff()
     while backoff.keep_going():
         try:
-            session = file(session_path, "r").read()
+            session = open(session_path, "r").read()
             zephyr._z.initialize()
             zephyr._z.load_session(session)
             zephyr.__inited = True
@@ -510,7 +513,7 @@ def zephyr_to_zulip(options):
         if options.nagios_class:
             zephyr_subscribe_autoretry((options.nagios_class, "*", "*"))
         if options.use_sessions:
-            file(options.session_path, "w").write(zephyr._z.dump_session())
+            open(options.session_path, "w").write(zephyr._z.dump_session())
 
     if options.logs_to_resend is not None:
         with open(options.logs_to_resend, 'r') as log:
@@ -804,9 +807,9 @@ def add_zulip_subscriptions(verbose):
         unauthorized = res.get("unauthorized")
         if verbose:
             if already is not None and len(already) > 0:
-                logger.info("\nAlready subscribed to: %s" % (", ".join(already.values()[0]),))
+                logger.info("\nAlready subscribed to: %s" % (", ".join(list(already.values())[0]),))
             if new is not None and len(new) > 0:
-                logger.info("\nSuccessfully subscribed to: %s" % (", ".join(new.values()[0]),))
+                logger.info("\nSuccessfully subscribed to: %s" % (", ".join(list(new.values())[0]),))
             if unauthorized is not None and len(unauthorized) > 0:
                 logger.info("\n" + "\n".join(textwrap.wrap("""\
 The following streams you have NOT been subscribed to,
@@ -857,7 +860,7 @@ def parse_zephyr_subs(verbose=False):
             logger.error("Couldn't find ~/.zephyr.subs!")
         return []
 
-    for line in file(subs_file, "r").readlines():
+    for line in open(subs_file, "r").readlines():
         line = line.strip()
         if len(line) == 0:
             continue
@@ -878,6 +881,7 @@ def parse_zephyr_subs(verbose=False):
     return zephyr_subscriptions
 
 def open_logger():
+    # type: () -> logging.Logger
     if options.log_path is not None:
         log_file = options.log_path
     elif options.forward_class_messages:
@@ -1025,7 +1029,7 @@ if __name__ == "__main__":
 
     signal.signal(signal.SIGINT, die_gracefully)
 
-    (options, args) = parse_args()
+    (options, args) = parse_args() # type: Any, List[str]
 
     logger = open_logger()
     configure_logger(logger, "parent")
@@ -1050,7 +1054,7 @@ Could not find API key file.
 You need to either place your api key file at %s,
 or specify the --api-key-file option.""" % (options.api_key_file,))))
             sys.exit(1)
-        api_key = file(options.api_key_file).read().strip()
+        api_key = open(options.api_key_file).read().strip()
         # Store the API key in the environment so that our children
         # don't need to read it in
         os.environ["HUMBUG_API_KEY"] = api_key
@@ -1113,7 +1117,7 @@ or specify the --api-key-file option.""" % (options.api_key_file,))))
         options.session_path = "/var/tmp/%s" % (options.user,)
 
     if options.forward_from_zulip:
-        child_pid = os.fork()
+        child_pid = os.fork() # type: int
         if child_pid == 0:
             CURRENT_STATE = States.ZulipToZephyr
             # Run the zulip => zephyr mirror in the child

changelog.md

@@ -1,53 +0,0 @@
-# Change Log
-
-All notable changes to this project will be documented in this file.
-
-[Unreleased]
-
-[1.3.10]
-- Added new integration for Travis CI.
-- Added settings option to control maximum file upload size.
-- Added support for running Zulip development environment in Docker.
-- Added easy configuration support for a remote postgres database.
-- Added extensive documentation on scalability, backups, and security.
-- Recent private message threads are now displayed expanded similar to
-  the pre-existing recent topics feature.
-- Made it possible to set LDAP and EMAIL_HOST passwords in
-  /etc/zulip/secrets.conf.
-- Improved the styling for the Administration page and added tabs.
-- Substantially improved loading performance on slow networks by enabling
-  GZIP compression on more assets.
-- Changed the page title in narrowed views to include the current narrow.
-- Fixed several backend performance issues affecting very large realms.
-- Fixed bugs where draft compose content might be lost when reloading site.
-- Fixed support for disabling the "zulip" notifications stream.
-- Fixed missing step in postfix_localmail installation instructions.
-- Fixed several bugs/inconveniences in the production upgrade process.
-- Fixed realm restrictions for servers with a unique, open realm.
-- Substantially cleaned up console logging from run-dev.py.
-
-[1.3.9] - 2015-11-16
-- Fixed buggy #! lines in upgrade scripts.
-
-[1.3.8] - 2015-11-15
-- Added options to the Python api for working with untrusted server certificates.
-- Added a lot of documentation on the development environment and testing.
-- Added partial support for translating the Zulip UI.
-- Migrated installing Node dependencies to use npm.
-- Fixed LDAP integration breaking autocomplete of @-mentions.
-- Fixed admin panel reactivation/deactivation of bots.
-- Fixed inaccurate documentation for downloading the desktop apps.
-- Fixed various minor bugs in production installation process.
-- Fixed security issue where recent history on private streams might
-  be visible to new users (to the Zulip team) who were invited with that
-  private stream as one of their initial streams
-  (https://github.com/zulip/zulip/issues/230).
-- Major preliminary progress towards supporting Python 3.
-
-[1.3.7] - 2015-10-19
-- Turn off desktop and audible notifications for streams by default.
-- Added support for the LDAP authentication integration creating new users.
-- Added new endpoint to support Google auth on mobile.
-- Fixed desktop notifications in modern Firefox.
-- Fixed several installation issues for both production and development environments.
-- Improved documentation for outgoing SMTP and the email mirror integration.

confirmation/__init__.py

@@ -2,23 +2,23 @@
 
 # Copyright: (c) 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 
-# Permission is hereby granted, free of charge, to any person obtaining a                                                       
-# copy of this software and associated documentation files (the                                                                 
-# "Software"), to deal in the Software without restriction, including                                                           
-# without limitation the rights to use, copy, modify, merge, publish, dis-                                                      
-# tribute, sublicense, and/or sell copies of the Software, and to permit                                                        
-# persons to whom the Software is furnished to do so, subject to the fol-                                                       
-# lowing conditions:                                                                                                            
-#                                                                                                                               
-# The above copyright notice and this permission notice shall be included                                                       
-# in all copies or substantial portions of the Software.                                                                        
-#                                                                                                                               
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS                                                       
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-                                                      
-# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT                                                        
-# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,                                                         
-# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,                                                            
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS                                                        
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 
 VERSION = (0, 9, 'pre')

confirmation/management/commands/cleanupconfirmation.py

@@ -4,6 +4,7 @@
 
 __revision__ = '$Id: cleanupconfirmation.py 5 2008-11-18 09:10:12Z jarek.zgoda $'
 
+from typing import Any
 
 from django.core.management.base import NoArgsCommand
 
@@ -14,4 +15,5 @@ class Command(NoArgsCommand):
     help = 'Delete expired confirmations from database'
 
     def handle_noargs(self, **options):
+        # type: (**Any) -> None
         Confirmation.objects.delete_expired_confirmations()

confirmation/models.py

@@ -110,4 +110,4 @@ class Confirmation(models.Model):
         verbose_name_plural = _('confirmation emails')
 
     def __unicode__(self):
-        return _('confirmation email for %s') % self.content_object
+        return _('confirmation email for %s') % (self.content_object,)

confirmation/settings.py

@@ -2,9 +2,10 @@
 
 # Copyright: (c) 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 
+from typing import Any, Dict
+
 __revision__ = '$Id: settings.py 12 2008-11-23 19:38:52Z jarek.zgoda $'
 
 STATUS_ACTIVE = 1
 
-STATUS_FIELDS = {
-}
+STATUS_FIELDS = {} # type: Dict[Any, Any]

confirmation/util.py

@@ -9,4 +9,4 @@ from django.conf import settings
 def get_status_field(app_label, model_name):
     model = '%s.%s' % (app_label, model_name)
     mapping = getattr(settings, 'STATUS_FIELDS', {})
-    return mapping.get(model, 'status')
+    return mapping.get(model, 'status')

confirmation/views.py

@@ -8,11 +8,14 @@ __revision__ = '$Id: views.py 21 2008-12-05 09:21:03Z jarek.zgoda $'
 from django.shortcuts import render_to_response
 from django.template import RequestContext
 from django.conf import settings
+from django.http import HttpRequest, HttpResponse
 
 from confirmation.models import Confirmation
+from zproject.jinja2 import render_to_response
 
 
 def confirm(request, confirmation_key):
+    # type: (HttpRequest, str) -> HttpResponse
     confirmation_key = confirmation_key.lower()
     obj = Confirmation.objects.confirm(confirmation_key)
     confirmed = True
@@ -38,6 +41,5 @@ def confirm(request, confirmation_key):
     ]
     if obj:
         # if we have an object, we can use specific template
-        templates.insert(0, 'confirmation/confirm_%s.html' % obj._meta.model_name)
-    return render_to_response(templates, ctx,
-        context_instance=RequestContext(request))
+        templates.insert(0, 'confirmation/confirm_%s.html' % (obj._meta.model_name,))
+    return render_to_response(templates, ctx, request=request)

corporate/urls.py

@@ -1,9 +1,16 @@
 from django.conf.urls import patterns, url
 from django.views.generic import TemplateView, RedirectView
 
-urlpatterns = patterns('',
+i18n_urlpatterns = [
     # Zephyr/MIT
     url(r'^zephyr/$', TemplateView.as_view(template_name='corporate/zephyr.html')),
     url(r'^mit/$', TemplateView.as_view(template_name='corporate/mit.html')),
     url(r'^zephyr-mirror/$', TemplateView.as_view(template_name='corporate/zephyr-mirror.html')),
-)
+
+    # Terms of service and privacy policy
+    url(r'^terms/$',   TemplateView.as_view(template_name='corporate/terms.html')),
+    url(r'^terms-enterprise/$',  TemplateView.as_view(template_name='corporate/terms-enterprise.html')),
+    url(r'^privacy/$', TemplateView.as_view(template_name='corporate/privacy.html')),
+]
+
+urlpatterns = patterns('', *i18n_urlpatterns)

docs/README.md

@@ -1,23 +1,23 @@
-These docs are written in rST, and are included on the zulip.org website
-as well as on each development installation.  Many of these docs
-have been ported from the internal docs of Zulip Inc.,
-and may need to be updated for use in the open source project.
+These docs are written in [Commonmark
+Markdown](http://commonmark.org/) with a small bit of rST.  We've
+chosen Markdown because it is [easy to
+write](http://commonmark.org/help).  The docs are served in production
+at [zulip.readthedocs.io](https://zulip.readthedocs.io/en/latest/).
 
-To generate HTML docs locally from rST:
+If you want to build the documentation locally (e.g. to test your
+changes), the dependencies are automatically installed as part of
+Zulip development environment provisioning, and you can build the
+documentation using:
 
-   * `pip install sphinx`
-   * In this directory, `make html`. Output appears in a `_build/html` subdirectory.
+```
+cd docs/
+make html
+```
 
-To create rST from MediaWiki input:
-
-   * Use `pandoc -r mediawiki -w rst` on MediaWiki source.
-   * Use unescape.py to remove any leftover HTML entities (often inside <pre>
-     tags and the like).
-
-We can use pandoc to translate mediawiki into reStructuredText, but some things need fixing up:
-
-   * Add page titles.
-   * Review pages for formatting (especially inline code chunks) and content.
-   * Fix wiki links?
-   * Add pages to the table of contents (`index.rst`).
+You can also usually test your changes by pushing a branch to GitHub
+and looking at the content on the GitHub web UI, since GitHub renders
+Markdown.
 
+When editing dependencies for the Zulip documentation, you should edit
+`requirements/docs.txt` (which is used by ReadTheDocs to build the
+documentation quickly, without installing all of Zulip's dependencies).

docs/architecture-overview.md

@@ -0,0 +1,234 @@
+Zulip architectural overview
+============================
+
+Key Codebases
+-------------
+
+The core Zulip application is at
+[<https://github.com/zulip/zulip>](https://github.com/zulip/zulip) and
+is a web application written in Python 2.7 (soon to also support
+Python 3) and using the Django framework. That codebase includes
+server-side code and the web client, as well as Python API bindings
+and most of our integrations with other services and applications (see
+[the directory structure
+guide](https://zulip.readthedocs.io/en/latest/directory-structure.html)).
+
+We maintain several separate repositories for integrations and other
+glue code: a [Hubot adapter](https://github.com/zulip/hubot-zulip);
+integrations with
+[Phabricator](https://github.com/zulip/phabricator-to-zulip),
+[Jenkins](https://github.com/zulip/zulip-jenkins-plugin),
+[Puppet](https://github.com/matthewbarr/puppet-zulip),
+[Redmine](https://github.com/zulip/zulip-redmine-plugin), and
+[Trello](https://github.com/zulip/trello-to-zulip); [node.js API
+bindings](https://github.com/zulip/zulip-node); and our [full-text
+search PostgreSQL extension](https://github.com/zulip/tsearch_extras) .
+
+Our mobile clients are separate code repositories:
+[Android](https://github.com/zulip/zulip-android), [iOS
+(stable)](https://github.com/zulip/zulip-ios) , and [our experimental
+React Native iOS app](https://github.com/zulip/zulip-mobile). Our
+[desktop application](https://github.com/zulip/zulip-desktop) is also a
+separate repository.
+
+We use [Transifex](https://www.transifex.com/zulip/zulip/) to do
+translations.
+
+In this overview we'll mainly discuss the core Zulip server and web
+application.
+
+Usage assumptions and concepts
+------------------------------
+
+Zulip is a real-time web-based chat application meant for companies and
+similar groups ranging in size from a small team to more than a thousand
+users. It features real-time notifications, message persistence and
+search, public group conversations (*streams*), invite-only streams,
+private one-on-one and group conversations, inline image previews, team
+presence/a buddy list, a rich API, Markdown message support, and several
+integrations with other services. The maintainer team aims to support
+users who connect to Zulip using dedicated iOS, Android, Linux, Windows,
+and Mac OS X clients, as well as people using modern web browsers or
+dedicated Zulip API clients.
+
+A server can host multiple Zulip *realms* (organizations) at the same
+domain, each of which is a private chamber with its own users, streams,
+customizations, and so on. This means that one person might be a user of
+multiple Zulip realms. The administrators of a realm can choose whether
+to allow anyone to register an account and join, or only allow people
+who have been invited, or restrict registrations to members of
+particular groups (using email domain names or corporate single-sign-on
+login for verification). For more on scalability and security
+considerations, see [Zulip in
+production](https://github.com/zulip/zulip/blob/master/README.prod.md).
+
+The default Zulip home screen is like a chronologically ordered inbox;
+it displays messages, starting at the oldest message that the user
+hasn't viewed yet. The home screen displays the most recent messages in
+all the streams a user has joined (except for the streams they've
+muted), as well as private messages from other users, in strict
+chronological order. A user can *narrow* to view only the messages in a
+single stream, and can further narrow to focus on a *topic* (thread)
+within that stream. Each narrow has its own URL.
+
+Zulip's philosophy is to provide sensible defaults but give the user
+fine-grained control over their incoming information flow; a user can
+mute topics and streams, and can make fine-grained choices to reduce
+real-time notifications they find irrelevant.
+
+Components
+----------
+
+### Tornado and Django
+
+We use both the [Tornado](http://www.tornadoweb.org) and
+[Django](https://www.djangoproject.com/) Python web frameworks.
+
+Django is the main web application server; Tornado runs the
+server-to-client real-time push system. The app servers are configured
+by the Supervisor configuration (which explains how to start the server
+processes; see "Supervisor" below) and the nginx configuration (which
+explains which HTTP requests get sent to which app server).
+
+Tornado is an asynchronous server and is meant specifically to hold open
+tens of thousands of long-lived (long-polling or websocket) connections
+-- that is to say, routes that maintain a persistent connection from
+every running client. For this reason, it's responsible for event
+(message) delivery, but not much else. We try to avoid any blocking
+calls in Tornado because we don't want to delay delivery to thousands of
+other connections (as this would make Zulip very much not real-time).
+For instance, we avoid doing cache or database queries inside the
+Tornado code paths, since those blocking requests carry a very high
+performance penalty for a single-threaded, asynchronous server.
+
+The parts that are activated relatively rarely (e.g. when people type or
+click on something) are processed by the Django application server. One
+exception to this is that Zulip uses websockets through Tornado to
+minimize latency on the code path for **sending** messages.
+
+### nginx
+
+nginx is the front-end web server to all Zulip traffic; it serves static
+assets and proxies to Django and Tornado. It handles HTTP requests
+according to the rules laid down in the many config files found in
+`zulip/puppet/zulip/files/nginx/`.
+
+`zulip/puppet/zulip/files/nginx/zulip-include-frontend/app` is the most
+important of these files. It explains what happens when requests come in
+from outside.
+
+-   In production, all requests to URLs beginning with `/static/` are
+    served from the corresponding files in `/home/zulip/prod-static/`,
+    and the production build process (`tools/build-release-tarball`)
+    compiles, minifies, and installs the static assets into the
+    `prod-static/` tree form. In development, files are served directly
+    from `/static/` in the git repository.
+-   Requests to `/json/get_events`, `/api/v1/events`, and `/sockjs` are
+    sent to the Tornado server. These are requests to the real-time push
+    system, because the user's web browser sets up a long-lived TCP
+    connection with Tornado to serve as [a channel for push
+    notifications](https://en.wikipedia.org/wiki/Push_technology#Long_Polling).
+    nginx gets the hostname for the Tornado server via
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`.
+-   Requests to all other paths are sent to the Django app via the UNIX
+    socket `unix:/home/zulip/deployments/fastcgi-socket` (defined in
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`). We use
+    `zproject/wsgi.py` to implement FastCGI here (see
+    `django.core.wsgi`).
+
+### Supervisor
+
+We use [supervisord](http://supervisord.org/) to start server processes,
+restart them automatically if they crash, and direct logging.
+
+The config file is
+`zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf`. This is where
+Tornado and Django are set up, as well as a number of background
+processes that process event queues. We use event queues for the kinds
+of tasks that are best run in the background because they are
+expensive (in terms of performance) and don't have to be synchronous
+-- e.g., sending emails or updating analytics. Also see [the queuing
+guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+
+### memcached
+
+memcached is used to cache database model objects. `zerver/lib/cache.py`
+and `zerver/lib/cache_helpers.py` manage putting things into memcached,
+and invalidating the cache when values change. The memcached
+configuration is in `puppet/zulip/files/memcached.conf`.
+
+### Redis
+
+Redis is used for a few very short-term data stores, such as in the
+basis of `zerver/lib/rate_limiter.py`, a per-user rate limiting scheme
+[example](http://blog.domaintools.com/2013/04/rate-limiting-with-redis/)),
+and the [email-to-Zulip
+integration](https://zulip.com/integrations/#email).
+
+Redis is configured in `zulip/puppet/zulip/files/redis` and it's a
+pretty standard configuration except for the last line, which turns off
+persistence:
+
+    # Zulip-specific configuration: disable saving to disk.
+    save ""
+
+memcached was used first and then we added Redis specifically to
+implement rate limiting. [We're discussing switching everything over to
+Redis.](https://github.com/zulip/zulip/issues/16)
+
+### RabbitMQ
+
+RabbitMQ is a queueing system. Its config files live in
+`zulip/puppet/zulip/files/rabbitmq`. Initial configuration happens in
+`zulip/scripts/setup/configure-rabbitmq`.
+
+We use RabbitMQ for queuing expensive work (e.g. sending emails
+triggered by a message, push notifications, some analytics, etc.) that
+require reliable delivery but which we don't want to do on the main
+thread. It's also used for communication between the application server
+and the Tornado push system.
+
+Two simple wrappers around `pika` (the Python RabbitMQ client) are in
+`zulip/server/lib/queue.py`. There's an asynchronous client for use in
+Tornado and a more general client for use elsewhere.
+
+`zerver/lib/event_queue.py` has helper functions for putting events into
+one queue or another. Most of the processes started by Supervisor are
+queue processors that continually pull things out of a RabbitMQ queue
+and handle them.
+
+Also see [the queuing
+guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+
+### PostgreSQL
+
+PostgreSQL (also known as Postgres) is the database that stores all
+persistent data, that is, data that's expected to live beyond a user's
+current session.
+
+In production, Postgres is installed with a default configuration. The
+directory that would contain configuration files
+(`puppet/zulip/files/postgresql`) has only a utility script and a custom
+list of stopwords used by a Postgresql extension.
+
+In a development environment, configuration of that postgresql extension
+is handled by `tools/postgres-init-dev-db` (invoked by `provision.py`).
+That file also manages setting up the development postgresql user.
+
+`provision.py` also invokes `tools/do-destroy-rebuild-database` to
+create the actual database with its schema.
+
+### Nagios
+
+Nagios is an optional component used for notifications to the system
+administrator, e.g., in case of outages.
+
+`zulip/puppet/zulip/manifests/nagios.pp` installs Nagios plugins from
+puppet/`zulip/files/nagios_plugins/`.
+
+This component is intended to install Nagios plugins intended to be run
+on a Nagios server; most of the Zulip Nagios plugins are intended to be
+run on the Zulip servers themselves, and are included with the relevant
+component of the Zulip server (e.g.
+`puppet/zulip/manifests/postgres_common.pp` installs a few under
+`/usr/lib/nagios/plugins/zulip_postgres_common`).

docs/changelog.md

@@ -0,0 +1,125 @@
+# Version History
+
+All notable changes to the Zulip server are documented in this file.
+
+### Unreleased
+
+### 1.3.13 - 2016-06-21
+- Added nearly complete internationalization of the Zulip UI.
+- Added warning when using @all/@everyone.
+- Added button offering to subscribe at bottom of narrows to streams
+  the user is not subscribed to.
+- Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT,
+  Transifex, and Updown.io.
+- Added menu option to mark all messages in a stream or topic as read.
+- Added new Attachment model to keep track of uploaded files.
+- Added caching of virtualenvs in development.
+- Added mypy static type annotations to about 85% of the Zulip Python codebase.
+- Added automated test of backend templates to test for regressions.
+- Added lots of detailed documentation on the Zulip development environment.
+- Added setting allowing only administrators to create new streams.
+- Added button to exit the Zulip tutorial early.
+- Added web UI for configuring default streams.
+- Added new OPEN_REALM_CREATION setting (default off), providing a UI
+  for creating additional realms on a Zulip server.
+- Fixed email_gateway_password secret not working properly.
+- Fixed missing helper scripts for RabbitMQ Nagios plugins.
+- Fixed skipping forward to latest messages ("More messages below" button).
+- Fixed netcat issue causing Zulip installation to hang on Scaleway machines.
+- Fixed rendering of /me status messages after message editing.
+- Fixed case sensitivity of right sidebar fading when compose is open.
+- Fixed error messages when composing to invalid PM recipients.
+- Fixed LDAP auth backend not working with Zulip mobile apps.
+- Fixed erroneous WWW-Authenticate headers with expired sessions.
+- Changed "coworkers" to "users" in the Zulip UI.
+- Changed add_default_stream REST API to correctly use PUT rather than PATCH.
+- Updated the Zulip emoji set (the Android Emoji) to a modern version.
+- Made numerous small improvements to the Zulip development experience.
+- Migrated backend templates to the faster Jinja2 templating system.
+- Migrated development environment setup scripts to tools/setup/.
+- Expanded test coverage for several areas of the product.
+- Simplified the API for writing new webhook integrations.
+- Removed most of the remaining javascript global variables.
+
+### 1.3.12 - 2016-05-10
+- CVE-2016-4426: Bot API keys were accessible to other users in the same realm.
+- CVE-2016-4427: Deactivated users could access messages if SSO was enabled.
+- Fixed a RabbitMQ configuration bug that resulted in reordered messages.
+- Added expansive test suite for authentication backends and decorators.
+- Added an option to logout_all_users to delete only sessions for deactivated users.
+
+### 1.3.11 - 2016-05-02
+- Moved email digest support into the default Zulip production configuration.
+- Added options for configuring Postgres, RabbitMQ, Redis, and memcached
+  in settings.py.
+- Added documentation on using Hubot to integrate with useful services
+  not yet integrated with Zulip directly (e.g. Google Hangouts).
+- Added new management command to test sending email from Zulip.
+- Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.
+- Added Nagios plugins to the main distribution.
+- Added ability for realm administrators to manage custom emoji.
+- Added guide to writing new integrations.
+- Enabled camo image proxy to fix mixed-content warnings for http images.
+- Refactored the Zulip puppet modules to be more modular.
+- Refactored the Tornado event system, fixing old memory leaks.
+- Removed many old-style /json API endpoints
+- Implemented running queue processors multithreaded in development,
+  decreasing RAM requirements for a Zulip development environment from
+  ~1GB to ~300MB.
+- Fixed rerendering the complete buddy list whenever a user came back from
+  idle, which was a significant performance issue in larger realms.
+- Fixed the disabling of desktop notifications from 1.3.7 for new users.
+- Fixed the (admin) create_user API enforcing restricted_to_domain, even
+  if that setting was disabled for the realm.
+- Fixed bugs changing certain settings in administration pages.
+- Fixed collapsing messages in narrowed views.
+- Fixed 500 errors when uploading a non-image file as an avatar.
+- Fixed Jira integration incorrectly not @-mentioning assignee.
+
+### 1.3.10 - 2016-01-21
+- Added new integration for Travis CI.
+- Added settings option to control maximum file upload size.
+- Added support for running Zulip development environment in Docker.
+- Added easy configuration support for a remote postgres database.
+- Added extensive documentation on scalability, backups, and security.
+- Recent private message threads are now displayed expanded similar to
+  the pre-existing recent topics feature.
+- Made it possible to set LDAP and EMAIL_HOST passwords in
+  /etc/zulip/secrets.conf.
+- Improved the styling for the Administration page and added tabs.
+- Substantially improved loading performance on slow networks by enabling
+  GZIP compression on more assets.
+- Changed the page title in narrowed views to include the current narrow.
+- Fixed several backend performance issues affecting very large realms.
+- Fixed bugs where draft compose content might be lost when reloading site.
+- Fixed support for disabling the "zulip" notifications stream.
+- Fixed missing step in postfix_localmail installation instructions.
+- Fixed several bugs/inconveniences in the production upgrade process.
+- Fixed realm restrictions for servers with a unique, open realm.
+- Substantially cleaned up console logging from run-dev.py.
+
+### 1.3.9 - 2015-11-16
+- Fixed buggy #! lines in upgrade scripts.
+
+### 1.3.8 - 2015-11-15
+- Added options to the Python api for working with untrusted server certificates.
+- Added a lot of documentation on the development environment and testing.
+- Added partial support for translating the Zulip UI.
+- Migrated installing Node dependencies to use npm.
+- Fixed LDAP integration breaking autocomplete of @-mentions.
+- Fixed admin panel reactivation/deactivation of bots.
+- Fixed inaccurate documentation for downloading the desktop apps.
+- Fixed various minor bugs in production installation process.
+- Fixed security issue where recent history on private streams might
+  be visible to new users (to the Zulip team) who were invited with that
+  private stream as one of their initial streams
+  (https://github.com/zulip/zulip/issues/230).
+- Major preliminary progress towards supporting Python 3.
+
+### 1.3.7 - 2015-10-19
+- Turn off desktop and audible notifications for streams by default.
+- Added support for the LDAP authentication integration creating new users.
+- Added new endpoint to support Google auth on mobile.
+- Fixed desktop notifications in modern Firefox.
+- Fixed several installation issues for both production and development environments.
+- Improved documentation for outgoing SMTP and the email mirror integration.

docs/code-contribution-checklist.rst

@@ -0,0 +1,24 @@
+=======================
+Code contribution guide
+=======================
+
+Thanks for contributing to Zulip! We hope this guide helps make the
+process smooth sailing for you.
+
+The "Ways to Contribute" and "How to get involved" sections of
+:doc:`readme-symlink` are a good place to start! After that, check out
+the tips below.
+
+
+.. toctree::
+   :maxdepth: 2
+
+   code-style
+   markdown
+   testing
+   logging
+   mypy
+   settings
+   front-end-build-process
+   queuing
+   schema-migrations

docs/code-style.md

@@ -0,0 +1,482 @@
+Code style and conventions
+==========================
+
+Be consistent!
+--------------
+
+Look at the surrounding code, or a similar part of the project, and try
+to do the same thing. If you think the other code has actively bad
+style, fix it (in a separate commit).
+
+When in doubt, send an email to <zulip-devel@googlegroups.com> with your
+question.
+
+Lint tools
+----------
+
+You can run them all at once with
+
+    ./tools/lint-all
+
+You can set this up as a local Git commit hook with
+
+    ``tools/setup-git-repo``
+
+The Vagrant setup process runs this for you.
+
+`lint-all` runs many lint checks in parallel, including
+
+-   Javascript ([JSLint](http://www.jslint.com/))
+
+    > `tools/jslint/check-all.js` contains a pretty fine-grained set of
+    > JSLint options, rule exceptions, and allowed global variables. If
+    > you add a new global, you'll need to add it to the list.
+
+-   Python ([Pyflakes](http://pypi.python.org/pypi/pyflakes))
+-   templates
+-   Puppet configuration
+-   custom checks (e.g. trailing whitespace and spaces-not-tabs)
+
+Secrets
+-------
+
+Please don't put any passwords, secret access keys, etc. inline in the
+code. Instead, use the `get_secret` function in `zproject/settings.py`
+to read secrets from `/etc/zulip/secrets.conf`.
+
+Dangerous constructs
+--------------------
+
+### Misuse of database queries
+
+Look out for Django code like this:
+
+    [Foo.objects.get(id=bar.x.id)
+    for bar in Bar.objects.filter(...)
+    if  bar.baz < 7]
+
+This will make one database query for each `Bar`, which is slow in
+production (but not in local testing!). Instead of a list comprehension,
+write a single query using Django's [QuerySet
+API](https://docs.djangoproject.com/en/dev/ref/models/querysets/).
+
+If you can't rewrite it as a single query, that's a sign that something
+is wrong with the database schema. So don't defer this optimization when
+performing schema changes, or else you may later find that it's
+impossible.
+
+### UserProfile.objects.get() / Client.objects.get / etc.
+
+In our Django code, never do direct `UserProfile.objects.get(email=foo)`
+database queries. Instead always use `get_user_profile_by_{email,id}`.
+There are 3 reasons for this:
+
+1.  It's guaranteed to correctly do a case-inexact lookup
+2.  It fetches the user object from remote cache, which is faster
+3.  It always fetches a UserProfile object which has been queried using
+    .selected\_related(), and thus will perform well when one later
+    accesses related models like the Realm.
+
+Similarly we have `get_client` and `get_stream` functions to fetch those
+commonly accessed objects via remote cache.
+
+### Using Django model objects as keys in sets/dicts
+
+Don't use Django model objects as keys in sets/dictionaries -- you will
+get unexpected behavior when dealing with objects obtained from
+different database queries:
+
+For example,
+`UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])`
+is False
+
+You should work with the IDs instead.
+
+### user\_profile.save()
+
+You should always pass the update\_fields keyword argument to .save()
+when modifying an existing Django model object. By default, .save() will
+overwrite every value in the column, which results in lots of race
+conditions where unrelated changes made by one thread can be
+accidentally overwritten by another thread that fetched its UserProfile
+object before the first thread wrote out its change.
+
+### Using raw saves to update important model objects
+
+In most cases, we already have a function in zephyr/lib/actions.py with
+a name like do\_activate\_user that will correctly handle lookups,
+caching, and notifying running browsers via the event system about your
+change. So please check whether such a function exists before writing
+new code to modify a model object, since your new code has a good chance
+of getting at least one of these things wrong.
+
+### `x.attr('zid')` vs. `rows.id(x)`
+
+Our message row DOM elements have a custom attribute `zid` which
+contains the numerical message ID. **Don't access this directly as**
+`x.attr('zid')` ! The result will be a string and comparisons (e.g. with
+`<=`) will give the wrong result, occasionally, just enough to make a
+bug that's impossible to track down.
+
+You should instead use the `id` function from the `rows` module, as in
+`rows.id(x)`. This returns a number. Even in cases where you do want a
+string, use the `id` function, as it will simplify future code changes.
+In most contexts in JavaScript where a string is needed, you can pass a
+number without any explicit conversion.
+
+### Javascript var
+
+Always declare Javascript variables using `var`:
+
+    var x = ...;
+
+In a function, `var` is necessary or else `x` will be a global variable.
+For variables declared at global scope, this has no effect, but we do it
+for consistency.
+
+Javascript has function scope only, not block scope. This means that a
+`var` declaration inside a `for` or `if` acts the same as a `var`
+declaration at the beginning of the surrounding `function`. To avoid
+confusion, declare all variables at the top of a function.
+
+### Javascript `for (i in myArray)`
+
+Don't use it:
+[[1]](http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays),
+[[2]](http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop),
+[[3]](http://www.jslint.com/lint.html#forin)
+
+### jQuery global state
+
+Don't mess with jQuery global state once the app has loaded. Code like
+this is very dangerous:
+
+    $.ajaxSetup({ async: false });
+    $.get(...);
+    $.ajaxSetup({ async: true });
+
+jQuery and the browser are free to run other code while the request is
+pending, which could perform other Ajax requests with the altered
+settings.
+
+Instead, switch to the more general `$.ajax`\_ function, which can take
+options like `async`.
+
+### State and logs files
+
+Do not write state and logs files inside the current working directory
+in the production environment. This will not how you expect, because the
+current working directory for the app changes every time we do a deploy.
+Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
+settings.py for an example.
+
+JS array/object manipulation
+----------------------------
+
+For generic functions that operate on arrays or JavaScript objects, you
+should generally use [Underscore](http://underscorejs.org/). We used to
+use jQuery's utility functions, but the Underscore equivalents are more
+consistent, better-behaved and offer more choices.
+
+A quick conversion table:
+
+       $.each → _.each (parameters to the callback reversed)
+       $.inArray → _.indexOf (parameters reversed)
+       $.grep → _.filter
+       $.map → _.map
+       $.extend → _.extend
+
+There's a subtle difference in the case of `_.extend`; it will replace
+attributes with undefined, whereas jQuery won't:
+
+       $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
+       _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
+
+Also, `_.each` does not let you break out of the iteration early by
+returning false, the way jQuery's version does. If you're doing this,
+you probably want `_.find`, `_.every`, or `_.any`, rather than 'each'.
+
+Some Underscore functions have multiple names. You should always use the
+canonical name (given in large print in the Underscore documentation),
+with the exception of `_.any`, which we prefer over the less clear
+'some'.
+
+More arbitrary style things
+---------------------------
+
+### General
+
+Indentation is four space characters for Python, JS, CSS, and shell
+scripts. Indentation is two space characters for HTML templates.
+
+We never use tabs anywhere in source code we write, but we have some
+third-party files which contain tabs.
+
+Keep third-party static files under the directory
+`zephyr/static/third/`, with one subdirectory per third-party project.
+
+We don't have an absolute hard limit on line length, but we should avoid
+extremely long lines. A general guideline is: refactor stuff to get it
+under 85 characters, unless that makes the code a lot uglier, in which
+case it's fine to go up to 120 or so.
+
+Whitespace guidelines:
+
+-   Put one space (or more for alignment) around binary arithmetic and
+    equality operators.
+-   Put one space around each part of the ternary operator.
+-   Put one space between keywords like `if` and `while` and their
+    associated open paren.
+-   Put one space between the closing paren for `if` and `while`-like
+    constructs and the opening curly brace. Put the curly brace on the
+    same line unless doing otherwise improves readability.
+-   Put no space before or after the open paren for function calls and
+    no space before the close paren for function calls.
+-   For the comma operator and colon operator in languages where it is
+    used for inline dictionaries, put no space before it and at least
+    one space after. Only use more than one space for alignment.
+
+### Javascript
+
+Don't use `==` and `!=` because these operators perform type coercions,
+which can mask bugs. Always use `===` and `!==`.
+
+End every statement with a semicolon.
+
+`if` statements with no braces are allowed, if the body is simple and
+its extent is abundantly clear from context and formatting.
+
+Anonymous functions should have spaces before and after the argument
+list:
+
+    var x = function (foo, bar) { // ...
+
+When calling a function with an anonymous function as an argument, use
+this style:
+
+    $.get('foo', function (data) {
+        var x = ...;
+        // ...
+    });
+
+The inner function body is indented one level from the outer function
+call. The closing brace for the inner function and the closing
+parenthesis for the outer call are together on the same line. This style
+isn't necessarily appropriate for calls with multiple anonymous
+functions or other arguments following them.
+
+Use
+
+    $(function () { ...
+
+rather than
+
+    $(document).ready(function () { ...
+
+and combine adjacent on-ready functions, if they are logically related.
+
+The best way to build complicated DOM elements is a Mustache template
+like `zephyr/static/templates/message.handlebars`. For simpler things
+you can use jQuery DOM building APIs like so:
+
+    var new_tr = $('<tr />').attr('id', zephyr.id);
+
+Passing a HTML string to jQuery is fine for simple hardcoded things:
+
+    foo.append('<p id="selected">foo</p>');
+
+but avoid programmatically building complicated strings.
+
+We used to favor attaching behaviors in templates like so:
+
+    <p onclick="select_zephyr({{id}})">
+
+but there are some reasons to prefer attaching events using jQuery code:
+
+-   Potential huge performance gains by using delegated events where
+    possible
+-   When calling a function from an `onclick` attribute, `this` is not
+    bound to the element like you might think
+-   jQuery does event normalization
+
+Either way, avoid complicated JavaScript code inside HTML attributes;
+call a helper function instead.
+
+### HTML / CSS
+
+Don't use the `style=` attribute. Instead, define logical classes and
+put your styles in `zulip.css`.
+
+Don't use the tag name in a selector unless you have to. In other words,
+use `.foo` instead of `span.foo`. We shouldn't have to care if the tag
+type changes in the future.
+
+Don't use inline event handlers (`onclick=`, etc. attributes). Instead,
+attach a jQuery event handler
+(`$('#foo').on('click', function () {...})`) when the DOM is ready
+(inside a `$(function () {...})` block).
+
+Use this format when you have the same block applying to multiple CSS
+styles (separate lines for each selector):
+
+    selector1,
+    selector2 {
+    };
+
+### Python
+
+-   Scripts should start with `#!/usr/bin/env python` and not
+    `#/usr/bin/python` (the right Python may not be installed in
+    `/usr/bin`) or `#/usr/bin/env/python2.7` (bad for Python 3
+    compatibility).  Don't put a shebang line on a Python file unless
+    it's meaningful to run it as a script. (Some libraries can also be
+    run as scripts, e.g. to run a test suite.)
+-   The first import in a file should be
+    `from __future__ import absolute_import`, per [PEP
+    328](http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports)
+-   Put all imports together at the top of the file, absent a compelling
+    reason to do otherwise.
+-   Unpacking sequences doesn't require list brackets:
+
+        [x, y] = xs    # unnecessary
+        x, y = xs      # better
+
+-   For string formatting, use `x % (y,)` rather than `x % y`, to avoid
+    ambiguity if `y` happens to be a tuple.
+-   When selecting by id, don't use `foo.pk` when you mean `foo.id`.
+    E.g.
+
+        recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
+
+    should be written as
+
+        recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
+
+    in case we ever change the primary keys.
+
+Version Control
+---------------
+
+### Commit Discipline
+
+We follow the Git project's own commit discipline practice of "Each
+commit is a minimal coherent idea". This discipline takes a bit of work,
+but it makes it much easier for code reviewers to spot bugs, and
+makes the commit history a much more useful resource for developers
+trying to understand why the code works the way it does, which also
+helps a lot in preventing bugs.
+
+Coherency requirements for any commit:
+
+-   It should pass tests (so test updates needed by a change should be
+    in the same commit as the original change, not a separate "fix the
+    tests that were broken by the last commit" commit).
+-   It should be safe to deploy individually, or comment in detail in
+    the commit message as to why it isn't (maybe with a [manual] tag).
+    So implementing a new API endpoint in one commit and then adding the
+    security checks in a future commit should be avoided -- the security
+    checks should be there from the beginning.
+-   Error handling should generally be included along with the code that
+    might trigger the error.
+-   TODO comments should be in the commit that introduces the issue or
+    functionality with further work required.
+
+When you should be minimal:
+
+-   Significant refactorings should be done in a separate commit from
+    functional changes.
+-   Moving code from one file to another should be done in a separate
+    commits from functional changes or even refactoring within a file.
+-   2 different refactorings should be done in different commits.
+-   2 different features should be done in different commits.
+-   If you find yourself writing a commit message that reads like a list
+    of somewhat dissimilar things that you did, you probably should have
+    just done 2 commits.
+
+When not to be overly minimal:
+
+-   For completely new features, you don't necessarily need to split out
+    new commits for each little subfeature of the new feature. E.g. if
+    you're writing a new tool from scratch, it's fine to have the
+    initial tool have plenty of options/features without doing separate
+    commits for each one. That said, reviewing a 2000-line giant blob of
+    new code isn't fun, so please be thoughtful about submitting things
+    in reviewable units.
+-   Don't bother to split back end commits from front end commits, even
+    though the backend can often be coherent on its own.
+
+Other considerations:
+
+-   Overly fine commits are easily squashed, but not vice versa, so err
+    toward small commits, and the code reviewer can advise on squashing.
+-   If a commit you write doesn't pass tests, you should usually fix
+    that by amending the commit to fix the bug, not writing a new "fix
+    tests" commit on top of it.
+
+Zulip expects you to structure the commits in your pull requests to form
+a clean history before we will merge them; it's best to write your
+commits following these guidelines in the first place, but if you don't,
+you can always fix your history using git rebase -i.
+
+It can take some practice to get used to writing your commits with a
+clean history so that you don't spend much time doing interactive
+rebases. For example, often you'll start adding a feature, and discover
+you need to a refactoring partway through writing the feature. When that
+happens, we recommend stashing your partial feature, do the refactoring,
+commit it, and then finish implementing your feature.
+
+### Commit Messages
+
+-   The first line of commit messages should be written in the
+    imperative and be kept relatively short while concisely explaining
+    what the commit does. For example:
+
+Bad:
+
+    bugfix
+    gather_subscriptions was broken
+    fix bug #234.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Use present-tense action verbs in your commit messages.
+
+Bad:
+
+    Fixing gather_subscriptions throwing an exception when given bad input.
+    Fixed gather_subscriptions throwing an exception when given bad input.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Please use a complete sentence in the summary, ending with a period.
+-   The rest of the commit message should be written in full prose and
+    explain why and how the change was made. If the commit makes
+    performance improvements, you should generally include some rough
+    benchmarks showing that it actually improves the performance.
+-   When you fix a GitHub issue, [mark that you've fixed the issue in
+    your commit
+    message](https://help.github.com/articles/closing-issues-via-commit-messages/)
+    so that the issue is automatically closed when your code is merged.
+    Zulip's preferred style for this is to have the final paragraph of
+    the commit message read e.g. "Fixes: \#123."
+-   Any paragraph content in the commit message should be line-wrapped
+    to less than 76 characters per line, so that your commit message
+    will be reasonably readable in git log in a normal terminal.
+-   In your commit message, you should describe any manual testing you
+    did in addition to running the automated tests, and any aspects of
+    the commit that you think are questionable and you'd like special
+    attention applied to.
+
+### Tests
+
+All significant new features should come with tests. See testing.
+
+### Third party code
+
+When adding new third-party packages to our codebase, please include
+"[third]" at the beginning of the commit message. You don't necessarily
+need to do this when patching third-party code that's already in tree.

docs/code-style.rst

@@ -1,484 +0,0 @@
-==========================
-Code style and conventions
-==========================
-
-Be consistent!
-==============
-
-Look at the surrounding code, or a similar part of the project, and
-try to do the same thing. If you think the other code has actively bad
-style, fix it (in a separate commit).
-
-When in doubt, send an email to zulip-devel@googlegroups.com with your
-question.
-
-Lint tools
-==========
-
-You can run them all at once with
-
-::
-
-    ./tools/lint-all
-
-You can set this up as a local Git commit hook with
-
-::
-
-    ``tools/setup-git-repo``
-
-The Vagrant setup process runs this for you.
-
-``lint-all`` runs many lint checks in parallel, including
-
-- Javascript (`JSLint <http://www.jslint.com/>`__)
-
-    ``tools/jslint/check-all.js`` contains a pretty fine-grained set of
-    JSLint options, rule exceptions, and allowed global variables. If you
-    add a new global, you'll need to add it to the list.
-
-- Python (`Pyflakes <http://pypi.python.org/pypi/pyflakes>`__)
-- templates
-- Puppet configuration
-- custom checks (e.g. trailing whitespace and spaces-not-tabs)
-
-Secrets
-=======
-
-Please don't put any passwords, secret access keys, etc. inline in the
-code.  Instead, use the ``get_secret`` function in
-``zproject/settings.py`` to read secrets from ``/etc/zulip/secrets.conf``.
-
-Dangerous constructs
-====================
-
-Misuse of database queries
---------------------------
-
-Look out for Django code like this::
-
-   [Foo.objects.get(id=bar.x.id)
-   for bar in Bar.objects.filter(...)
-   if  bar.baz < 7]
-
-This will make one database query for each ``Bar``, which is slow in
-production (but not in local testing!). Instead of a list comprehension,
-write a single query using Django's `QuerySet
-API <https://docs.djangoproject.com/en/dev/ref/models/querysets/>`__.
-
-If you can't rewrite it as a single query, that's a sign that something
-is wrong with the database schema. So don't defer this optimization when
-performing schema changes, or else you may later find that it's
-impossible.
-
-UserProfile.objects.get() / Client.objects.get / etc.
------------------------------------------------------
-
-In our Django code, never do direct
-``UserProfile.objects.get(email=foo)`` database queries. Instead always
-use ``get_user_profile_by_{email,id}``. There are 3 reasons for this:
-
-#. It's guaranteed to correctly do a case-inexact lookup
-#. It fetches the user object from memcached, which is faster
-#. It always fetches a UserProfile object which has been queried using
-   .selected\_related(), and thus will perform well when one later
-   accesses related models like the Realm.
-
-Similarly we have ``get_client`` and ``get_stream`` functions to fetch
-those commonly accessed objects via memcached.
-
-Using Django model objects as keys in sets/dicts
-------------------------------------------------
-
-Don't use Django model objects as keys in sets/dictionaries -- you will
-get unexpected behavior when dealing with objects obtained from
-different database queries:
-
-For example,
-``UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])``
-is False
-
-You should work with the IDs instead.
-
-user\_profile.save()
---------------------
-
-You should always pass the update\_fields keyword argument to .save()
-when modifying an existing Django model object. By default, .save() will
-overwrite every value in the column, which results in lots of race
-conditions where unrelated changes made by one thread can be
-accidentally overwritten by another thread that fetched its UserProfile
-object before the first thread wrote out its change.
-
-Using raw saves to update important model objects
--------------------------------------------------
-
-In most cases, we already have a function in zephyr/lib/actions.py with
-a name like do\_activate\_user that will correctly handle lookups,
-caching, and notifying running browsers via the event system about your
-change. So please check whether such a function exists before writing
-new code to modify a model object, since your new code has a good chance
-of getting at least one of these things wrong.
-
-``x.attr('zid')`` vs. ``rows.id(x)``
-------------------------------------
-
-Our message row DOM elements have a custom attribute ``zid`` which
-contains the numerical message ID. **Don't access this directly as**
-``x.attr('zid')`` ! The result will be a string and comparisons (e.g.
-with ``<=``) will give the wrong result, occasionally, just enough to
-make a bug that's impossible to track down.
-
-You should instead use the ``id`` function from the ``rows`` module, as
-in ``rows.id(x)``. This returns a number. Even in cases where you do
-want a string, use the ``id`` function, as it will simplify future code
-changes. In most contexts in JavaScript where a string is needed, you
-can pass a number without any explicit conversion.
-
-Javascript var
---------------
-
-Always declare Javascript variables using ``var``::
-
-   var x = ...;
-
-In a function, ``var`` is necessary or else ``x`` will be a global
-variable. For variables declared at global scope, this has no effect,
-but we do it for consistency.
-
-Javascript has function scope only, not block scope. This means that a
-``var`` declaration inside a ``for`` or ``if`` acts the same as a
-``var`` declaration at the beginning of the surrounding ``function``. To
-avoid confusion, declare all variables at the top of a function.
-
-Javascript ``for (i in myArray)``
----------------------------------
-
-Don't use it:
-`[1] <http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays>`__,
-`[2] <http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop>`__,
-`[3] <http://www.jslint.com/lint.html#forin>`__
-
-jQuery global state
--------------------
-
-Don't mess with jQuery global state once the app has loaded. Code like
-this is very dangerous::
-
-   $.ajaxSetup({ async: false });
-   $.get(...);
-   $.ajaxSetup({ async: true });
-
-jQuery and the browser are free to run other code while the request is
-pending, which could perform other Ajax requests with the altered
-settings.
-
-Instead, switch to the more general |ajax|_ function, which can take options
-like ``async``.
-
-.. |ajax| replace:: ``$.ajax``
-.. _ajax: http://api.jquery.com/jQuery.ajax
-
-State and logs files
---------------------
-
-Do not write state and logs files inside the current working directory
-in the production environment. This will not how you expect, because the
-current working directory for the app changes every time we do a deploy.
-Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
-settings.py for an example.
-
-JS array/object manipulation
-============================
-
-For generic functions that operate on arrays or JavaScript objects, you
-should generally use `Underscore <http://underscorejs.org/>`__. We used
-to use jQuery's utility functions, but the Underscore equivalents are
-more consistent, better-behaved and offer more choices.
-
-A quick conversion table::
-
-      $.each → _.each (parameters to the callback reversed)
-      $.inArray → _.indexOf (parameters reversed)
-      $.grep → _.filter
-      $.map → _.map
-      $.extend → _.extend
-
-There's a subtle difference in the case of ``_.extend``; it will replace
-attributes with undefined, whereas jQuery won't::
-
-      $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
-      _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
-
-Also, ``_.each`` does not let you break out of the iteration early by
-returning false, the way jQuery's version does. If you're doing this,
-you probably want ``_.find``, ``_.every``, or ``_.any``, rather than
-'each'.
-
-Some Underscore functions have multiple names. You should always use the
-canonical name (given in large print in the Underscore documentation),
-with the exception of ``_.any``, which we prefer over the less clear
-'some'.
-
-More arbitrary style things
-===========================
-
-General
--------
-
-Indentation is four space characters for Python, JS, CSS, and shell
-scripts. Indentation is two space characters for HTML templates.
-
-We never use tabs anywhere in source code we write, but we have some
-third-party files which contain tabs.
-
-Keep third-party static files under the directory
-``zephyr/static/third/``, with one subdirectory per third-party project.
-
-We don't have an absolute hard limit on line length, but we should avoid
-extremely long lines. A general guideline is: refactor stuff to get it
-under 85 characters, unless that makes the code a lot uglier, in which
-case it's fine to go up to 120 or so.
-
-Whitespace guidelines:
-
--  Put one space (or more for alignment) around binary arithmetic and
-   equality operators.
--  Put one space around each part of the ternary operator.
--  Put one space between keywords like ``if`` and ``while`` and their
-   associated open paren.
--  Put one space between the closing paren for ``if`` and ``while``-like
-   constructs and the opening curly brace. Put the curly brace on the
-   same line unless doing otherwise improves readability.
--  Put no space before or after the open paren for function calls and no
-   space before the close paren for function calls.
--  For the comma operator and colon operator in languages where it is
-   used for inline dictionaries, put no space before it and at least one
-   space after. Only use more than one space for alignment.
-
-Javascript
-----------
-
-Don't use ``==`` and ``!=`` because these operators perform type
-coercions, which can mask bugs. Always use ``===`` and ``!==``.
-
-End every statement with a semicolon.
-
-``if`` statements with no braces are allowed, if the body is simple and
-its extent is abundantly clear from context and formatting.
-
-Anonymous functions should have spaces before and after the argument
-list::
-
-   var x = function (foo, bar) { // ...
-
-When calling a function with an anonymous function as an argument, use
-this style::
-
-   $.get('foo', function (data) {
-       var x = ...;
-       // ...
-   });
-
-The inner function body is indented one level from the outer function
-call. The closing brace for the inner function and the closing
-parenthesis for the outer call are together on the same line. This style
-isn't necessarily appropriate for calls with multiple anonymous
-functions or other arguments following them.
-
-Use
-
-::
-
-   $(function () { ...
-
-rather than
-
-::
-
-   $(document).ready(function () { ...
-
-and combine adjacent on-ready functions, if they are logically related.
-
-The best way to build complicated DOM elements is a Mustache template
-like ``zephyr/static/templates/message.handlebars``. For simpler things
-you can use jQuery DOM building APIs like so::
-
-   var new_tr = $('<tr />').attr('id', zephyr.id);
-
-Passing a HTML string to jQuery is fine for simple hardcoded things::
-
-   foo.append('<p id="selected">foo</p>');
-
-but avoid programmatically building complicated strings.
-
-We used to favor attaching behaviors in templates like so::
-
-    <p onclick="select_zephyr({{id}})">
-
-but there are some reasons to prefer attaching events using jQuery code:
-
--  Potential huge performance gains by using delegated events where
-   possible
--  When calling a function from an ``onclick`` attribute, ``this`` is
-   not bound to the element like you might think
--  jQuery does event normalization
-
-Either way, avoid complicated JavaScript code inside HTML attributes;
-call a helper function instead.
-
-HTML / CSS
-----------
-
-Don't use the ``style=`` attribute. Instead, define logical classes and
-put your styles in ``zulip.css``.
-
-Don't use the tag name in a selector unless you have to. In other words,
-use ``.foo`` instead of ``span.foo``. We shouldn't have to care if the
-tag type changes in the future.
-
-Don't use inline event handlers (``onclick=``, etc. attributes).
-Instead, attach a jQuery event handler
-(``$('#foo').on('click', function () {...})``) when the DOM is ready
-(inside a ``$(function () {...})`` block).
-
-Use this format when you have the same block applying to multiple CSS
-styles (separate lines for each selector)::
-
-    selector1,
-    selector2 {
-    };
-
-Python
-------
-
--  Scripts should start with ``#!/usr/bin/env python2.7`` and not
-   ``#!/usr/bin/env python2.7``. See commit ``437d4aee`` for an explanation of
-   why. Don't put such a line on a Python file unless it's meaningful to
-   run it as a script. (Some libraries can also be run as scripts, e.g.
-   to run a test suite.)
--  The first import in a file should be
-   ``from __future__ import absolute_import``, per `PEP
-   328 <http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports>`__
--  Put all imports together at the top of the file, absent a compelling
-   reason to do otherwise.
--  Unpacking sequences doesn't require list brackets::
-
-      [x, y] = xs    # unnecessary
-      x, y = xs      # better
-
--  For string formatting, use ``x % (y,)`` rather than ``x % y``, to
-   avoid ambiguity if ``y`` happens to be a tuple.
--  When selecting by id, don't use ``foo.pk`` when you mean ``foo.id``.
-   E.g.
-
-   ::
-
-      recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
-
-   should be written as
-
-   ::
-
-      recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
-
-   in case we ever change the primary keys.
-
-Version Control
-===============
-
-Commit Discipline
------------------
-
-We follow the Git project's own commit discipline practice of "Each
-commit is a minimal coherent idea".
-
-Coherency requirements for any commit:
-
--  It should pass tests (so test updates needed by a change should be in
-   the same commit as the original change, not a separate "fix the tests
-   that were broken by the last commit" commit).
--  It should be safe to deploy individually, or comment in detail in the
-   commit message as to why it isn't (maybe with a [manual] tag). So
-   implementing a new API endpoint in one commit and then adding the
-   security checks in a future commit should be avoided -- the security
-   checks should be there from the beginning.
--  Error handling should generally be included along with the code that
-   might trigger the error.
--  TODO comments should be in the commit that introduces the
-   issue or functionality with further work required.
-
-When you should be minimal:
-
--  Significant refactorings should be done in a separate commit from
-   functional changes.
--  Moving code from one file to another should be done in a separate
-   commits from functional changes or even refactoring within a file.
--  2 different refactorings should be done in different commits.
--  2 different features should be done in different commits.
--  If you find yourself writing a commit message that reads like a list
-   of somewhat dissimilar things that you did, you probably should have
-   just done 2 commits.
-
-When not to be overly minimal:
-
--  For completely new features, you don't necessarily need to split out
-   new commits for each little subfeature of the new feature. E.g. if
-   you're writing a new tool from scratch, it's fine to have the initial
-   tool have plenty of options/features without doing separate commits
-   for each one.  That said, reviewing a 2000-line giant blob of new
-   code isn't fun, so please be thoughtful about submitting things in
-   reviewable units.
--  Don't bother to split back end commits from front end commits, even
-   though the backend can often be coherent on its own.
-
-Other considerations:
-
--  Overly fine commits are easily squashed, but not vice versa, so err
-   toward small commits, and the code reviewer can advise on squashing.
-
-It can take some practice to get used to writing your commits this
-way.  For example, often you'll start adding a feature, and discover
-you need to a refactoring partway through writing the feature.  When
-that happens, we recommend stashing your partial feature, do the
-refactoring, commit it, and then finish implementing your feature.
-
-Commit Messages
----------------
-
--  The first line of commit messages should be written in the imperative
-   and be kept relatively short while concisely explaining what the
-   commit does. For example:
-
-Bad::
-
-   bugfix
-   gather_subscriptions was broken
-
-Good::
-
-   Prevent gather_subscriptions from throwing an exception when given bad input.
-
--  Please use a complete sentence, ending with a period.
-
--  The rest of the commit message should be written in full prose and
-   explain why and how the change was made. If the commit makes
-   performance improvements, you should generally include some rough
-   benchmarks showing that it actually improves the performance.
-
--  In your commit message, you should describe any manual testing you
-   did in addition to running the automated tests, and any aspects of
-   the commit that you think are questionable and you'd like special
-   attention applied to.
-
-Tests
------
-
-All significant new features should come with tests.
-
-Third party code
-----------------
-
-When adding new third-party packages to our codebase, please include
-"[third]" at the beginning of the commit message. You don't necessarily
-need to do this when patching third-party code that's already in tree.

docs/conf.py

@@ -15,6 +15,7 @@
 import sys
 import os
 import shlex
+from typing import Dict, List, Optional
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -29,16 +30,11 @@ import shlex
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = [] # type: List[str]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
 
@@ -64,7 +60,7 @@ release = '0.1'
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = None # type: Optional[str]
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
@@ -227,7 +223,7 @@ latex_elements = {
 
 # Latex figure (float) alignment
 #'figure_align': 'htbp',
-}
+} # type: Dict[str, str]
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
@@ -293,3 +289,13 @@ texinfo_documents = [
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 #texinfo_no_detailmenu = False
+
+from recommonmark.parser import CommonMarkParser
+
+source_parsers = {
+        '.md': CommonMarkParser,
+        }
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+source_suffix = ['.rst', '.md']

docs/directory-structure.md

@@ -0,0 +1,115 @@
+Directory structure
+===================
+
+This page documents the Zulip directory structure and how to decide
+where to put a file.
+
+Scripts
+-------
+
+* `scripts/` Scripts that production deployments might run manually
+  (e.g., `restart-server`).
+
+* `scripts/lib/` Scripts that are needed on production deployments but
+  humans should never run.
+
+* `scripts/setup/` Tools that production deployments will only run
+  once, during installation.
+
+* `tools/` Development tools.
+
+---------------------------------------------------------
+
+Bots
+----
+
+* `api/integrations/` Bots distributed as part of the Zulip API bundle.
+
+* `bots/` Previously Zulip internal bots. These usually need a bit of
+   work.
+
+-----------------------------------------------------
+
+Management commands
+-------------------
+
+* `zerver/management/commands/` Management commands one might run at a
+  production deployment site (e.g. scripts to change a value or
+  deactivate a user properly)
+
+-------------------------------------------------------------------------
+
+Views
+-----
+
+* `zerver/tornadoviews.py` Tornado views
+
+* `zerver/views/webhooks.py` Webhook views
+
+* `zerver/views/messages.py` message-related views
+
+* `zerver/views/__init__.py` other Django views
+
+----------------------------------------
+
+Jinja2 Compatibility Files
+--------------------------
+
+* `zproject/jinja2/__init__.py` Jinja2 environment
+
+* `zproject/jinja2/backends.py` Jinja2 backend
+
+* `zproject/jinja2/compressors.py` Jinja2 compatible functions of
+   Django-Pipeline
+
+-----------------------------------------------------------------------
+
+Static assets
+-------------
+
+* `assets/` For assets not to be served to the web (e.g. the system to
+            generate our favicons)
+
+* `static/` For things we do want to both serve to the web and
+            distribute to production deployments (e.g. the webpages)
+
+---------------------------------------------------------------
+
+Puppet
+------
+
+* `puppet/zulip/` For configuration for production deployments
+
+-------------------------------------------------------------------
+
+Templates
+---------
+
+* `templates/zerver/` For Jinja2 templates for the backend (for zerver app)
+
+* `static/templates/` Handlebars templates for the frontend
+
+-----------------------------------------------------------------------
+
+Tests
+-----
+
+* `zerver/tests/` Backend tests
+
+* `frontend_tests/node_tests/` Node Frontend unit tests
+
+* `frontend_tests/casper_tests/` Casper frontend tests
+
+-----------------------------------------------------------------------
+
+
+Documentation
+-------------
+
+*  `docs/`        Source for this documentation
+
+--------------------------------------------------------------
+
+You can consult the repository's `.gitattributes` file to see exactly
+which components are excluded from production releases (release
+tarballs are generated using `tools/build-release-tarball`).

docs/directory-structure.rst

@@ -1,95 +0,0 @@
-===================
-Directory structure
-===================
-
-This page documents the Zulip directory structure and how to decide where to
-put a file.
-
-Scripts
-=======
-
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/``       | Scripts that production deployments might run manually (e.g. ``restart-server``)  |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``bin/``           | Scripts that are needed on production deployments but humans should never run     |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/setup/`` | Tools that production deployments will only run once, during installation         |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``tools/``         | Development tools                                                                 |
-+--------------------+-----------------------------------------------------------------------------------+
-
-Bots
-====
-
-+------------------------+----------------------------------------------------------------------+
-| ``api/integrations``   | Bots distributed as part of the Zulip API bundle.                    |
-+------------------------+----------------------------------------------------------------------+
-| ``bots/``              | Previously Zulip internal bots.  These usually need a bit of work.   |
-+------------------------+----------------------------------------------------------------------+
-
-Management commands
-===================
-
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-| ``zerver/management/commands/``     | Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly)   |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-
-Views
-=====
-
-+--------------------------------+-----------------------------------------+
-| ``zerver/tornadoviews.py``     | Tornado views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/webhooks.py``   | Webhook views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/messages.py``   | message-related views                   |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/__init__.py``   | other Django views                      |
-+--------------------------------+-----------------------------------------+
-
-Static assets
-=============
-
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``assets/``   | For assets not to be served to the web (e.g. the system to generate our favicons)                             |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``static/``   | For things we do want to both serve to the web and distribute to production deployments (e.g. the webpages)   |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-
-Puppet
-======
-
-+--------------------+----------------------------------------------------------------------------------+
-| ``puppet/zulip``   | For configuration for production deployments                                     |
-+--------------------+----------------------------------------------------------------------------------+
-
-Templates
-=========
-
-+--------------------------+--------------------------------------------------------+
-| ``templates/zerver``     | For templates related to zerver views                  |
-+--------------------------+--------------------------------------------------------+
-| ``static/templates``     | Handlebars templates for the frontend                  |
-+--------------------------+--------------------------------------------------------+
-
-Tests
-=====
-
-+------------------------+-----------------------------------+
-| ``zerver/test*.py``             | Backend tests            |
-+------------------------+-----------------------------------+
-| ``frontend_tests/node``         | Node Frontend unit tests |
-+------------------------+-----------------------------------+
-| ``frontend_tests/tests``        | Casper frontend tests    |
-+------------------------+-----------------------------------+
-
-Documentation
-=============
-
-+-------------+-----------------------------------------------+
-| ``docs/``   | Source for this documentation                 |
-+-------------+-----------------------------------------------+
-
-You can consult the repository's .gitattributes file to see exactly
-which components are excluded from production releases (release
-tarballs are generated using tools/build-release-tarball).

docs/front-end-build-process.md

@@ -0,0 +1,73 @@
+Front End Build Process
+=======================
+
+This page documents additional information that may be useful when
+developing new features for Zulip that require front-end changes. For a
+more general overview, see the new-feature-tutorial. The code-style
+documentation also has relevant information about how Zulip's code is
+structured.
+
+Primary build process
+---------------------
+
+Most of the existing JS in Zulip is written in IIFE-wrapped modules, one
+per file in the static/js directory. When running Zulip in development
+mode, each file is loaded seperately. In production mode (and when
+creating a release tarball using tools/build-release-tarball),
+JavaScript files are concatenated and minified.
+
+If you add a new JavaScript file, it needs to be specified in the
+JS\_SPECS dictionary defined in zproject/settings.py to be included in
+the concatenated file.
+
+Webpack/CommonJS modules
+------------------------
+
+New JS written for Zulip can be written as CommonJS modules (bundled
+using [webpack](https://webpack.github.io/), though this will taken care
+of automatically whenever `run-dev.py` is running). (CommonJS is the
+same module format that Node uses, so see the [Node
+documentation](https://nodejs.org/docs/latest/api/modules.html) for
+more information on the syntax.)
+
+Benefits of using CommonJS modules over the
+[IIFE](http://benalman.com/news/2010/11/immediately-invoked-function-expression/)
+module approach:
+
+-   namespacing/module boilerplate will be added automatically in the
+    bundling process
+-   dependencies between modules are more explicit and easier to trace
+-   no separate list of JS files needs to be maintained for
+    concatenation and minification
+-   third-party libraries can be more easily installed/versioned using
+    npm
+-   running the same code in the browser and in Node for testing is
+    simplified (as both environments use the same module syntax)
+
+The entry point file for the bundle generated by webpack is
+`static/js/src/main.js`. Any modules you add will need to be required
+from this file (or one of its dependencies) in order to be included in
+the script bundle.
+
+Adding static files
+-------------------
+
+To add a static file to the app (JavaScript, CSS, images, etc), first
+add it to the appropriate place under `static/`.
+
+-   Third-party files should all go in `static/third/`. Tag the commit
+    with "[third]" when adding or modifying a third-party package.
+-   Our own JS lives under `static/js`; CSS lives under `static/styles`.
+-   JavaScript and CSS files are combined and minified in production. In
+    this case all you need to do is add the filename to PIPELINE\_CSS or
+    JS\_SPECS in `zproject/settings.py`. (If you plan to only use the
+    JS/CSS within the app proper, and not on the login page or other
+    standalone pages, put it in the 'app' category.)
+
+If you want to test minified files in development, look for the
+`PIPELINE =` line in `zproject/settings.py` and set it to `True` -- or
+just set `DEBUG = False`.
+
+Note that `static/html/{400,5xx}.html` will only render properly if
+minification is enabled, since they hardcode the path
+`static/min/portico.css`.

docs/front-end-build-process.rst

@@ -1,26 +0,0 @@
-=======================
-Front End Build Process
-=======================
-
-This page documents additional information that may be useful when developing new features for Zulip that require front-end changes. For a more general overview, see the new feature tutorial. The code style documentation also has relevant information about how Zulip's code is structured.
-
-Primary build process
-=====================
-
-Most of the exisiting JS in Zulip is written in IIFE-wrapped modules, one per file in the `static/js` directory. When running Zulip in development mode each file is loaded seperately. In production mode (and when creating a release tarball) JavaScript files are concatenated and minified.
-
-If you add a new JavaScript file it needs to be specified in the `JS_SPECS` dictionary defined in `zproject/settings.py` to be included in the concatenated file.
-
-Webpack/CommonJS modules
-========================
-New JS written for Zulip can be written as CommonJS modules (bundled using `webpack <https://webpack.github.io/>`_, though this will taken care of automatically whenever ``run-dev.py`` is running). (CommonJS is the same module format that Node uses, so see `the Node documentation <https://nodejs.org/docs/latest/api/modules.html>` for more information on the syntax.)
-
-Benefits of using CommonJS modules over the `IIFE <http://benalman.com/news/2010/11/immediately-invoked-function-expression/>`_ module approach:
-
-* namespacing/module boilerplate will be added automatically in the bundling process
-* dependencies between modules are more explicit and easier to trace
-* no separate list of JS files needs to be maintained for concatenation and minification
-* third-party libraries can be more easily installed/versioned using npm
-* running the same code in the browser and in Node for testing is simplified (as both environments use the same module syntax)
-
-The entry point file for the bundle generated by webpack is ``static/js/src/main.js``. Any modules you add will need to be required from this file (or one of its dependencies) in order to be included in the script bundle.

docs/html_unescape.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import print_function
 
 # Remove HTML entity escaping left over from MediaWiki->rST conversion.

docs/index.rst

@@ -6,16 +6,35 @@
 Welcome to Zulip documentation!
 ===============================
 
+Zulip is a powerful, open source group chat application. Written in
+Python and using the Django framework, Zulip supports both private
+messaging and group chats via conversation streams.
+
+Zulip also supports fast search, drag-and-drop file uploads, image
+previews, group private messages, audible notifications, missed-message
+emails, desktop apps, and much more.
+
+Further information on the Zulip project and its features can be found
+at `https://www.zulip.org <https://www.zulip.org>`__ and in these
+docs.  Our code is available at `our GitHub repository
+<https://github.com/zulip/>`__.
+
+This set of documents covers installation and contribution instructions.
+
 Contents:
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 3
 
+   readme-symlink
+   architecture-overview
+   integration-guide
    new-feature-tutorial
-   code-style
-   front-end-build-process
+   code-contribution-checklist
    directory-structure
-   testing
+   translating
+   changelog
+   roadmap
 
 Indices and tables
 ==================

docs/integration-guide.md

@@ -0,0 +1,182 @@
+# How to write a new integration
+
+Integrations are one of the most important parts of a group chat tool
+like Zulip, and we are committed to making integrating with Zulip and
+getting you integration merged upstream so everyone else can benefit
+from it as easy as possible while maintaining the high quality of the
+Zulip integrations library.
+
+Contributions to this guide are very welcome, so if you run into any
+issues following these instructions or come up with any tips or tools
+that help writing integration, please email
+zulip-devel@googlegroups.com, open an issue, or submit a pull request
+to share your ideas!
+
+## Types of integrations
+
+We have several different ways that we integrate with 3rd part
+products, ordered here by which types we prefer to write:
+
+1. Webhook integrations (examples: Freshdesk, GitHub), where the
+third-party service supports posting content to a particular URI on
+our site with data about the event.  For these, you usually just need
+to add a new handler in `zerver/views/webhooks.py` (plus
+test/document/etc.).  An example commit implementing a new webhook is:
+https://github.com/zulip/zulip/pull/324.
+
+2. Python script integrations (examples: SVN, Git), where we can get
+the service to call our integration (by shelling out or otherwise),
+passing in the required data.  Our preferred model for these is to
+ship these integrations in our API release tarballs (by writing the
+integration in `api/integrations`).
+
+3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user
+needs to install a plugin into their existing software.  These are
+often more work, but for some products are the only way to integrate
+with the product at all.
+
+## General advice for writing integrations
+
+* Consider using our Zulip markup to make the output from your
+  integration especially attractive or useful (e.g.  emoji, markdown
+  emphasis, @-mentions, or `!avatar(email)`).
+
+* Use topics effectively to ensure sequential messages about the same
+  thing are threaded together; this makes for much better consumption
+  by users.  E.g. for a bug tracker integration, put the bug number in
+  the topic for all messages; for an integration like Nagios, put the
+  service in the topic.
+
+* Integrations that don't match a team's workflow can often be
+  uselessly spammy.  Give careful thought to providing options for
+  triggering Zulip messages only for certain message types, certain
+  projects, or sending different messages to different streams/topics,
+  to make it easy for teams to configure the integration to support
+  their workflow.
+
+* Consistently capitalize the name of the integration in the
+  documentation and the Client name the way the vendor does.  It's OK
+  to use all-lower-case in the implementation.
+
+* Sometimes it can be helpful to contact the vendor if it appears they
+  don't have an API or webhook we can use -- sometimes the right API
+  is just not properly documented.
+
+## Writing Webhook integrations
+
+New Zulip webhook integrations can take just a few hours to write,
+including tests and documentation, if you use the right process.
+Here's how we recommend doing it:
+
+* First, use http://requestb.in/ or a similar site to capture an
+  example webhook payload from the service you're integrating.  You
+  can use these captured payloads to create a set of test fixtures for
+  your integration under `zerver/fixtures`.
+
+* Then write a draft webhook handler under `zerver/views/webhooks/`;
+  there are a lot of examples in that directory.  We recommend
+  templating off a short one (like `stash.py` or `zendesk.py`), since
+  the longer ones usually just have more complex parsing which can
+  obscure what's common to all webhook integrations.  In addition to
+  writing the integration itself, you'll need to add an entry in
+  `zproject/urls.py` for your webhook; search for `webhook` in that
+  file to find the existing ones (and please add yours in the
+  alphabetically correct place).
+
+* Then write a test for your fixture in `zerver/tests/test_hooks.py`, and
+  you can iterate on the tests and webhooks handler until they work,
+  all without ever needing to post directly from the server you're
+  integrating to your Zulip development machine.  To run just the
+  tests from the test class you wrote, you can use e.g.
+
+  ```
+  test-backend zerver.tests.test_hooks.PagerDutyHookTests
+  ```
+
+  See
+  https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+  for more details on the Zulip test runner.
+
+* Once you've gotten your webhook working and passing a test, capture
+  payloads for the other common types of posts the service's webhook
+  will make, and add tests for them; usually this part of the process
+  is pretty fast.  Webhook integration tests should all use fixtures
+  (as opposed to contacting the service), since otherwise the tests
+  can't run without Internet access and some sort of credentials for
+  the service.
+
+* Finally, write documentation for the integration (see below)!
+
+## Writing Python script and plugin integrations integrations
+
+For plugin integrations, usually you will need to consult the
+documentation for the third party software in order to learn how to
+write the integration.  But we have a few notes on how to do these:
+
+* You should always send messages by POSTing to URLs of the form
+`https://zulip.example.com/v1/messages/`, not the legacy
+`/api/v1/send_message` message sending API.
+
+* We usually build Python script integration with (at least) 2 files:
+`zulip_foo_config.py`` containing the configuration for the
+integration including the bots' API keys, plus a script that reads
+from this configuration to actually do the work (that way, it's
+possible to update the script without breaking users' configurations).
+
+* Be sure to test your integration carefully and document how to
+  install it (see notes on documentation below).
+
+* You should specify a clear HTTP User-Agent for your integration. The
+user agent should at a minimum identify the integration and version
+number, separated by a slash. If possible, you should collect platform
+information and include that in `()`s after the version number. Some
+examples of ideal UAs are:
+
+```
+ZulipDesktop/0.7.0 (Ubuntu; 14.04)
+ZulipJenkins/0.1.0 (Windows; 7.2)
+ZulipMobile/0.5.4 (Android; 4.2; maguro)
+```
+
+## Documenting your integration
+
+Every Zulip integration must be documented in
+`templates/zerver/integrations.html`.  Usually, this involves a few
+steps:
+
+* Add an `integration-lozenge` class block in the alphabetically
+  correct place in the main integration list, using the logo for the
+  integrated software.
+
+* Add an `integration-instructions` class block also in the
+  alphabetically correct place, explaining all the steps required to
+  setup the integration, including what URLs to use, etc.  If there
+  are any screens in the product involved, take a few screenshots with
+  the input fields filled out with sample values in order to make the
+  instructions really easy to follow.  For the screenshots, use
+  something like `github-bot@example.com` for the email addresses and
+  an obviously fake API key like `abcdef123456790`.
+
+* Finally, generate a message sent by the integration and take a
+  screenshot of the message to provide an example message in the
+  documentation. If your new integration is a webhook integration,
+  you can generate such a message from your test fixtures
+  using `send_webhook_fixture_message`:
+
+  ```
+  ./manage.py send_webhook_fixture_message \
+       --fixture=zerver/fixtures/pingdom/pingdom_imap_down_to_up.json \
+       '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'
+  ```
+
+  When generating the screenshot of a sample message, give your test
+  bot a nice name like "GitHub Bot", use the project's logo as the
+  bot's avatar, and take the screenshots showing the stream/topic bar
+  for the message, not just the message body.
+
+When writing documentation for your integration, be sure to use the
+`{{ external_api_uri }}` template variable, so that your integration
+documentation will provide the correct URL for whatever server it is
+deployed on.  If special configuration is required to set the SITE
+variable, you should document that too, inside an `{% if
+api_site_required %}` check.

docs/logging.md

@@ -0,0 +1,24 @@
+# Logging and Performance Debugging
+
+It's good to have the terminal running `run-dev.py` up as you work since error
+messages including tracebacks along with every backend request will be printed
+there.
+
+The messages will look similar to:
+
+```
+2016-05-20 14:50:22,056 INFO     127.0.0.1       GET     302 528ms (db: 1ms/1q) (+start: 123ms) / (unauth via ?)
+[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0
+2016-05-20 14:50:22,272 INFO     127.0.0.1       GET     200 124ms (db: 3ms/2q) /login/ (unauth via ?)
+2016-05-20 14:50:26,333 INFO     127.0.0.1       POST    302  37ms (db: 6ms/7q) /accounts/login/local/ (unauth via ?)
+[20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0
+2016-05-20 14:50:26,538 INFO     127.0.0.1       GET     200  12ms (db: 1ms/2q) (+start: 53ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
+2016-05-20 14:50:26,657 INFO     127.0.0.1       GET     200  10ms (+start: 8ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
+2016-05-20 14:50:26,959 INFO     127.0.0.1       GET     200 588ms (db: 26ms/21q) / [1463769771:0] (cordelia@zulip.com via website)
+```
+
+The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status
+code, time to process, (optional perf data details, e.g. database time/queries,
+memcached time/queries, Django process startup time, markdown processing time,
+etc.), URL, and "email via client" showing user account involved (if logged in)
+and the type of client they used ("web", "Android", etc.).

docs/markdown.md

@@ -0,0 +1,136 @@
+# Zulip's Markdown implementation
+
+Zulip has a special flavor of Markdown, currently called 'bugdown'
+after Zulip's original name of "humbug". End users are using Bugdown
+within the client, not original Markdown.
+
+Zulip has two implementations of Bugdown.  The first is based on
+Python-Markdown (`zerver/lib/bugdown/`) and is used to authoritatively
+render messages on the backend (and implements expensive features like
+querying the Twitter API to render tweets nicely).  The other is in
+JavaScript, based on marked (`static/js/echo.js`), and is used to
+preview and locally echo messages the moment the sender hits enter,
+without waiting for round trip from the server.  The two
+implementations are tested for compatibility via
+`zerver/tests/test_bugdown.py` and the fixtures under
+`zerver/fixtures/bugdown-data.json`.
+
+The JavaScript implementation knows which types of messages it can
+render correctly, and thus while there is code to rerender messages
+based on the authoritative backend rendering (which would clause a
+change in the rendering visible only to the sender shortly after a
+message is sent), this should never happen, and whenever it does it is
+considered a bug.  Instead, if the frontend doesn't know how to
+correctly render a message, we simply won't echo the message for the
+sender until it's rendered by the backend.  So for example, a message
+containing a link to Twitter will not be rendered by the JavaScript
+implementation because it doesn't support doing the 3rd party API
+queries required to render tweets nicely.
+
+I should note that the below documentation is based on a comparison
+with original Markdown, not newer Markdown variants like CommonMark.
+
+## Zulip's Markdown philosophy
+
+Markdown is great for group chat for the same reason it's been
+successful in products ranging from blogs to wikis to bug trackers:
+it's close enough to how people try to express themselves when writing
+plain text (e.g. emails) that is helps more than getting in the way.
+
+The main issue for using Markdown in instant messaging is that the
+Markdown standard syntax used in a lot of wikis/blogs has nontrivial
+error rates, where the author needs to go back and edit the post to
+fix the formatting after typing it the first time.  While that's
+basically fine when writing a blog, it gets annoying very fast in a
+chat product; even though you can edit messages to fix formatting
+mistakes, you don't want to be doing that often.  There are basically
+2 types of error rates that are important for a product like Zulip:
+
+* What fraction of the time, if you pasted a short technical email
+that you wrote to your team and passed it through your Markdown
+implementation, would you need to change the text of your email for it
+to render in a reasonable way?  This is the "accidental Markdown
+syntax" problem, common with Markdown syntax like the italics syntax
+interacting with talking about `char *`s.
+
+* What fraction of the time do users attempting to use a particular
+Markdown syntax actually succeed at doing so correctly?  Syntax like
+required a blank line between text and the start of a bulleted list
+raise this figure substantially.
+
+Both of these are minor issues for most products using Markdown, but
+they are major problems in the instant messaging context, because one
+can't edit a message that has already been sent and users are
+generally writing quickly.  Zulip's Markdown strategy is based on the
+principles of giving users the power they need to express complicated
+ideas in a chat context while minimizing those two error rates.
+
+## Zulip's Changes to Markdown
+
+Below, we document the changes that Zulip has against stock
+Python-Markdown; some of the features we modify / disable may already
+be non-standard.
+
+### Basic syntax
+
+* Enable `nl2br</tt> extension: this means one newline creates a line
+  break (not paragraph break).
+
+* Disable italics entirely.  This resolves an issue where people were
+  using `*` and `_` and hitting it by mistake too often.  E.g. with
+  stock Markdown `You should use char * instead of void * there` would
+  trigger italics.
+
+* Allow only `**` syntax for bold, not `__` (easy to hit by mistake if
+  discussing Python `__init__` or something)
+
+* Disable special use of `\` to escape other syntax. Rendering `\\` as
+  `\` was hugely controversial, but having no escape syntax is also
+  controversial.  We may revisit this.  For now you can always put
+  things in code blocks.
+
+### Lists
+
+* Allow tacking a bulleted list or block quote onto the end of a
+  paragraph, i.e. without a blank line before it
+
+* Allow only `*` for bulleted lists, not `+` or `-` (previously
+  created confusion with diff-style text sloppily not included in a
+  code block)
+
+* Disable ordered list syntax: it automatically renumbers, which can
+  be really confusing when sending a numbered list across multiple
+  messages.
+
+### Links
+
+* Enable auto-linkification, both for `http://...` and guessing at
+  things like `t.co/foo`.
+
+* Force links to be absolute. `[foo](google.com)` will go to
+  `http://google.com`, and not `http://zulip.com/google.com` which
+  is the default behavior.
+
+* Set `target="_blank"` and `title=`(the url) on every link tag so
+  clicking always opens a new window
+
+* Disable link-by-reference syntax, `[foo][bar]` ... `[bar]: http://google.com`
+
+### Code
+
+* Enable fenced code block extension, with syntax highlighting
+
+* Disable line-numbering within fenced code blocks -- the `<table>`
+  output confused our web client code.
+
+### Other
+
+* Disable headings, both `# foo` and `== foo ==` syntax: they don't
+  make much sense for chat messages.
+
+* Disabled images.
+
+* Allow embedding any avatar as a tiny (list bullet size) image.  This
+  is used primarily by version control integrations.
+
+* We added the `~~~ quote` block quote syntax.

docs/mypy.md

@@ -0,0 +1,165 @@
+# Testing with the mypy Python static type checker
+
+[mypy](http://mypy-lang.org/) is a compile-time static type checker
+for Python, allowing optional, gradual typing of Python code.  Zulip
+is using mypy's Python 2 compatible syntax for type annotations, which
+means that type annotations are written inside comments that start
+with `# type: `.  Here's a brief example of the mypy syntax we're
+using in Zulip:
+
+```
+user_dict = {} # type: Dict[str, UserProfile]
+
+def get_user_profile_by_email(email):
+    # type: (str) -> UserProfile
+    ... # Actual code of the function here
+```
+
+You can learn more about it at:
+
+* [The mypy cheat
+  sheet](https://github.com/python/mypy/blob/master/docs/source/cheat_sheet.rst)
+  is the best resource for quickly understanding how to write the PEP
+  484 type annotations used by mypy correctly.
+
+* The [Python 2 type annotation syntax spec in PEP
+  484](https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code)
+
+* [Using mypy with Python 2 code](http://mypy.readthedocs.io/en/latest/python2.html)
+
+The mypy type checker is run automatically as part of Zulip's Travis
+CI testing process.
+
+## Zulip goals
+
+Zulip is hoping to reach 100% of the codebase annotated with mypy
+static types, and then enforce that it stays that way.  Our current
+coverage is shown in
+[Coveralls](https://coveralls.io/github/zulip/zulip).
+
+## Installing mypy
+
+If you installed Zulip's development environment correctly, mypy
+should already be installed inside the Python 3 virtualenv at
+`zulip-py3-venv` (mypy only supports Python 3).  If it isn't installed
+(e.g. because you haven't reprovisioned recently), you can run
+`tools/install-mypy` to install it.
+
+## Running mypy on Zulip's code locally
+
+To run mypy on Zulip's python code, run the command:
+
+    tools/run-mypy
+
+It will output errors in the same style of a compiler.  For example,
+if your code has a type error like this:
+
+```
+foo = 1
+foo = '1'
+```
+
+you'll get an error like this:
+
+```
+test.py: note: In function "test":
+test.py:200: error: Incompatible types in assignment (expression has type "str", variable has type "int")
+```
+
+If you need help interpreting or debugging mypy errors, please feel
+free to mention @sharmaeklavya2 or @timabbott on your pull request (or
+email zulip-devel@googlegroups.com) to get help; we'd love to both
+build a great troubleshooting guide in this doc and also help
+contribute improvements to error messages upstream.
+
+Since mypy is a new tool under rapid development and occasionally
+makes breaking changes, Zulip is using a pinned version of mypy from
+its [git repository](https://github.com/python/mypy) rather than
+tracking the (older) latest mypy release on pypi.
+
+## Excluded files
+
+Since several python files in Zulip's code don't pass mypy's checks
+(even for unannotated code) right now, a list of files to be excluded
+from the check for CI is present in tools/run-mypy.
+
+To run mypy on all python files, ignoring the exclude list, you can
+pass the `--all` option to tools/run-mypy.
+
+    tools/run-mypy --all
+
+If you type annotate some of those files so that they pass without
+errors, please remove them from the exclude list.
+
+## Mypy is there to find bugs in Zulip before they impact users
+
+For the purposes of Zulip development, you can treat `mypy` like a
+much more powerful linter that can catch a wide range of bugs.  If,
+after running tools/run-mypy on your Zulip branch, you get mypy
+errors, it's important to get to the bottom of the issue, not just do
+something quick to silence the warnings.  Possible explanations include:
+
+* A bug in any new type annotations you added.
+* A bug in the existing type annotations.
+* A bug in Zulip!
+* Some Zulip code is correct but confusingly reuses variables with
+  different types.
+* A bug in mypy (though this is increasingly rare as mypy is now
+  fairly mature as a project).
+
+Each explanation has its own solution, but in every case the result
+should be solving the mypy warning in a way that makes the Zulip
+codebase better.  If you need help understanding an issue, please feel
+free to, mention @sharmaeklavya2 or @timabbott on the relevant pull
+request or issue on GitHub.
+
+If you think you have found a bug in Zulip or mypy, inform the zulip
+developers by opening an issue on [Zulip's github
+repository](https://github.com/zulip/zulip/issues) or posting on
+[zulip-devel](https://groups.google.com/d/forum/zulip-devel).  If it's
+indeed a mypy bug, we can help with reporting it upstream.
+
+## Annotating strings
+
+In python 3, strings can have non-ASCII characters without any problems.
+Such characters are required to support languages which use non-latin
+scripts like Japanese and Hindi.  They are also needed to support special
+characters like mathematical symbols, musical symbols, etc.
+In python 2, however, `str` generally doesn't work well with non-ASCII
+characters.  That's why `unicode` was introduced in python 2.
+
+But there are problems with the `unicode` and `str` system.  Implicit
+conversions between `str` and `unicode` use the `ascii` codec, which
+fails on strings containing non-ASCII characters.  Such errors are hard
+to detect by people who always write in English.  To minimize such
+implicit conversions, we should have a strict separation between `str`
+and `unicode` in python 2.  It might seem that using `unicode` everywhere
+will solve all problems, but unfortunately it doesn't.  This is because
+some parts of the standard library and the python language (like keyword
+argument unpacking) insist that parameters passed to them are `str`.
+
+To make our code work correctly on python 2, we have to identify strings
+which contain data which could come from non-ASCII sources like stream
+names, people's names, domain names, content of messages, emails, etc.
+These strings should be `unicode`.  We also have to identify strings
+which should be `str` like Exception names, attribute names, parameter
+names, etc.
+
+Mypy can help with this.  We just have to annotate each string as either
+`str` or `unicode` and mypy's static type checking will tell us if we
+are incorrectly mixing the two.  However, `unicode` is not defined in
+python 3.  We want our code to be python 3 compatible in the future.
+This can be achieved using 'six', a Python 2 and 3 compatibility library.
+
+`six.text_type` is defined as `str` on python 3 and as `unicode` on
+python 2.  We'll be using `text_type` (instead of `unicode`) and `str`
+to annotate strings in Zulip's code.  We follow the style of doing
+`from six import text_type` and using `text_type` for annotation instead
+of doing `import six` and using `six.text_type` for annotation, because
+`text_type` is used so extensively for type annotations that we don't
+need to be that verbose.
+
+Sometimes you'll find that you have to convert strings from one type to
+another.  `zerver/lib/str_utils.py` has utility functions to help with that.
+It also has documentation (in docstrings) which explains the right way
+to use them.

docs/new-feature-tutorial.md

@@ -0,0 +1,218 @@
+How to write a new application feature
+======================================
+
+The changes needed to add a new feature will vary, of course, but this
+document provides a general outline of what you may need to do, as well
+as an example of the specific steps needed to add a new feature: adding
+a new option to the application that is dynamically synced through the
+data system in real-time to all browsers the user may have open.
+
+General Process
+---------------
+
+### Adding a field to the database
+
+**Update the model:** The server accesses the underlying database in
+`zerver/ models.py`. Add a new field in the appropriate class.
+
+**Create and run the migration:** To create and apply a migration, run:
+:
+
+./manage.py makemigrations ./manage.py migrate
+
+**Test your changes:** Once you've run the migration, restart memcached
+on your development server (`/etc/init.d/memcached restart`) and then
+restart `run-dev.py` to avoid interacting with cached objects.
+
+### Backend changes
+
+**Database interaction:** Add any necessary code for updating and
+interacting with the database in `zerver/lib/actions.py`. It should
+update the database and send an event announcing the change.
+
+**Application state:** Modify the `fetch_initial_state_data` and
+`apply_events` functions in `zerver/lib/actions.py` to update the state
+based on the event you just created.
+
+**Backend implementation:** Make any other modifications to the backend
+required for your change.
+
+**Testing:** At the very least, add a test of your event data flowing
+through the system in `test_events.py`.
+
+### Frontend changes
+
+**JavaScript:** Zulip's JavaScript is located in the directory
+`static/js/`. The exact files you may need to change depend on your
+feature. If you've added a new event that is sent to clients, be sure to
+add a handler for it to `static/js/server_events.js`.
+
+**CSS:** The primary CSS file is `static/styles/zulip.css`. If your new
+feature requires UI changes, you may need to add additional CSS to this
+file.
+
+**Templates:** The initial page structure is rendered via Jinja2
+templates located in `templates/zerver`. For JavaScript, Zulip uses
+Handlebars templates located in `static/templates`. Templates are
+precompiled as part of the build/deploy process.
+
+**Testing:** There are two types of frontend tests: node-based unit
+tests and blackbox end-to-end tests. The blackbox tests are run in a
+headless browser using Casper.js and are located in
+`frontend_tests/casper_tests/`. The unit tests use Node's `assert`
+module are located in `frontend_tests/node_tests/`. For more information
+on writing and running tests see the testing documentation \<testing\>.
+
+Example Feature
+---------------
+
+This example describes the process of adding a new setting to Zulip: a
+flag that restricts inviting new users to admins only (the default
+behavior is that any user can invite other users). It is based on an
+actual Zulip feature, and you can review [the original commit in the
+Zulip git
+repo](https://github.com/zulip/zulip/commit/5b7f3466baee565b8e5099bcbd3e1ccdbdb0a408).
+(Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the
+migration format has changed.)
+
+First, update the database and model to store the new setting. Add a new
+boolean field, `realm_invite_by_admins_only`, to the Realm model in
+`zerver/models.py`.
+
+Then create a Django migration that adds a new field,
+`invite_by_admins_only`, to the `zerver_realm` table.
+
+In `zerver/lib/actions.py`, create a new function named
+`do_set_realm_invite_by_admins_only`. This function will update the
+database and trigger an event to notify clients when this setting
+changes. In this case there was an existing `realm|update` event type
+which was used for setting similar flags on the Realm model, so it was
+possible to add a new property to that event rather than creating a new
+one. The property name matches the database field to make it easy to
+understand what it indicates.
+
+The second argument to `send_event` is the list of users whose browser
+sessions should be notified. Depending on the setting, this can be a
+single user (if the setting is a personal one, like time display
+format), only members in a particular stream or all active users in a
+realm. :
+
+    # zerver/lib/actions.py
+
+    def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only):
+      realm.invite_by_admins_only = invite_by_admins_only
+      realm.save(update_fields=['invite_by_admins_only'])
+      event = dict(
+        type="realm",
+        op="update",
+        property='invite_by_admins_only',
+        value=invite_by_admins_only,
+      )
+      send_event(event, active_user_ids(realm))
+      return {}
+
+You then need to add code that will handle the event and update the
+application state. In `zerver/lib/actions.py` update the
+`fetch_initial_state` and `apply_events` functions. :
+
+    def fetch_initial_state_data(user_profile, event_types, queue_id):
+      # ...
+      state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`
+
+In this case you don't need to change `apply_events` because there is
+already code that will correctly handle the realm update event type: :
+
+    def apply_events(state, events, user_profile):
+      for event in events:
+        # ...
+        elif event['type'] == 'realm':
+           field = 'realm_' + event['property']
+           state[field] = event['value']
+
+You then need to add a view for clients to access that will call the
+newly-added `actions.py` code to update the database. This example
+feature adds a new parameter that should be sent to clients when the
+application loads and be accessible via JavaScript, and there is already
+a view that does this for related flags: `update_realm`. So in this
+case, we can add out code to the existing view instead of creating a
+new one. :
+
+    # zerver/views/__init__.py
+
+    def home(request):
+      # ...
+      page_params = dict(
+        # ...
+        realm_invite_by_admins_only = register_ret['realm_invite_by_admins_only'],
+        # ...
+      )
+
+Since this feature also adds a checkbox to the admin page, and adds a
+new property the Realm model that can be modified from there, you also
+need to make changes to the `update_realm` function in the same file: :
+
+    # zerver/views/__init__.py
+
+    def update_realm(request, user_profile,
+      name=REQ(validator=check_string, default=None),
+      restricted_to_domain=REQ(validator=check_bool, default=None),
+      invite_by_admins_only=REQ(validator=check_bool,default=None)):
+
+      # ...
+
+      if invite_by_admins_only is not None and
+        realm.invite_by_admins_only != invite_by_admins_only:
+          do_set_realm_invite_by_admins_only(realm, invite_by_admins_only)
+          data['invite_by_admins_only'] = invite_by_admins_only
+
+Then make the required front end changes: in this case a checkbox needs
+to be added to the admin page (and its value added to the data sent back
+to server when a realm is updated) and the change event needs to be
+handled on the client.
+
+To add the checkbox to the admin page, modify the relevant template,
+`static/templates/admin_tab.handlebars` (omitted here since it is
+relatively straightforward). Then add code to handle changes to the new
+form control in `static/js/admin.js`. :
+
+    var url = "/json/realm";
+    var new_invite_by_admins_only =
+      $("#id_realm_invite_by_admins_only").prop("checked");
+    data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);
+
+    channel.patch({
+      url: url,
+      data: data,
+      success: function (data) {
+        # ...
+        if (data.invite_by_admins_only) {
+          ui.report_success("New users must be invited by an admin!", invite_by_admins_only_status);
+        } else {
+          ui.report_success("Any user may now invite new users!", invite_by_admins_only_status);
+        }
+        # ...
+      }
+    });
+
+Finally, update `server_events.js` to handle related events coming from
+the server. :
+
+    # static/js/server_events.js
+
+    function get_events_success(events) {
+      # ...
+      var dispatch_event = function dispatch_event(event) {
+          switch (event.type) {
+          # ...
+          case 'realm':
+            if (event.op === 'update' && event.property === 'invite_by_admins_only') {
+              page_params.realm_invite_by_admins_only = event.value;
+          }
+      }
+    }
+
+Any code needed to update the UI should be placed in `dispatch_event`
+callback (rather than the `channel.patch`) function. This ensures the
+appropriate code will run even if the changes are made in another
+browser window. In this example most of the changes are on the backend,
+so no UI updates are required.

docs/new-feature-tutorial.rst

@@ -1,216 +0,0 @@
-====================
-New Feature Tutorial
-====================
-
-The changes needed to add a new feature will vary, of course, but this document
-provides a general outline of what you may need to do, as well as an example of
-the specific steps needed to add a new feature: adding a new option to the 
-application that is dynamically synced through the data system in real-time to
-all browsers the user may have open.
-
-General Process
-===============
-
-Adding a field to the database
-------------------------------
-
-**Update the model:** The server accesses the underlying database in `zerver/
-models.py`. Add a new field in the appropriate class.
-
-**Create and run the migration:** To create and apply a migration, run: ::
-
-./manage.py makemigrations
-./manage.py migrate
-
-**Test your changes:** Once you've run the migration, restart memcached on your 
-development server (``/etc/init.d/memcached restart``) and then restart 
-``run-dev.py`` to avoid interacting with cached objects.
-
-Backend changes
----------------
-
-**Database interaction:** Add any necessary code for updating and interacting
-with the database in ``zerver/lib/actions.py``. It should update the database and 
-send an event announcing the change.
-
-**Application state:** Modify the ``fetch_initial_state_data`` and ``apply_events`` 
-functions in ``zerver/lib/actions.py`` to update the state based on the event you 
-just created.
-
-**Backend implementation:** Make any other modifications to the backend required for 
-your change.
-
-**Testing:** At the very least, add a test of your event data flowing through 
-the system in ``test_events.py``.
-
-
-Frontend changes
-----------------
-
-**JavaScript:** Zulip's JavaScript is located in the directory ``static/js/``. 
-The exact files you may need to change depend on your feature. If you've added a 
-new event that is sent to clients, be sure to add a handler for it to
-``static/js/server_events.js``.
-
-**CSS:** The primary CSS file is ``static/styles/zulip.css``. If your new 
-feature requires UI changes, you may need to add additional CSS to this file.
-
-**Templates:** The initial page structure is rendered via Django templates 
-located in ``template/server``. For JavaScript, Zulip uses Handlebars templates located in
-``static/templates``. Templates are precompiled as part of the build/deploy
-process.
-
-**Testing:** There are two types of frontend tests: node-based unit tests and 
-blackbox end-to-end tests. The blackbox tests are run in a headless browser 
-using Casper.js and are located in ``zerver/tests/frontend/tests/``. The unit
-tests use Node's ``assert`` module are located in ``zerver/tests/frontend/node/``.
-For more information on writing and running tests see the :doc:`testing 
-documentation <testing>`.
-
-Example Feature
-===============
-
-This example describes the process of adding a new setting to Zulip:
-a flag that restricts inviting new users to admins only (the default behavior
-is that any user can invite other users). It is based on an actual Zulip feature,
-and you can review `the original commit in the Zulip git repo <https://github.com/zulip/zulip/commit/5b7f3466baee565b8e5099bcbd3e1ccdbdb0a408>`_.
-(Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the migration
-format has changed.)
-
-First, update the database and model to store the new setting. Add a 
-new boolean field, ``realm_invite_by_admins_only``, to the Realm model in
-``zerver/models.py``.
-
-Then create a Django migration that adds a new field, ``invite_by_admins_only``,
-to the ``zerver_realm`` table.
-
-In ``zerver/lib/actions.py``, create a new function named 
-``do_set_realm_invite_by_admins_only``. This function will update the database
-and trigger an event to notify clients when this setting changes. In this case 
-there was an exisiting ``realm|update`` event type which was used for setting 
-similar flags on the Realm model, so it was possible to add a new property to 
-that event rather than creating a new one. The property name matches the 
-database field to make it easy to understand what it indicates.
-
-The second argument to ``send_event`` is the list of users whose browser 
-sessions should be notified. Depending on the setting, this can be a single user
-(if the setting is a personal one, like time display format), only members in a
-particular stream or all active users in a realm. ::
-
-  # zerver/lib/actions.py
-
-  def do_set_realm_invite_by_admins_only(realm, invite_by_admins_only):
-    realm.invite_by_admins_only = invite_by_admins_only
-    realm.save(update_fields=['invite_by_admins_only'])
-    event = dict(
-      type="realm",
-      op="update",
-      property='invite_by_admins_only',
-      value=invite_by_admins_only,
-    )
-    send_event(event, active_user_ids(realm))
-    return {}
-
-You then need to add code that will handle the event and update the application
-state. In ``zerver/lib/actions.py`` update the ``fetch_initial_state`` and
-``apply_events`` functions. ::
-
-  def fetch_initial_state_data(user_profile, event_types, queue_id):
-    # ...
-    state['realm_invite_by_admins_only'] = user_profile.realm.invite_by_admins_only`
-
-In this case you don't need to change ``apply_events`` because there is already
-code that will correctly handle the realm update event type: ::
-
-  def apply_events(state, events, user_profile):
-    for event in events:
-      # ...
-      elif event['type'] == 'realm':
-         field = 'realm_' + event['property']
-         state[field] = event['value']
-
-You then need to add a view for clients to access that will call the newly-added
-``actions.py`` code to update the database. This example feature adds a new
-parameter that should be sent to clients when the application loads and be
-accessible via JavaScript, and there is already a view that does this for
-related flags: ``update_realm``. So in this case, we can add out code to the
-exisiting view instead of creating a new one. ::
-
-  # zerver/views/__init__.py
-
-  def home(request):
-    # ...
-    page_params = dict(
-      # ...
-      realm_invite_by_admins_only = register_ret['realm_invite_by_admins_only'],
-      # ...
-    )
-
-Since this feature also adds a checkbox to the admin page, and adds a new
-property the Realm model that can be modified from there, you also need to make
-changes to the ``update_realm`` function in the same file: ::
-
-  # zerver/views/__init__.py
-
-  def update_realm(request, user_profile,
-    name=REQ(validator=check_string, default=None),
-    restricted_to_domain=REQ(validator=check_bool, default=None),
-    invite_by_admins_only=REQ(validator=check_bool,default=None)):
-
-    # ...
-
-    if invite_by_admins_only is not None and
-      realm.invite_by_admins_only != invite_by_admins_only:
-        do_set_realm_invite_by_admins_only(realm, invite_by_admins_only)
-        data['invite_by_admins_only'] = invite_by_admins_only
-
-Then make the required front end changes: in this case a checkbox needs to be
-added to the admin page (and its value added to the data sent back to server
-when a realm is updated) and the change event needs to be handled on the client.
-
-To add the checkbox to the admin page, modify the relevant template,
-``static/templates/admin_tab.handlebars`` (omitted here since it is relatively
-straightforward). Then add code to handle changes to the new form control in
-``static/js/admin.js``. ::
-
-  var url = "/json/realm";
-  var new_invite_by_admins_only =
-    $("#id_realm_invite_by_admins_only").prop("checked");
-  data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);
-
-  channel.patch({
-    url: url,
-    data: data,
-    success: function (data) {
-      # ...
-      if (data.invite_by_admins_only) {
-        ui.report_success("New users must be invited by an admin!", invite_by_admins_only_status);
-      } else {
-        ui.report_success("Any user may now invite new users!", invite_by_admins_only_status);
-      }
-      # ...
-    }
-  });
-
-Finally, update ``server_events.js`` to handle related events coming from the
-server. ::
-
-  # static/js/server_events.js
-
-  function get_events_success(events) {
-    # ...
-    var dispatch_event = function dispatch_event(event) {
-        switch (event.type) {
-        # ...
-        case 'realm':
-          if (event.op === 'update' && event.property === 'invite_by_admins_only') {
-            page_params.realm_invite_by_admins_only = event.value;
-        }
-    }
-  }
-
-Any code needed to update the UI should be placed in ``dispatch_event`` callback
-(rather than the ``channel.patch``) function. This ensures the appropriate code
-will run even if the changes are made in another browser window. In this example
-most of the changes are on the backend, so no UI updates are required.
-

docs/queuing.md

@@ -0,0 +1,81 @@
+# RabbitMQ queues
+
+Zulip uses RabbitMQ to manage a system of internal queues.  These are
+used for a variety of purposes:
+
+* Asynchronously doing expensive operations like sending email
+  notifications which can take seconds per email and thus would
+  otherwise timeout when 100s are triggered at once (E.g. inviting a
+  lot of new users to a realm).
+
+* Asynchronously doing non-time-critical somewhat expensive operations
+  like updating analytics tables (e.g. UserActivityInternal) which
+  don't have any immediate runtime effect.
+
+* Communicating events to push to clients (browsers, etc.) from the
+  main Zulip Django application process to the Tornado-based events
+  system.  Example events might be that a new message was sent, a user
+  has changed their subscriptions, etc.
+
+* Processing mobile push notifications and email mirroring system
+  messages.
+
+* Processing various errors, frontend tracebacks, and slow database
+  queries in a batched fashion.
+
+* Doing markdown rendering for messages delivered to the Tornado via
+  websockets.
+
+Needless to say, the RabbitMQ-based queuing system is an important
+part of the overall Zulip architecture, since it's in critical code
+paths for everything from signing up for account, to rendering
+messages, to delivering updates to clients.
+
+We use the `pika` library to interface with RabbitMQ, using a simple
+custom integration defined in `zerver/lib/queue.py`.
+
+### Adding a new queue processor
+
+To add a new queue processor:
+
+* Define the processor in `zerver/worker/queue_processors.py` using
+  the `@assign_queue` decorator; it's pretty easy to get the template
+  for an existing similar queue processor.  This suffices to test your
+  queue worker in the Zulip development environment, though you'll
+  need to restart `tools/run-dev.py` in order to run your new queue
+  processor.  You can also run a single queue processor manually using
+  e.g. `./manage.py process_queue --queue=user_activity`.
+
+* So that supervisord will known to run the queue processor in
+  production, you will need to define a program entry for it in
+  `servers/puppet/modules/zulip/files/supervisor/conf.d/zulip.conf`
+  and add it to the `zulip-workers` group further down in the file.
+
+* For monitoring, you need to add a check that your worker is running
+  to puppet/zulip/files/cron.d/rabbitmq-numconsumers if it's a
+  one-at-a-time consumer like `user_activity_internal` or a custom
+  nagios check if it is a bulk processor like `slow_queries`.
+
+### Publishing events into a queue
+
+You can publish events to a RabbitMQ queue using the
+`queue_json_publish` function defined in `zerver/lib/queue.py`.
+
+### Clearing a RabbitMQ queue
+
+If you need to clear a queue (delete all the events in it), run
+`./manage.py purge_queue <queue_name>`, for example:
+
+```
+./manage.py purge_queue user_activity
+```
+
+You can also use the amqp tools directly.  Install `amqp-tools` from
+apt and then run:
+
+```
+amqp-delete-queue --username=zulip --password='...' --server=localhost \
+   --queue=user_presence
+```
+
+with the RabbitMQ password from `/etc/zulip/zulip-secrets.conf`.

docs/readme-symlink.md

@@ -0,0 +1 @@
+../README.md