Update changelog for Zulip 1.4.3 release.

A bug in Zulip's implementation of the "stream exists" endpoint meant
that any user of a Zulip server could subscribe to an invite-only
stream without needing to be invited by using the "autosubscribe"
argument.

Thanks to Rafid Aslam for discovering this issue.

.coveralls.yml

@@ -0,0 +1,2 @@
+service_name: travis-pro
+repo_token: hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG

.gitattributes

@@ -18,3 +18,4 @@
 /frontend_tests export-ignore
 /node_modules export-ignore
 /humbug export-ignore
+/locale export-ignore

.gitignore

@@ -1,46 +1,26 @@
 *.pyc
 *~
-/all_messages_log.*
-/event_log/*
-/digest.log*
-/errors.log*
-/manage.log*
-/server.log*
-/workers.log*
-/email-deliverer.log
-/email-mirror.log
-/sync_ldap_user_data.log
-/update-prod-static.log
-frontend_tests/casper_tests/server.log
-frontend_tests/casper_lib/test_credentials.js
-memcached_prefix
 /prod-static
 /errors/*
 *.sw[po]
 *.DS_Store
-event_queues.pickle
 stats/
-zerver/fixtures/available-migrations
-zerver/fixtures/migration-status
-zerver/fixtures/test_data1.json
 .kdev4
+.idea
 zulip.kdev4
-remote_cache_prefix
 coverage/
+.coverage
 /queue_error
-.test-js-with-node.html
 .kateproject.d/
 .kateproject
 *.kate-swp
-event_queues.json
 .vagrant
 /zproject/dev-secrets.conf
 static/js/bundle.js
 static/third/gemoji/
 static/third/zxcvbn/
-tools/emoji_dump/bitmaps/
-tools/emoji_dump/*.ttx
-tools/phantomjs
+static/locale/language_options.json
 node_modules
-uploads/
-test_uploads/
+npm-debug.log
+*.mo
+var/*

.travis.yml

@@ -1,28 +1,47 @@
+dist: trusty
 before_install:
    - nvm install 0.10
 install:
+  - pip install coveralls
   - tools/travis/setup-$TEST_SUITE
+  - tools/clean-venv-cache --travis
 cache:
   - apt: false
   - directories:
-    - /srv/phantomjs
+    - $HOME/phantomjs
+    - $HOME/zulip-venv-cache
+    - node_modules
+    - $HOME/node
 env:
-  - TEST_SUITE=frontend
-  - TEST_SUITE=backend
-  - TEST_SUITE=production
-  - TEST_SUITE=py3k
+  global:
+    - COVERALLS_PARALLEL=true
+    - COVERALLS_SERVICE_NAME=travis-pro
+    - COVERALLS_REPO_TOKEN=hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG
+    - BOTO_CONFIG=/tmp/nowhere
+  matrix:
+    - TEST_SUITE=frontend
+    - TEST_SUITE=backend
 language: python
 python:
   - "2.7"
+  - "3.4"
 matrix:
   include:
     - python: "3.4"
-      env: TEST_SUITE=mypy
+      env: TEST_SUITE=static-analysis
+    - python: "2.7"
+      env: TEST_SUITE=production
+      sudo: required
 # command to run tests
 script:
+  - unset GEM_PATH
   - ./tools/travis/$TEST_SUITE
 sudo: required
 services:
 - docker
 addons:
   postgresql: "9.3"
+after_success:
+  coveralls
+notifications:
+  webhooks: https://coveralls.io/webhook?repo_token=$COVERALLS_REPO_TOKEN

.tx/config

@@ -0,0 +1,17 @@
+[main]
+host = https://www.transifex.com
+
+[zulip.djangopo]
+source_file = static/locale/en/LC_MESSAGES/django.po
+source_lang = en
+type = PO
+file_filter = static/locale/<lang>/LC_MESSAGES/django.po
+lang_map = zh-Hans: zh_CN
+
+[zulip.translationsjson]
+source_file = static/locale/en/translations.json
+source_lang = en
+type = KEYVALUEJSON
+file_filter = static/locale/<lang>/translations.json
+lang_map = zh-Hans: zh-CN
+

Dockerfile

@@ -9,4 +9,7 @@ RUN apt-get update && apt-get install -y \
 RUN useradd -d /home/zulip -m zulip && echo 'zulip ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 
 USER zulip
+
+RUN ln -nsf /srv/zulip ~/zulip
+
 WORKDIR /srv/zulip

README.dev.md

@@ -1,554 +0,0 @@
-
-Installing the Zulip Development environment
-============================================
-
-You will need a machine with at least 2GB of RAM available (see
-https://github.com/zulip/zulip/issues/32 for a plan for how to
-dramatically reduce this requirement).
-
-Start by cloning this repository: `git clone https://github.com/zulip/zulip.git`
-
-Using Vagrant
--------------
-
-This is the recommended approach for all platforms, and will install
-the Zulip development environment inside a VM or container and works
-on any platform that supports Vagrant.
-
-The best performing way to run the Zulip development environment is
-using an LXC container on a Linux host, but we support other platforms
-such as Mac via Virtualbox (but everything will be 2-3x slower).
-
-* If your host is Ubuntu 15.04 or newer, you can install and configure
-  the LXC Vagrant provider directly using apt:
-  ```
-  sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir
-  vagrant plugin install vagrant-lxc
-  ```
-  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
-
-* If your host is Ubuntu 14.04, you will need to [download a newer
-  version of Vagrant][vagrant-dl], and then do the following:
-  ```
-  sudo apt-get install lxc lxc-templates cgroup-lite redir
-  sudo dpkg -i vagrant*.deb # in directory where you downloaded vagrant
-  vagrant plugin install vagrant-lxc
-  ```
-  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
-
-* For other Linux hosts with a kernel above 3.12, [follow the Vagrant
-  LXC installation instructions][vagrant-lxc] to get Vagrant with LXC
-  for your platform.
-
-* If your host is OS X or older Linux, [download VirtualBox][vbox-dl],
-  [download Vagrant][vagrant-dl], and install them both.
-
-* If you're on OS X and have VMWare, it should be possible to patch
-  Vagrantfile to use the VMWare vagrant provider which should perform
-  much better than Virtualbox.  Patches to do this by default if
-  VMWare is available are welcome!
-
-* On Windows: You can use Vagrant and Virtualbox/VMWare on Windows
-  with Cygwin, similar to the Mac setup.  Be sure to create your git
-  clone using `git clone https://github.com/zulip/zulip.git -c
-  core.autocrlf=false` to avoid Windows line endings being added to
-  files (this causes weird errors).
-
-[vagrant-dl]: https://www.vagrantup.com/downloads.html
-[vagrant-lxc]: https://github.com/fgrehm/vagrant-lxc
-[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
-[avoiding-sudo]: https://github.com/fgrehm/vagrant-lxc#avoiding-sudo-passwords
-
-Once that's done, simply change to your zulip directory and run
-`vagrant up` in your terminal to install the development server.  This
-will take a long time on the first run because Vagrant needs to
-download the Ubuntu Trusty base image, but later you can run `vagrant
-destroy` and then `vagrant up` again to rebuild the environment and it
-will be much faster.
-
-Once that finishes, you can run the development server as follows:
-
-```
-vagrant ssh -- -L9991:localhost:9991
-# Now inside the container
-cd /srv/zulip
-source /srv/zulip-venv/bin/activate
-./tools/run-dev.py --interface=''
-```
-
-To get shell access to the virtual machine running the server to run
-lint, management commands, etc., use `vagrant ssh`.
-
-(A small note on tools/run-dev.py: the `--interface=''` option will
-make the development server listen on all network interfaces.  While
-this is correct for the Vagrant guest sitting behind a NAT, you
-probably don't want to use that option when using run-dev.py in other
-environments).
-
-At this point you should [read about using the development
-environment][using-dev].
-
-[using-dev]: #using-the-development-environment
-
-## Specifying a proxy
-
-If you need to use a proxy server to access the Internet, you will
-need to specify the proxy settings before running `Vagrant up`.
-First, install the Vagrant plugin `vagrant-proxyconf`:
-
-```
-vagrant plugin install vagrant-proxyconf.
-```
-
-Then create `~/.zulip-vagrant-config` and add the following lines to
-it (with the appropriate values in it for your proxy):
-
-```
-HTTP_PROXY http://proxy_host:port
-HTTPS_PROXY http://proxy_host:port
-NO_PROXY localhost,127.0.0.1,.example.com
-
-```
-
-Now run `vagrant up` in your terminal to install the development
-server. If you ran `vagrant up` before and failed, you'll need to run
-`vagrant destroy` first to clean up the failed installation.
-
-Using provision.py without Vagrant
-----------------------------------
-
-If you'd like to install a Zulip development environment on a server
-that's already running Ubuntu 14.04 Trusty, you can do that by just
-running:
-
-```
-sudo apt-get update
-python /srv/zulip/provision.py
-
-cd /srv/zulip
-source /srv/zulip-venv/bin/activate
-./tools/run-dev.py
-```
-
-Note that there is no supported uninstallation process without Vagrant
-(with Vagrant, you can just do `vagrant destroy` to clean up the
-development environment).
-
-By hand
--------
-If you really want to install everything by hand, the below
-instructions should work.
-
-Install the following non-Python dependencies:
- * libffi-dev — needed for some Python extensions
- * postgresql 9.1 or later — our database (client, server, headers)
- * nodejs 0.10 (and npm)
- * memcached (and headers)
- * rabbitmq-server
- * libldap2-dev
- * python-dev
- * redis-server — rate limiting
- * tsearch-extras — better text search
- * libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs
-
-### On Debian or Ubuntu systems:
-
-```
-sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
-    memcached rabbitmq-server libldap2-dev redis-server \
-    postgresql-server-dev-all libmemcached-dev python-dev \
-    hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \
-    puppet gettext
-
-# If on 12.04 or wheezy:
-sudo apt-get install postgresql-9.1
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
-
-# If on 14.04:
-sudo apt-get install postgresql-9.3
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-
-# If on 15.04 or jessie:
-sudo apt-get install postgresql-9.4
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb
-sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
-```
-
-Now continue with the "All systems" instructions below.
-
-### On Fedora 22 (experimental):
-
-These instructions are experimental and may have bugs; patches
-welcome!
-
-```
-sudo dnf install libffi-devel memcached rabbitmq-server \
-    openldap-devel python-devel redis postgresql-server \
-    postgresql-devel postgresql libmemcached-devel freetype-devel \
-    nodejs npm yuicompressor closure-compiler gettext
-```
-
-Now continue with the Common to Fedora/CentOS instructions below.
-
-### On CentOS 7 Core (experimental):
-
-These instructions are experimental and may have bugs; patches
-welcome!
-
-```
-# Add user zulip to the system (not necessary if you configured zulip
-# as the administrator user during the install process of CentOS 7).
-useradd zulip
-
-# Create a password for zulip user
-passwd zulip
-
-# Allow zulip to sudo
-visudo
-# Add this line after line `root    ALL=(ALL)       ALL`
-zulip   ALL=(ALL)       ALL
-
-# Switch to zulip user
-su zulip
-
-# Enable EPEL 7 repo so we can install rabbitmq-server, redis and
-# other dependencies
-sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
-
-# Install dependencies
-sudo yum install libffi-devel memcached rabbitmq-server openldap-devel
-    python-devel redis postgresql-server postgresql-devel postgresql \
-    libmemcached-devel wget python-pip openssl-devel freetype-devel \
-    libjpeg-turbo-devel zlib-devel nodejs yuicompressor \
-    closure-compiler gettext
-
-# We need these packages to compile tsearch-extras
-sudo yum groupinstall "Development Tools"
-
-# clone Zulip's git repo and cd into it
-cd && git clone https://github.com/zulip/zulip && cd zulip/
-
-## NEEDS TESTING: The next few DB setup items may not be required at all.
-# Initialize the postgres db
-sudo postgresql-setup initdb
-
-# Edit the postgres settings:
-sudo vi /var/lib/pgsql/data/pg_hba.conf
-
-# Change these lines:
-host    all             all             127.0.0.1/32            ident
-host    all             all             ::1/128                 ident
-# to this:
-host    all             all             127.0.0.1/32            md5
-host    all             all             ::1/128                 md5
-```
-
-Now continue with the Common to Fedora/CentOS instructions below.
-
-### On OpenBSD 5.8 (experimental):
-
-These instructions are experimental and may have bugs; patches
-welcome!
-
-```
-doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \
-    memcached node libmemcached py-Pillow py-cryptography py-cffi
-
-# Get tsearch_extras and build it (using a modified version which
-# aliases int4 on OpenBSD):
-git clone https://github.com/blablacio/tsearch_extras
-cd tsearch_extras
-gmake && sudo gmake install
-
-# Point environment to custom include locations and use newer GCC
-# (needed for Node modules):
-export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl"
-export CXX=eg++
-
-# Create tsearch_data directory:
-sudo mkdir /usr/local/share/postgresql/tsearch_data
-
-
-# Hack around missing dictionary files -- need to fix this to get the
-# proper dictionaries from what in debian is the hunspell-en-us
-# package.
-sudo touch /usr/local/share/postgresql/tsearch_data/english.stop
-sudo touch /usr/local/share/postgresql/tsearch_data/en_us.dict
-sudo touch /usr/local/share/postgresql/tsearch_data/en_us.affix
-```
-
-Now continue with the All Systems instructions below.
-
-### Common to Fedora/CentOS instructions
-
-```
-# Build and install postgres tsearch-extras module
-wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz
-tar xvzf tsearch-extras_0.1.3.tar.gz
-cd ts2
-make
-sudo make install
-
-# Hack around missing dictionary files -- need to fix this to get the
-# proper dictionaries from what in debian is the hunspell-en-us
-# package.
-sudo touch /usr/share/pgsql/tsearch_data/english.stop
-sudo touch /usr/share/pgsql/tsearch_data/en_us.dict
-sudo touch /usr/share/pgsql/tsearch_data/en_us.affix
-
-# Edit the postgres settings:
-sudo vi /var/lib/pgsql/data/pg_hba.conf
-
-# Add this line before the first uncommented line to enable password
-# auth:
-host    all             all             127.0.0.1/32            md5
-
-# Start the services
-sudo systemctl start redis memcached rabbitmq-server postgresql
-
-# Enable automatic service startup after the system startup
-sudo systemctl enable redis rabbitmq-server memcached postgresql
-```
-
-Finally continue with the All Systems instructions below.
-
-### All Systems:
-
-```
-pip install --no-deps -r requirements.txt
-./tools/install-phantomjs
-./tools/install-mypy
-./tools/download-zxcvbn
-./tools/emoji_dump/build_emoji
-./scripts/setup/generate_secrets.py -d
-if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/tsearch_data/; else sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.3/tsearch_data/; fi
-./scripts/setup/configure-rabbitmq
-./tools/postgres-init-dev-db
-./tools/do-destroy-rebuild-database
-./tools/postgres-init-test-db
-./tools/do-destroy-rebuild-test-database
-npm install
-```
-
-If `npm install` fails, the issue may be that you need a newer version
-of `npm`.  You can use `npm install -g npm` to update your version of
-`npm` and try again.
-
-To start the development server:
-
-```
-./tools/run-dev.py
-```
-
-… and visit [http://localhost:9991/](http://localhost:9991/).
-
-#### Proxy setup for by-hand installation
-
-If you are building the development environment on a network where a
-proxy is required to access the Internet, you will need to set the
-proxy in the environment as follows:
-
-- On Ubuntu, set the proxy environment variables using:
- ```
- export https_proxy=http://proxy_host:port
- export http_proxy=http://proxy_host:port
- ```
-
-- And set the npm proxy and https-proxy using:
- ```
- npm config set proxy http://proxy_host:port
- npm config set https-proxy http://proxy_host:port
- ```
-
-Using Docker (experimental)
----------------------------
-
-The docker instructions for development are experimental, so they may
-have bugs.  If you try them and run into any issues, please report
-them!
-
-You can also use Docker to run a Zulip development environment.
-First, you need to install Docker in your development machine
-following the [instructions][docker-install].  Some other interesting
-links for somebody new in Docker are:
-
-* [Get Started](https://docs.docker.com/linux/started/)
-* [Understand the architecture](https://docs.docker.com/engine/introduction/understanding-docker/)
-* [Docker run reference](https://docs.docker.com/engine/reference/run/)
-* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)
-
-[docker-install]: https://docs.docker.com/engine/installation/
-
-Then you should create the Docker image based on Ubuntu Linux, first
-go to the directory with the Zulip source code:
-
-```
-docker build -t user/zulipdev .
-```
-
-Now you're going to install Zulip dependencies in the image:
-
-```
-docker run -itv $(pwd):/srv/zulip -p 80:9991 user/zulipdev /bin/bash
-$ /usr/bin/python /srv/zulip/provision.py --docker
-docker ps -af ancestor=user/zulipdev
-docker commit -m "Zulip installed" <container id> user/zulipdev:v2
-```
-
-Finally you can run the docker server with:
-
-```
-docker run -itv $(pwd):/srv/zulip -p 80:9991 user/zulipdev:v2 \
-    /srv/zulip/scripts/start-dockers
-```
-
-If you want to connect to the Docker instance to build a release
-tarball you can use:
-
-```
-docker ps
-docker exec -it <container id> /bin/bash
-$ source /home/zulip/.bash_profile
-$ <Your commands>
-$ exit
-```
-
-To stop the server use:
-```
-docker ps
-docker kill <container id>
-```
-
-If you want to run all the tests you need to start the servers first,
-you can do it with:
-
-```
-docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash
-$ scripts/test-all-docker
-```
-
-You can modify the source code in your development machine and review
-the results in your browser.
-
-
-Using the Development Environment
-=================================
-
-Once the development environment is running, you can visit
-<http://localhost:9991/> in your browser.  By default, the development
-server homepage just shows a list of the users that exist on the
-server and you can login as any of them by just clicking on a user.
-This setup saves time for the common case where you want to test
-something other than the login process; to test the login process
-you'll want to change `AUTHENTICATION_BACKENDS` in the not-PRODUCTION
-case of `zproject/settings.py` from zproject.backends.DevAuthBackend
-to use the auth method(s) you'd like to test.
-
-While developing, it's helpful to watch the `run-dev.py` console
-output, which will show any errors your Zulip development server
-encounters.
-
-When you make a change, here's a guide for what you need to do in
-order to see your change take effect in Development:
-
-* If you change Javascript or CSS, you'll just need to reload the
-browser window to see changes take effect.
-
-* If you change Python code used by the the main Django/Tornado server
-processes, these services are run on top of Django's [manage.py
-runserver][django-runserver] which will automatically restart the
-Zulip Django and Tornado servers whenever you save changes to Python
-code.  You can watch this happen in the `run-dev.py` console to make
-sure the backend has reloaded.
-
-* The Python queue workers don't automatically restart when you save
-changes (or when they stop running), so you will want to ctrl-C and
-then restart `run-dev.py` manually if you are testing changes to the
-queue workers or if a queue worker has crashed.
-
-* If you change the database schema, you'll need to use the standard
-Django migrations process to create and then run your migrations; see
-the [new feature tutorial][new-feature-tutorial] for an example.
-Additionally you should check out the [detailed testing
-docs][testing-docs] for how to run the tests properly after doing a
-migration.
-
-(In production, everything runs under supervisord and thus will
-restart if it crashes, and `upgrade-zulip` will take care of running
-migrations and then cleanly restaring the server for you).
-
-[django-runserver]: https://docs.djangoproject.com/en/1.8/ref/django-admin/#runserver-port-or-address-port
-[new-feature-tutorial]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
-[testing-docs]: http://zulip.readthedocs.io/en/latest/testing.html
-
-Running the test suite
-======================
-
-For more details, especially on how to write tests, check out the
-[detailed testing docs][tdocs].
-
-[tdocs]: http://zulip.readthedocs.io/en/latest/testing.html
-
-To run all the tests, do this:
-```
-./tools/test-all
-```
-
-For the Vagrant environment, you'll want to first enter the
-environment:
-```
-vagrant ssh
-source /srv/zulip-venv/bin/activate
-cd /srv/zulip
-```
-
-This runs the linter (`tools/lint-all`) plus all of our test suites;
-they can all be run separately (just read `tools/test-all` to see
-them).  You can also run individual tests which can save you a lot of
-time debugging a test failure, e.g.:
-
-```
-./tools/lint-all # Runs all the linters in parallel
-./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube
-./tools/test-js-with-casper 10-navigation.js
-./tools/test-js-with-node # Runs all node tests but is very fast
-```
-
-The above setup instructions include the first-time setup of test
-databases, but you may need to rebuild the test database occasionally
-if you're working on new database migrations.  To do this, run:
-
-```
-./tools/postgres-init-test-db
-./tools/do-destroy-rebuild-test-database
-```
-
-Possible testing issues
-=======================
-
-- When running the test suite, if you get an error like this:
-
-  ```
-      sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not   exist
-      LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
-                                                                   ^
-  ```
-
-  … then you need to install tsearch-extras, described
-  above. Afterwards, re-run the `init*-db` and the
-  `do-destroy-rebuild*-database` scripts.
-
-- When building the development environment using Vagrant and the LXC
-  provider, if you encounter permissions errors, you may need to
-  `chown -R 1000:$(whoami) /path/to/zulip` on the host before running
-  `vagrant up` in order to ensure that the synced directory has the
-  correct owner during provision. This issue will arise if you run `id
-  username` on the host where `username` is the user running Vagrant
-  and the output is anything but 1000.
-  This seems to be caused by Vagrant behavior; for more information,
-  see [the vagrant-lxc FAQ entry about shared folder permissions
-  ][lxc-sf].
-
-[lxc-sf]: https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner)

README.md

@@ -1,5 +1,12 @@
-Zulip
-=====
+**[Zulip overview](#zulip-overview)** |
+**[Community](#community)** |
+**[Installing for dev](#installing-the-zulip-development-environment)** |
+**[Installing for production](#running-zulip-in-production)** |
+**[Ways to contribute](#ways-to-contribute)** |
+**[How to get involved](#how-to-get-involved-with-contributing-to-zulip)** |
+**[License](#license)**
+
+# Zulip overview
 
 Zulip is a powerful, open source group chat application. Written in
 Python and using the Django framework, Zulip supports both private
@@ -12,28 +19,45 @@ missed-message emails, desktop apps, and much more.
 Further information on the Zulip project and its features can be found
 at https://www.zulip.org.
 
-[![Build Status][1]][2]
+[![Build Status](https://travis-ci.org/zulip/zulip.svg?branch=master)](https://travis-ci.org/zulip/zulip) [![Coverage Status](https://coveralls.io/repos/github/zulip/zulip/badge.svg?branch=master)](https://coveralls.io/github/zulip/zulip?branch=master)
 
-[1]: https://travis-ci.org/zulip/zulip.svg?branch=master
-[2]: https://travis-ci.org/zulip/zulip
+## Community
 
-Installing the Zulip Development environment
-============================================
+There are several places online where folks discuss Zulip.
 
-The Zulip development environment is the recommened option for folks
-interested in trying out Zulip.  This is documented in
-[README.dev.md](README.dev.md).
+One of those places is our [public Zulip instance](https://zulip.tabbott.net/).
+You can go through the simple signup process at that link, and then you
+will soon be talking to core Zulip developers and other users.  To get
+help in real time, you will have the best luck finding core developers
+roughly between 16:00 UTC and 23:59 UTC.  Most questions get answered
+within a day.
 
-Running Zulip in production
-===========================
+We have a [Google mailing list](https://groups.google.com/forum/#!forum/zulip-devel)
+that is currently pretty low traffic.  It is where we do things like
+announce public meetings or major releases.  You can also use it to
+ask questions about features or possible bugs.
+
+Last but not least, we use [GitHub](https://github.com/zulip/zulip) to
+track Zulip-related issues (and store our code, of course).
+Anybody with a Github account should be able to create Issues there
+pertaining to bugs or enhancement requests.  We also use Pull
+Requests as our primary mechanism to receive code contributions.
+
+## Installing the Zulip Development environment
+
+The Zulip development environment is the recommended option for folks
+interested in trying out Zulip.  This is documented in [the developer
+installation guide][dev-install].
+
+## Running Zulip in production
 
 Zulip in production only supports Ubuntu 14.04 right now, but work is
 ongoing on adding support for additional platforms. The installation
-process is documented in https://zulip.org/server.html and in more
-detail in [README.prod.md](README.prod.md).
+process is documented at https://zulip.org/server.html and in more
+detail in [the
+documentation](https://zulip.readthedocs.io/en/latest/prod-install.html).
 
-Contributing to Zulip
-=====================
+## Ways to contribute
 
 Zulip welcomes all forms of contributions!  The page documents the
 Zulip development process.
@@ -45,12 +69,15 @@ please skim our [commit message style guidelines][doc-commit-style].
 * **Testing**. The Zulip automated tests all run automatically when
 you submit a pull request, but you can also run them all in your
 development environment following the instructions in the [testing
-docs][doc-test].
+docs][doc-test]. You can also try out [our new desktop
+client][electron], which is in alpha; we'd appreciate testing and
+[feedback](https://github.com/zulip/zulip-electron/issues/new).
 
 * **Developer Documentation**.  Zulip has a growing collection of
 developer documentation on [Read The Docs][doc].  Recommended reading
 for new contributors includes the [directory structure][doc-dirstruct]
-and [new feature tutorial][doc-newfeat].
+and [new feature tutorial][doc-newfeat]. You can also improve
+[Zulip.org][z-org].
 
 * **Mailing lists and bug tracker**. Zulip has a [development
 discussion mailing list][gg-devel] and uses [GitHub issues
@@ -61,31 +88,49 @@ relevant list!  Please report any security issues you discover to
 zulip-security@googlegroups.com.
 
 * **App codebases**. This repository is for the Zulip server and web
-app; the [desktop][], [Android][], and [iOS][] apps are separate
-repositories.
+app (including most integrations); the [desktop][], [Android][], and
+[iOS][] apps, are separate repositories, as are our [experimental
+React Native iOS app][ios-exp] and [alpha Electron desktop
+app][electron].
+
+* **Glue code**. We maintain a [Hubot adapter][hubot-adapter] and several
+integrations ([Phabricator][phab], [Jenkins][], [Puppet][], [Redmine][],
+and [Trello][]), plus [node.js API bindings][node], and a [full-text search
+PostgreSQL extension][tsearch], as separate repos.
 
 * **Translations**.  Zulip is in the process of being translated into
 10+ languages, and we love contributions to our translations.  See our
-[translating documentation](transifex) if you're interested in
+[translating documentation][transifex] if you're interested in
 contributing!
 
 [cla]: https://opensource.dropbox.com/cla/
+[dev-install]: https://zulip.readthedocs.io/en/latest/dev-overview.html
 [doc]: https://zulip.readthedocs.io/
 [doc-commit-style]: http://zulip.readthedocs.io/en/latest/code-style.html#commit-messages
 [doc-dirstruct]: http://zulip.readthedocs.io/en/latest/directory-structure.html
 [doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
-[doc-test]: https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+[doc-test]: http://zulip.readthedocs.io/en/latest/testing.html
+[electron]: https://github.com/zulip/zulip-electron
 [gg-devel]: https://groups.google.com/forum/#!forum/zulip-devel
 [gh-issues]: https://github.com/zulip/zulip/issues
 [desktop]: https://github.com/zulip/zulip-desktop
 [android]: https://github.com/zulip/zulip-android
 [ios]: https://github.com/zulip/zulip-ios
+[ios-exp]: https://github.com/zulip/zulip-mobile
 [email-android]: https://groups.google.com/forum/#!forum/zulip-android
 [email-ios]: https://groups.google.com/forum/#!forum/zulip-ios
-[transifex]: https://www.transifex.com/zulip/zulip/
+[hubot-adapter]: https://github.com/zulip/hubot-zulip
+[jenkins]: https://github.com/zulip/zulip-jenkins-plugin
+[node]: https://github.com/zulip/zulip-node
+[phab]: https://github.com/zulip/phabricator-to-zulip
+[puppet]: https://github.com/matthewbarr/puppet-zulip
+[redmine]: https://github.com/zulip/zulip-redmine-plugin
+[trello]: https://github.com/zulip/trello-to-zulip
+[tsearch]: https://github.com/zulip/tsearch_extras
+[transifex]: https://zulip.readthedocs.io/en/latest/translating.html#testing-translations
+[z-org]: https://github.com/zulip/zulip.github.io
 
-How to get involved with contributing to Zulip
-==============================================
+## How to get involved with contributing to Zulip
 
 First, subscribe to the Zulip [development discussion mailing
 list][gg-devel].
@@ -159,8 +204,7 @@ Feedback on how to make this development process more efficient, fun,
 and friendly to new contributors is very welcome!  Just send an email
 to the Zulip Developers list with your thoughts.
 
-License
-=======
+## License
 
 Copyright 2011-2015 Dropbox, Inc.
 

THIRDPARTY

@@ -42,13 +42,6 @@ Files: puppet/apt/*
 Copyright: 2011, Evolving Web Inc.
 License: Expat
 
-Files: puppet/common/*
-Copyright: 2007, David Schmitt
-License: BSD-3-Clause
-Comment: https://github.com/DavidS/puppet-common
- Distribution includes a file `lib/puppet/parser/functions/ip_to_cron.rb` which
- we removed due to unclear license
-
 Files: puppet/stdlib/*
 Copyright: 2011, Krzysztof Wilczynski
  2011, Puppet Labs Inc
@@ -122,7 +115,7 @@ Copyright: 2013 Nijiko Yonskai
 License: Apache-2.0
 Comment: The software has been modified.
 
-Files: static/third/gemoji/images/emoji/unicode/* tools/emoji_dump/*.ttf
+Files: static/third/gemoji/images/emoji/unicode/* tools/setup/emoji_dump/*.ttf
 Copyright: Google, Inc.
 License: Apache-2.0
 Comment: These are actually Noto Emoji, not gemoji.
@@ -142,7 +135,7 @@ Copyright: 2013 Jack Moore
 License: Expat
 
 Files: static/third/jquery-caret/*
-Copyright: 2010 C.F., Wong
+Copyright: 2012, 2013 Andrew C. Dvorak
 License: Expat
 
 Files: static/third/jquery-filedrop/jquery.filedrop.js
@@ -182,6 +175,10 @@ Files: static/third/marked/*
 Copyright: 2011-2013, Christopher Jeffrey
 License: Expat
 
+Files: static/third/string-prototype-codepointat/*
+Copyright: 2014 Mathias Bynens
+License: Expat
+
 Files: static/third/sockjs/sockjs-0.3.4.js
 Copyright: 2011-2012 VMware, Inc.
  2012 Douglas Crockford
@@ -204,7 +201,7 @@ Copyright: 2011-2013 Felix Gnass
 License: Expat
 
 Files: static/third/underscore/underscore.js
-Copyright: 2009-2013 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
+Copyright: 2009-2015 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
 License: Expat
 Comment: https://github.com/jashkenas/underscore/blob/master/LICENSE
 

Vagrantfile

@@ -3,7 +3,7 @@
 VAGRANTFILE_API_VERSION = "2"
 
 def command?(name)
-  `which #{name}`
+  `which #{name} > /dev/null 2>&1`
   $?.success?
 end
 
@@ -13,16 +13,15 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.box = "fgrehm/trusty64-lxc"
 
   # The Zulip development environment runs on 9991 on the guest.
-  config.vm.network "forwarded_port", guest: 9991, host: 9991, host_ip: "127.0.0.1"
+  host_port = 9991
+  http_proxy = https_proxy = no_proxy = ""
 
   config.vm.synced_folder ".", "/vagrant", disabled: true
   config.vm.synced_folder ".", "/srv/zulip"
 
-  proxy_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
-  if File.file?(proxy_config_file)
-    http_proxy = https_proxy = no_proxy = ""
-
-    IO.foreach(proxy_config_file) do |line|
+  vagrant_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
+  if File.file?(vagrant_config_file)
+    IO.foreach(vagrant_config_file) do |line|
       line.chomp!
       key, value = line.split(nil, 2)
       case key
@@ -30,19 +29,22 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
       when "HTTP_PROXY"; http_proxy = value
       when "HTTPS_PROXY"; https_proxy = value
       when "NO_PROXY"; no_proxy = value
+      when "HOST_PORT"; host_port = value.to_i
       end
     end
+  end
 
-    if Vagrant.has_plugin?("vagrant-proxyconf")
-      if http_proxy != ""
-        config.proxy.http = http_proxy
-      end
-      if https_proxy != ""
-        config.proxy.https = https_proxy
-      end
-      if https_proxy != ""
-        config.proxy.no_proxy = no_proxy
-      end
+  config.vm.network "forwarded_port", guest: 9991, host: host_port, host_ip: "127.0.0.1"
+
+  if Vagrant.has_plugin?("vagrant-proxyconf")
+    if http_proxy != ""
+      config.proxy.http = http_proxy
+    end
+    if https_proxy != ""
+      config.proxy.https = https_proxy
+    end
+    if https_proxy != ""
+      config.proxy.no_proxy = no_proxy
     end
   end
 
@@ -63,14 +65,16 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
   config.vm.provider "virtualbox" do |vb, override|
     override.vm.box = "ubuntu/trusty64"
-    # 2GiB seemed reasonable here. The VM OOMs with only 1024MiB.
+    # It's possible we can get away with just 1.5GB; more testing needed
     vb.memory = 2048
   end
 
 $provision_script = <<SCRIPT
 set -x
 set -e
-/usr/bin/python /srv/zulip/provision.py
+set -o pipefail
+ln -nsf /srv/zulip ~/zulip
+/usr/bin/python /srv/zulip/tools/provision.py | sudo tee -a /var/log/zulip_provision.log
 SCRIPT
 
   config.vm.provision "shell",

analytics/management/commands/active_user_stats.py

@@ -2,6 +2,7 @@ from __future__ import absolute_import
 from __future__ import print_function
 
 from django.core.management.base import BaseCommand
+from typing import Any
 
 from zerver.models import UserPresence, UserActivity
 from zerver.lib.utils import statsd, statsd_key
@@ -15,6 +16,7 @@ class Command(BaseCommand):
     Run as a cron job that runs every 10 minutes."""
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         # Get list of all active users in the last 1 week
         cutoff = datetime.now() - timedelta(minutes=30, hours=168)
 
@@ -31,7 +33,7 @@ class Command(BaseCommand):
                 known_active = last_presence.timestamp
 
             for bucket in hour_buckets:
-                if not bucket in user_info[last_presence.user_profile.realm.domain]:
+                if bucket not in user_info[last_presence.user_profile.realm.domain]:
                     user_info[last_presence.user_profile.realm.domain][bucket] = []
                 if datetime.now(known_active.tzinfo) - known_active < timedelta(hours=bucket):
                     user_info[last_presence.user_profile.realm.domain][bucket].append(last_presence.user_profile.email)
@@ -40,14 +42,14 @@ class Command(BaseCommand):
             print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.active.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.active.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))
 
         # Also do stats for how many users have been reading the app.
         users_reading = UserActivity.objects.select_related().filter(query="/json/messages/flags")
         user_info = defaultdict(dict)
         for activity in users_reading:
             for bucket in hour_buckets:
-                if not bucket in user_info[activity.user_profile.realm.domain]:
+                if bucket not in user_info[activity.user_profile.realm.domain]:
                     user_info[activity.user_profile.realm.domain][bucket] = []
                 if datetime.now(activity.last_visit.tzinfo) - activity.last_visit < timedelta(hours=bucket):
                     user_info[activity.user_profile.realm.domain][bucket].append(activity.user_profile.email)
@@ -55,4 +57,4 @@ class Command(BaseCommand):
             print("Realm %s" % (realm,))
             for hr, users in sorted(buckets.items()):
                 print("\tUsers reading for %s: %s" % (hr, len(users)))
-                statsd.gauge("users.reading.%s.%shr" %  (statsd_key(realm, True), statsd_key(hr, True)), len(users))
+                statsd.gauge("users.reading.%s.%shr" % (statsd_key(realm, True), statsd_key(hr, True)), len(users))

analytics/management/commands/active_user_stats_by_day.py

@@ -5,6 +5,7 @@ import datetime
 import pytz
 
 from optparse import make_option
+from typing import Any
 from django.core.management.base import BaseCommand
 from zerver.lib.statistics import activity_averages_during_day
 
@@ -16,6 +17,7 @@ class Command(BaseCommand):
                      help="Day to query in format 2013-12-05.  Default is yesterday"),)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options["date"] is None:
             date = datetime.datetime.now() - datetime.timedelta(days=1)
         else:

analytics/management/commands/analyze_mit.py

@@ -1,6 +1,8 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
 from optparse import make_option
 from django.core.management.base import BaseCommand
 from zerver.models import Recipient, Message
@@ -10,6 +12,7 @@ import time
 import logging
 
 def compute_stats(log_level):
+    # type: (int) -> None
     logger = logging.getLogger()
     logger.setLevel(log_level)
 
@@ -76,6 +79,7 @@ class Command(BaseCommand):
     help = "Compute statistics on MIT Zephyr usage."
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         level = logging.INFO
         if options["verbose"]:
             level = logging.DEBUG

analytics/management/commands/analyze_user_activity.py

@@ -2,6 +2,8 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+from typing import Any, Dict
+
 from zerver.lib.statistics import seconds_usage_between
 
 from optparse import make_option
@@ -11,6 +13,7 @@ import datetime
 from django.utils.timezone import utc
 
 def analyze_activity(options):
+    # type: (Dict[str, Any]) -> None
     day_start = datetime.datetime.strptime(options["date"], "%Y-%m-%d").replace(tzinfo=utc)
     day_end = day_start + datetime.timedelta(days=options["duration"])
 
@@ -27,7 +30,7 @@ def analyze_activity(options):
             continue
 
         total_duration += duration
-        print("%-*s%s" % (37, user_profile.email, duration, ))
+        print("%-*s%s" % (37, user_profile.email, duration,))
 
     print("\nTotal Duration:                      %s" % (total_duration,))
     print("\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,))
@@ -44,7 +47,7 @@ It will correctly not count server-initiated reloads in the activity statistics.
 
 The duration flag can be used to control how many days to show usage duration for
 
-Usage: python2.7 manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
+Usage: python manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
 
 By default, if no date is selected 2013-09-10 is used. If no realm is provided, information
 is shown for all realms"""
@@ -52,8 +55,10 @@ is shown for all realms"""
     option_list = BaseCommand.option_list + (
         make_option('--realm', action='store'),
         make_option('--date', action='store', default="2013-09-06"),
-        make_option('--duration', action='store', default=1, type=int, help="How many days to show usage information for"),
+        make_option('--duration', action='store', default=1, type=int,
+                    help="How many days to show usage information for"),
         )
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         analyze_activity(options)

analytics/management/commands/client_activity.py

@@ -1,8 +1,11 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
-from django.db.models import Count
+from django.db.models import Count, QuerySet
 
 from zerver.models import UserActivity, UserProfile, Realm, \
     get_realm, get_user_profile_by_email
@@ -14,15 +17,17 @@ class Command(BaseCommand):
 
 Usage examples:
 
-python2.7 manage.py client_activity
-python2.7 manage.py client_activity zulip.com
-python2.7 manage.py client_activity jesstess@zulip.com"""
+python manage.py client_activity
+python manage.py client_activity zulip.com
+python manage.py client_activity jesstess@zulip.com"""
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('arg', metavar='<arg>', type=str, nargs='?', default=None,
                             help="realm or user to estimate client activity for")
 
     def compute_activity(self, user_activity_objects):
+        # type: (QuerySet) -> None
         # Report data from the past week.
         #
         # This is a rough report of client activity because we inconsistently
@@ -54,6 +59,7 @@ python2.7 manage.py client_activity jesstess@zulip.com"""
 
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['arg'] is None:
             # Report global activity.
             self.compute_activity(UserActivity.objects.all())

analytics/management/commands/realm_stats.py

@@ -2,6 +2,9 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 import datetime
 import pytz
 
@@ -19,10 +22,12 @@ class Command(BaseCommand):
     help = "Generate statistics on realm activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def active_users(self, realm):
+        # type: (Realm) -> List[UserProfile]
         # Has been active (on the website, for now) in the last 7 days.
         activity_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=7)
         return [activity.user_profile for activity in \
@@ -33,36 +38,44 @@ class Command(BaseCommand):
                                                 client__name="website")]
 
     def messages_sent_by(self, user, days_ago):
+        # type: (UserProfile, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender=user, pub_date__gt=sent_time_cutoff).count()
 
     def total_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return Message.objects.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def human_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).count()
 
     def api_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         return (self.total_messages(realm, days_ago) - self.human_messages(realm, days_ago))
 
     def stream_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff,
                                      recipient__type=Recipient.STREAM).count()
 
     def private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.HUDDLE).count()
 
     def group_private_messages(self, realm, days_ago):
+        # type: (Realm, int) -> int
         sent_time_cutoff = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=days_ago)
         return human_messages.filter(sender__realm=realm, pub_date__gt=sent_time_cutoff).exclude(
             recipient__type=Recipient.STREAM).exclude(recipient__type=Recipient.PERSONAL).count()
 
     def report_percentage(self, numerator, denominator, text):
+        # type: (float, float, str) -> None
         if not denominator:
             fraction = 0.0
         else:
@@ -70,6 +83,7 @@ class Command(BaseCommand):
         print("%.2f%% of" % (fraction * 100,), text)
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/management/commands/stream_stats.py

@@ -1,6 +1,9 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from typing import Any
+
+from argparse import ArgumentParser
 from django.core.management.base import BaseCommand
 from django.db.models import Q
 from zerver.models import Realm, Stream, Message, Subscription, Recipient, get_realm
@@ -9,10 +12,12 @@ class Command(BaseCommand):
     help = "Generate statistics on the streams for a realm."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def handle(self, *args, **options):
+        # type: (*Any, **str) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/management/commands/user_stats.py

@@ -1,8 +1,10 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
+from argparse import ArgumentParser
 import datetime
 import pytz
+from typing import Any
 
 from django.core.management.base import BaseCommand
 from zerver.models import UserProfile, Realm, Stream, Message, get_realm
@@ -12,15 +14,18 @@ class Command(BaseCommand):
     help = "Generate statistics on user activity."
 
     def add_arguments(self, parser):
+        # type: (ArgumentParser) -> None
         parser.add_argument('realms', metavar='<realm>', type=str, nargs='*',
                             help="realm to generate statistics for")
 
     def messages_sent_by(self, user, week):
+        # type: (UserProfile, int) -> int
         start = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=(week + 1)*7)
         end = datetime.datetime.now(tz=pytz.utc) - datetime.timedelta(days=week*7)
         return Message.objects.filter(sender=user, pub_date__gt=start, pub_date__lte=end).count()
 
     def handle(self, *args, **options):
+        # type: (*Any, **Any) -> None
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]

analytics/urls.py

@@ -1,8 +1,9 @@
 from django.conf.urls import patterns, url
 
-urlpatterns = patterns('analytics.views',
-    url(r'^activity$', 'get_activity'),
-    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'get_realm_activity'),
-    url(r'^user_activity/(?P<email>[\S]+)/$', 'get_user_activity'),
-)
+i18n_urlpatterns = [
+    url(r'^activity$', 'analytics.views.get_activity'),
+    url(r'^realm_activity/(?P<realm>[\S]+)/$', 'analytics.views.get_realm_activity'),
+    url(r'^user_activity/(?P<email>[\S]+)/$', 'analytics.views.get_user_activity'),
+]
 
+urlpatterns = patterns('', *i18n_urlpatterns)

analytics/views.py

@@ -1,13 +1,14 @@
 from __future__ import absolute_import
 from __future__ import division
-from typing import Any, Dict, List, Tuple
+from six import text_type
+from typing import Any, Dict, List, Tuple, Optional, Sequence, Callable, Union
 
 from django.db import connection
+from django.db.models.query import QuerySet
 from django.template import RequestContext, loader
-from django.utils.html import mark_safe
-from django.shortcuts import render_to_response
 from django.core import urlresolvers
-from django.http import HttpResponseNotFound
+from django.http import HttpResponseNotFound, HttpRequest, HttpResponse
+from jinja2 import Markup as mark_safe
 
 from zerver.decorator import has_request_variables, REQ, zulip_internal
 from zerver.models import get_realm, UserActivity, UserActivityInterval, Realm
@@ -25,10 +26,14 @@ from six.moves import range
 from six.moves import zip
 eastern_tz = pytz.timezone('US/Eastern')
 
+from zproject.jinja2 import render_to_response
+
 def make_table(title, cols, rows, has_row_class=False):
+    # type: (str, List[str], List[Any], bool) -> str
 
     if not has_row_class:
         def fix_row(row):
+            # type: (Any) -> Dict[str, Any]
             return dict(cells=row, row_class=None)
         rows = list(map(fix_row, rows))
 
@@ -42,6 +47,7 @@ def make_table(title, cols, rows, has_row_class=False):
     return content
 
 def dictfetchall(cursor):
+    # type: (connection.cursor) -> List[Dict[str, Any]]
     "Returns all rows from a cursor as a dict"
     desc = cursor.description
     return [
@@ -51,6 +57,7 @@ def dictfetchall(cursor):
 
 
 def get_realm_day_counts():
+    # type: () -> Dict[str, Dict[str, str]]
     query = '''
         select
             r.domain,
@@ -85,11 +92,12 @@ def get_realm_day_counts():
 
     result = {}
     for domain in counts:
-        cnts = [counts[domain].get(age, 0) for age in range(8)]
-        min_cnt = min(cnts)
-        max_cnt = max(cnts)
+        raw_cnts = [counts[domain].get(age, 0) for age in range(8)]
+        min_cnt = min(raw_cnts)
+        max_cnt = max(raw_cnts)
 
         def format_count(cnt):
+            # type: (int) -> str
             if cnt == min_cnt:
                 good_bad = 'bad'
             elif cnt == max_cnt:
@@ -99,12 +107,13 @@ def get_realm_day_counts():
 
             return '<td class="number %s">%s</td>' % (good_bad, cnt)
 
-        cnts = ''.join(map(format_count, cnts))
+        cnts = ''.join(map(format_count, raw_cnts))
         result[domain] = dict(cnts=cnts)
 
     return result
 
 def realm_summary_table(realm_minutes):
+    # type: (Dict[str, float]) -> str
     query = '''
         SELECT
             realm.domain,
@@ -217,10 +226,10 @@ def realm_summary_table(realm_minutes):
             row['history'] = ''
 
     # augment data with realm_minutes
-    total_hours = 0
+    total_hours = 0.0
     for row in rows:
         domain = row['domain']
-        minutes = realm_minutes.get(domain, 0)
+        minutes = realm_minutes.get(domain, 0.0)
         hours = minutes / 60.0
         total_hours += hours
         row['hours'] = str(int(hours))
@@ -235,6 +244,7 @@ def realm_summary_table(realm_minutes):
 
     # Count active sites
     def meets_goal(row):
+        # type: (Dict[str, int]) -> bool
         return row['active_user_count'] >= 5
 
     num_active_sites = len(list(filter(meets_goal, rows)))
@@ -243,10 +253,12 @@ def realm_summary_table(realm_minutes):
     total_active_user_count = 0
     total_user_profile_count = 0
     total_bot_count = 0
+    total_at_risk_count = 0
     for row in rows:
         total_active_user_count += int(row['active_user_count'])
         total_user_profile_count += int(row['user_profile_count'])
         total_bot_count += int(row['bot_count'])
+        total_at_risk_count += int(row['at_risk_count'])
 
 
     rows.append(dict(
@@ -254,7 +266,8 @@ def realm_summary_table(realm_minutes):
         active_user_count=total_active_user_count,
         user_profile_count=total_user_profile_count,
         bot_count=total_bot_count,
-        hours=int(total_hours)
+        hours=int(total_hours),
+        at_risk_count=total_at_risk_count,
     ))
 
     content = loader.render_to_string(
@@ -265,6 +278,7 @@ def realm_summary_table(realm_minutes):
 
 
 def user_activity_intervals():
+    # type: () -> Tuple[mark_safe, Dict[str, float]]
     day_end = timestamp_to_datetime(time.time())
     day_start = day_end - timedelta(hours=24)
 
@@ -304,7 +318,7 @@ def user_activity_intervals():
 
             total_duration += duration
             realm_duration += duration
-            output += "  %-*s%s\n" % (37, email, duration, )
+            output += "  %-*s%s\n" % (37, email, duration)
 
         realm_minutes[domain] = realm_duration.total_seconds() / 60
 
@@ -315,6 +329,7 @@ def user_activity_intervals():
     return content, realm_minutes
 
 def sent_messages_report(realm):
+    # type: (str) -> str
     title = 'Recently sent messages for ' + realm
 
     cols = [
@@ -382,7 +397,9 @@ def sent_messages_report(realm):
     return make_table(title, cols, rows)
 
 def ad_hoc_queries():
+    # type: () -> List[Dict[str, str]]
     def get_page(query, cols, title):
+        # type: (str, List[str], str) -> Dict[str, str]
         cursor = connection.cursor()
         cursor.execute(query)
         rows = cursor.fetchall()
@@ -390,6 +407,7 @@ def ad_hoc_queries():
         cursor.close()
 
         def fix_rows(i, fixup_func):
+            # type: (int, Union[Callable[[Realm], mark_safe], Callable[[datetime], str]]) -> None
             for row in rows:
                 row[i] = fixup_func(row[i])
 
@@ -552,8 +570,9 @@ def ad_hoc_queries():
 @zulip_internal
 @has_request_variables
 def get_activity(request):
-    duration_content, realm_minutes = user_activity_intervals()
-    counts_content = realm_summary_table(realm_minutes)
+    # type: (HttpRequest) -> HttpResponse
+    duration_content, realm_minutes = user_activity_intervals() # type: Tuple[mark_safe, Dict[str, float]]
+    counts_content = realm_summary_table(realm_minutes) # type: str
     data = [
         ('Counts', counts_content),
         ('Durations', duration_content),
@@ -566,10 +585,11 @@ def get_activity(request):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title, is_home=True),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 def get_user_activity_records_for_realm(realm, is_bot):
+    # type: (str, bool) -> QuerySet
     fields = [
         'user_profile__full_name',
         'user_profile__email',
@@ -589,6 +609,7 @@ def get_user_activity_records_for_realm(realm, is_bot):
     return records
 
 def get_user_activity_records_for_email(email):
+    # type: (str) -> List[QuerySet]
     fields = [
         'user_profile__full_name',
         'query',
@@ -605,6 +626,7 @@ def get_user_activity_records_for_email(email):
     return records
 
 def raw_user_activity_table(records):
+    # type: (List[QuerySet]) -> str
     cols = [
         'query',
         'client',
@@ -613,6 +635,7 @@ def raw_user_activity_table(records):
     ]
 
     def row(record):
+        # type: (QuerySet) -> List[Any]
         return [
                 record.query,
                 record.client.name,
@@ -625,9 +648,15 @@ def raw_user_activity_table(records):
     return make_table(title, cols, rows)
 
 def get_user_activity_summary(records):
-    # type: (Any) -> Any
+    # type: (List[QuerySet]) -> Dict[str, Dict[str, Any]]
+    #: `Any` used above should be `Union(int, datetime)`.
+    #: However current version of `Union` does not work inside other function.
+    #: We could use something like:
+    # `Union[Dict[str, Dict[str, int]], Dict[str, Dict[str, datetime]]]`
+    #: but that would require this long `Union` to carry on throughout inner functions.
     summary = {} # type: Dict[str, Dict[str, Any]]
     def update(action, record):
+        # type: (str, QuerySet) -> None
         if action not in summary:
             summary[action] = dict(
                     count=record.count,
@@ -669,24 +698,28 @@ def get_user_activity_summary(records):
     return summary
 
 def format_date_for_activity_reports(date):
+    # type: (Optional[datetime]) -> str
     if date:
         return date.astimezone(eastern_tz).strftime('%Y-%m-%d %H:%M')
     else:
         return ''
 
 def user_activity_link(email):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_user_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(email=email))
     email_link = '<a href="%s">%s</a>' % (url, email)
     return mark_safe(email_link)
 
 def realm_activity_link(realm):
+    # type: (str) -> mark_safe
     url_name = 'analytics.views.get_realm_activity'
     url = urlresolvers.reverse(url_name, kwargs=dict(realm=realm))
     realm_link = '<a href="%s">%s</a>' % (url, realm)
     return mark_safe(realm_link)
 
 def realm_client_table(user_summaries):
+    # type: (Dict[str, Dict[str, Dict[str, Any]]]) -> str
     exclude_keys = [
             'internal',
             'name',
@@ -731,6 +764,7 @@ def realm_client_table(user_summaries):
     return make_table(title, cols, rows)
 
 def user_activity_summary_table(user_summary):
+    # type: (Dict[str, Dict[str, Any]]) -> str
     rows = []
     for k, v in user_summary.items():
         if k == 'name':
@@ -757,28 +791,33 @@ def user_activity_summary_table(user_summary):
     return make_table(title, cols, rows)
 
 def realm_user_summary_table(all_records, admin_emails):
+    # type: (List[QuerySet], Set[text_type]) -> Tuple[Dict[str, Dict[str, Any]], str]
     user_records = {}
 
     def by_email(record):
+        # type: (QuerySet) -> str
         return record.user_profile.email
 
     for email, records in itertools.groupby(all_records, by_email):
         user_records[email] = get_user_activity_summary(list(records))
 
     def get_last_visit(user_summary, k):
+        # type: (Dict[str, Dict[str, datetime]], str) -> Optional[datetime]
         if k in user_summary:
             return user_summary[k]['last_visit']
         else:
             return None
 
     def get_count(user_summary, k):
+        # type: (Dict[str, Dict[str, str]], str) -> str
         if k in user_summary:
             return user_summary[k]['count']
         else:
             return ''
 
     def is_recent(val):
-        age = datetime.now(val.tzinfo) - val
+        # type: (Optional[datetime]) -> bool
+        age = datetime.now(val.tzinfo) - val # type: ignore # datetie.now tzinfo bug.
         return age.total_seconds() < 5 * 60
 
     rows = []
@@ -788,18 +827,19 @@ def realm_user_summary_table(all_records, admin_emails):
         cells = [user_summary['name'], email_link, sent_count]
         row_class = ''
         for field in ['use', 'send', 'pointer', 'desktop', 'ZulipiOS', 'Android']:
-            val = get_last_visit(user_summary, field)
+            visit = get_last_visit(user_summary, field)
             if field == 'use':
-                if val and is_recent(val):
+                if visit and is_recent(visit):
                     row_class += ' recently_active'
                 if email in admin_emails:
                     row_class += ' admin'
-            val = format_date_for_activity_reports(val)
+            val = format_date_for_activity_reports(visit)
             cells.append(val)
         row = dict(cells=cells, row_class=row_class)
         rows.append(row)
 
     def by_used_time(row):
+        # type: (Dict[str, Sequence[str]]) -> str
         return row['cells'][3]
 
     rows = sorted(rows, key=by_used_time, reverse=True)
@@ -823,7 +863,7 @@ def realm_user_summary_table(all_records, admin_emails):
 
 @zulip_internal
 def get_realm_activity(request, realm):
-    # type: (Any, Any) -> Any
+    # type: (HttpRequest, str) -> HttpResponse
     data = [] # type: List[Tuple[str, str]]
     all_user_records = {} # type: Dict[str, Any]
 
@@ -860,11 +900,12 @@ def get_realm_activity(request, realm):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, realm_link=realm_link, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )
 
 @zulip_internal
 def get_user_activity(request, email):
+    # type: (HttpRequest, str) -> HttpResponse
     records = get_user_activity_records_for_email(email)
 
     data = [] # type: List[Tuple[str, str]]
@@ -880,5 +921,5 @@ def get_user_activity(request, email):
     return render_to_response(
         'analytics/activity.html',
         dict(data=data, title=title),
-        context_instance=RequestContext(request)
+        request=request
     )

api/bin/zulip-send

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 # zulip-send -- Sends a message to the specified recipients.
 

api/examples/create-user

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012-2014 Zulip, Inc.

api/examples/edit-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/get-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/list-members

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2014 Zulip, Inc.

api/examples/list-subscriptions

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-events

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-messages

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-next-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/recent-messages

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/send-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/subscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/unsubscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/integrations/asana/zulip_asana_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/asana/zulip_asana_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Asana integration for Zulip

api/integrations/basecamp/zulip_basecamp_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/basecamp/zulip_basecamp_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Basecamp activity

api/integrations/codebase/zulip_codebase_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/codebase/zulip_codebase_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Codebase HQ activity
@@ -135,7 +135,7 @@ def handle_event(event):
             content = "%s deleted branch %s from %s" % (actor_name, branch, project)
         else:
             if new_ref:
-                branch = "new branch %s" % (branch, )
+                branch = "new branch %s" % (branch,)
             content = "%s pushed %s commit(s) to %s in project %s:\n\n" % \
                         (actor_name, num_commits, branch, project)
             for commit in raw_props.get('commits'):
@@ -208,7 +208,8 @@ def handle_event(event):
             subject = "Discussion: %s" % (subj,)
 
             if category:
-                content = "%s started a new discussion in %s:\n\n~~~ quote\n%s\n~~~" % (actor_name, category, comment_content)
+                format_str = "%s started a new discussion in %s:\n\n~~~ quote\n%s\n~~~"
+                content = format_str % (actor_name, category, comment_content)
             else:
                 content = "%s posted:\n\n~~~ quote\n%s\n~~~" % (actor_name, comment_content)
 
@@ -223,7 +224,8 @@ def handle_event(event):
 
         start_ref_url = make_url("projects/%s/repositories/%s/commit/%s" % (project_link, repo_link, start_ref))
         end_ref_url = make_url("projects/%s/repositories/%s/commit/%s" % (project_link, repo_link, end_ref))
-        between_url = make_url("projects/%s/repositories/%s/compare/%s...%s" % (project_link, repo_link, start_ref, end_ref))
+        between_url = make_url("projects/%s/repositories/%s/compare/%s...%s" % (
+            project_link, repo_link, start_ref, end_ref))
 
         subject = "Deployment to %s" % (environment,)
 

api/integrations/git/post-receive

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-receive hook.

api/integrations/git/zulip_git_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/hg/zulip-changegroup.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip hook for Mercurial changeset pushes.
@@ -24,7 +24,7 @@
 #
 #
 # This hook is called when changesets are pushed to the master repository (ie
-# `hg push`). See https://zulip.com/integrations for installation instructions.
+# `hg push`). See https://zulipchat.com/integrations for installation instructions.
 from __future__ import absolute_import
 
 import zulip

api/integrations/nagios/nagios-notify-zulip

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import optparse
 import zulip
 

api/integrations/perforce/git_p4.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # git-p4.py -- A tool for bidirectional operation between a Perforce depot and git.
 #
@@ -33,7 +33,7 @@ import stat
 try:
     from subprocess import CalledProcessError
 except ImportError:
-    # from python2.7:subprocess.py
+    # from python:subprocess.py
     # Exception classes used by this module.
     class CalledProcessError(Exception):
         """This exception is raised when a process run by check_call() returns

api/integrations/perforce/zulip_change-commit.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2012-2014 Zulip, Inc.

api/integrations/perforce/zulip_perforce_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/rss/rss-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # RSS integration for Zulip

api/integrations/svn/post-commit

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-commit hook.

api/integrations/svn/zulip_svn_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/twitter/twitter-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Twitter integration for Zulip
@@ -45,11 +45,12 @@ parser = optparse.OptionParser(r"""
 
     Slurp tweets on your timeline into a specific zulip stream.
 
-    Run this on your personal machine.  Your API key and twitter id are revealed to local
-    users through the command line or config file.
+    Run this on your personal machine.  Your API key and twitter id
+    are revealed to local users through the command line or config
+    file.
 
-    This bot uses OAuth to authenticate with twitter. Please create a ~/.zulip_twitterrc with
-    the following contents:
+    This bot uses OAuth to authenticate with twitter. Please create a
+    ~/.zulip_twitterrc with the following contents:
 
     [twitter]
     consumer_key =
@@ -57,14 +58,16 @@ parser = optparse.OptionParser(r"""
     access_token_key =
     access_token_secret =
 
-    In order to obtain a consumer key & secret, you must register a new application under your twitter account:
+    In order to obtain a consumer key & secret, you must register a
+    new application under your twitter account:
 
     1. Go to http://dev.twitter.com
     2. Log in
     3. In the menu under your username, click My Applications
     4. Create a new application
 
-    Make sure to go the application you created and click "create my access token" as well. Fill in the values displayed.
+    Make sure to go the application you created and click "create my
+    access token" as well.  Fill in the values displayed.
 
     Depends on: twitter-python
 """)

api/integrations/twitter/twitter-search-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
 # Twitter search integration for Zulip

api/setup.py

@@ -1,8 +1,8 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 from __future__ import print_function
-from typing import Any, Generator, List, Tuple
+if False: from typing import Any, Generator, List, Tuple
 
 import os
 import sys
@@ -10,6 +10,7 @@ import sys
 import itertools
 
 def version():
+    # type: () -> str
     version_py = os.path.join(os.path.dirname(__file__), "zulip", "__init__.py")
     with open(version_py) as in_handle:
         version_line = next(itertools.dropwhile(lambda x: not x.startswith("__version__"),
@@ -46,7 +47,7 @@ package_info = dict(
                                            "examples/print-messages", "examples/recent-messages"])] + \
         list(recur_expand('share/zulip', 'integrations/')),
     scripts=["bin/zulip-send"],
-)
+) # type: Dict[str, Any]
 
 setuptools_info = dict(
     install_requires=['requests>=0.12.1',

api/zulip/__init__.py

@@ -38,6 +38,7 @@ from six.moves.configparser import SafeConfigParser
 from six.moves import urllib
 import logging
 import six
+from typing import Any, Dict
 
 
 __version__ = "0.2.5"
@@ -46,7 +47,7 @@ logger = logging.getLogger(__name__)
 
 # Check that we have a recent enough version
 # Older versions don't provide the 'json' attribute on responses.
-assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
+assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1')) # type: ignore # https://github.com/python/mypy/issues/1165 and https://github.com/python/typeshed/pull/206
 # In newer versions, the 'json' attribute is a function, not a property
 requests_json_is_function = callable(requests.Response.json)
 
@@ -133,6 +134,17 @@ def generate_option_group(parser, prefix=''):
                           CA certificates. This will be used to
                           verify the server's identity. All
                           certificates should be PEM encoded.''')
+    group.add_option('--client-cert',
+                     action='store',
+                     dest='client_cert',
+                     help='''Specify a file containing a client
+                          certificate (not needed for most deployments).''')
+    group.add_option('--client-cert-key',
+                     action='store',
+                     dest='client_cert_key',
+                     help='''Specify a file containing the client
+                          certificate's key (if it is in a separate
+                          file).''')
     return group
 
 def init_from_options(options, client=None):
@@ -143,20 +155,24 @@ def init_from_options(options, client=None):
     return Client(email=options.zulip_email, api_key=options.zulip_api_key,
                   config_file=options.zulip_config_file, verbose=options.verbose,
                   site=options.zulip_site, client=client,
-                  cert_bundle=options.cert_bundle, insecure=options.insecure)
+                  cert_bundle=options.cert_bundle, insecure=options.insecure,
+                  client_cert=options.client_cert,
+                  client_cert_key=options.client_cert_key)
 
 def get_default_config_filename():
     config_file = os.path.join(os.environ["HOME"], ".zuliprc")
     if (not os.path.exists(config_file) and
         os.path.exists(os.path.join(os.environ["HOME"], ".humbugrc"))):
-        raise RuntimeError("The Zulip API configuration file is now ~/.zuliprc; please run:\n\n  mv ~/.humbugrc ~/.zuliprc\n")
+        raise RuntimeError("The Zulip API configuration file is now ~/.zuliprc; please run:\n\n"
+                           "  mv ~/.humbugrc ~/.zuliprc\n")
     return config_file
 
 class Client(object):
     def __init__(self, email=None, api_key=None, config_file=None,
                  verbose=False, retry_on_errors=True,
                  site=None, client=None,
-                 cert_bundle=None, insecure=None):
+                 cert_bundle=None, insecure=None,
+                 client_cert=None, client_cert_key=None):
         if client is None:
             client = _default_client()
 
@@ -172,6 +188,10 @@ class Client(object):
                 email = config.get("api", "email")
             if site is None and config.has_option("api", "site"):
                 site = config.get("api", "site")
+            if client_cert is None and config.has_option("api", "client_cert"):
+                client_cert = config.get("api", "client_cert")
+            if client_cert_key is None and config.has_option("api", "client_cert_key"):
+                client_cert_key = config.get("api", "client_cert_key")
             if cert_bundle is None and config.has_option("api", "cert_bundle"):
                 cert_bundle = config.get("api", "cert_bundle")
             if insecure is None and config.has_option("api", "insecure"):
@@ -218,6 +238,21 @@ class Client(object):
             # Default behavior: verify against system CA certificates
             self.tls_verification=True
 
+        if client_cert is None:
+            if client_cert_key is not None:
+                raise RuntimeError("client cert key '%s' specified, but no client cert public part provided"
+                                   %(client_cert_key,))
+        else: # we have a client cert
+            if not os.path.isfile(client_cert):
+                raise RuntimeError("client cert '%s' does not exist"
+                                   %(client_cert,))
+            if client_cert_key is not None:
+                if not os.path.isfile(client_cert_key):
+                    raise RuntimeError("client cert key '%s' does not exist"
+                                       %(client_cert_key,))
+        self.client_cert = client_cert
+        self.client_cert_key = client_cert_key
+
     def get_user_agent(self):
         vendor = ''
         vendor_version = ''
@@ -246,16 +281,16 @@ class Client(object):
         request = {}
 
         for (key, val) in six.iteritems(orig_request):
-            if not (isinstance(val, str) or isinstance(val, six.text_type)):
-                request[key] = simplejson.dumps(val)
-            else:
+            if isinstance(val, str) or isinstance(val, six.text_type):
                 request[key] = val
+            else:
+                request[key] = simplejson.dumps(val)
 
         query_state = {
             'had_error_retry': False,
             'request': request,
             'failures': 0,
-        }
+        } # type: Dict[str, Any]
 
         def error_retry(error_string):
             if not self.retry_on_errors or query_state["failures"] >= 10:
@@ -287,12 +322,21 @@ class Client(object):
                 else:
                     kwarg = "data"
                 kwargs = {kwarg: query_state["request"]}
+
+                # Build a client cert object for requests
+                if self.client_cert_key is not None:
+                    client_cert = (self.client_cert, self.client_cert_key)
+                else:
+                    client_cert = self.client_cert
+
                 res = requests.request(
                         method,
                         urllib.parse.urljoin(self.base_url, url),
                         auth=requests.auth.HTTPBasicAuth(self.email,
                                                          self.api_key),
-                        verify=self.tls_verification, timeout=90,
+                        verify=self.tls_verification,
+                        cert=client_cert,
+                        timeout=90,
                         headers={"User-agent": self.get_user_agent()},
                         **kwargs)
 
@@ -343,10 +387,15 @@ class Client(object):
                     "status_code": res.status_code}
 
     @classmethod
-    def _register(cls, name, url=None, make_request=(lambda request={}: request),
+    def _register(cls, name, url=None, make_request=None,
                   method="POST", computed_url=None, **query_kwargs):
         if url is None:
             url = name
+        if make_request is None:
+            def make_request(request=None):
+                if request is None:
+                    request = {}
+                return request
         def call(self, *args, **kwargs):
             request = make_request(*args, **kwargs)
             if computed_url is not None:
@@ -357,7 +406,9 @@ class Client(object):
         call.__name__ = name
         setattr(cls, name, call)
 
-    def call_on_each_event(self, callback, event_types=None, narrow=[]):
+    def call_on_each_event(self, callback, event_types=None, narrow=None):
+        if narrow is None:
+            narrow = []
         def do_register():
             while True:
                 if event_types is None:
@@ -425,9 +476,11 @@ def _mk_rm_subs(streams):
 def _mk_deregister(queue_id):
     return {'queue_id': queue_id}
 
-def _mk_events(event_types=None, narrow=[]):
+def _mk_events(event_types=None, narrow=None):
     if event_types is None:
         return dict()
+    if narrow is None:
+        narrow = []
     return dict(event_types=event_types, narrow=narrow)
 
 def _kwargs_to_dict(**kwargs):

assets/favicon/generate

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import absolute_import
 import xml.etree.ElementTree as ET
 import subprocess

bin/write-rabbitmq-consumers-state-file

@@ -1,16 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-queue=$1
-
-if [ -z "$queue" ]; then
-    echo "Usage: $0 <queue-name>"
-    exit 2
-fi
-
-ZULIP_DIR=/home/zulip/deployments/current
-STATE_DIR=/var/lib/nagios_state
-STATE_FILE=$STATE_DIR/check-rabbitmq-consumers-$queue
-
-"$ZULIP_DIR/bots/check-rabbitmq-consumers" "--queue=$queue" &> "${STATE_FILE}-tmp";
-mv "${STATE_FILE}-tmp" "$STATE_FILE"

bots/check-mirroring

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import print_function
 from __future__ import absolute_import
 import sys

bots/check-rabbitmq-consumers

@@ -1,56 +0,0 @@
-#!/usr/bin/env python2.7
-
-from __future__ import print_function
-import sys
-import time
-import optparse
-from collections import defaultdict
-import os
-import subprocess
-
-states = {
-    0: "OK",
-    1: "WARNING",
-    2: "CRITICAL",
-    3: "UNKNOWN"
-}
-
-if 'USER' in os.environ and not os.environ['USER'] in ['root', 'rabbitmq']:
-    print("This script must be run as the root or rabbitmq user")
-
-
-usage = """Usage: check-rabbitmq-consumers --queue=[queue-name] --min-threshold=[min-threshold]"""
-
-parser = optparse.OptionParser(usage=usage)
-parser.add_option('--queue',
-                  dest='queue_name',
-                  default="notify_tornado",
-                  action='store')
-parser.add_option('--min-threshold',
-                  dest='min_count',
-                  type="int",
-                  default=1,
-                  action='store')
-
-(options, args) = parser.parse_args()
-
-output = subprocess.check_output(['/usr/sbin/rabbitmqctl', 'list_consumers'], shell=False)
-
-consumers = defaultdict(int)
-
-for line in output.split('\n'):
-    parts = line.split('\t')
-    if len(parts) and parts[0] == options.queue_name:
-        consumers[parts[0]] += 1
-
-
-now = int(time.time())
-
-if consumers[options.queue_name] < options.min_count:
-    status = 2
-else:
-    status = 0
-
-print("%s|%s|%s|queue %s has %s consumers, needs %s" % (
-            now, status, states[status], options.queue_name,
-            consumers[options.queue_name], options.min_count))

bots/gcal-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import print_function
 import sys
 import time

bots/irc-mirror.py

@@ -1,4 +1,4 @@
-#! /usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # EXPERIMENTAL
 # IRC <=> Zulip mirroring bot
@@ -93,11 +93,11 @@ class IRCBot(irc.bot.SingleServerIRCBot):
                 return
             self.dcc_connect(address, port)
 
-usage = """python2.7 irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
+usage = """python irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
 
 Example:
 
-python2.7 irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
+python irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
   --site=https://zulip.example.com --user=irc-bot@example.com
   --api-key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 

bots/jabber_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2014 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -27,10 +27,13 @@ import subprocess
 import os
 import traceback
 import signal
+from types import FrameType
+from typing import Any
 from zulip import RandomExponentialBackoff
 
 def die(signal, frame):
-    # We actually want to exit, so run os._exit (so as not to be caught and restarted)
+    # type: (int, FrameType) -> None
+    """We actually want to exit, so run os._exit (so as not to be caught and restarted)"""
     os._exit(1)
 
 signal.signal(signal.SIGINT, die)

bots/jabber_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 #
 # Copyright (C) 2013 Permabit, Inc.
 # Copyright (C) 2013--2014 Zulip, Inc.
@@ -381,7 +381,8 @@ option does not affect login credentials.'''.replace("\n", " "))
             config.readfp(f, config_file)
     except IOError:
         pass
-    for option in ("jid", "jabber_password", "conference_domain", "mode", "zulip_email_suffix", "jabber_server_address", "jabber_server_port"):
+    for option in ("jid", "jabber_password", "conference_domain", "mode", "zulip_email_suffix",
+                   "jabber_server_address", "jabber_server_port"):
         if (getattr(options, option) is None
             and config.has_option("jabber_mirror", option)):
             setattr(options, option, config.get("jabber_mirror", option))

bots/log2zulip

@@ -1,16 +1,25 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import print_function
-import subprocess
-import os
-import sys
-import shutil
+
 import errno
-import json
-import ujson
+import os
 import platform
 import re
+import sys
+import shutil
+import subprocess
 import traceback
 
+try:
+    # Use the Zulip virtualenv if available
+    sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
+    import scripts.lib.setup_path_on_import
+except ImportError:
+    pass
+
+import json
+import ujson
+
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../api"))
 import zulip
 

bots/process_ccache

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import sys
 import subprocess
 import base64

bots/sync-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 import sys
 import os
 import logging

bots/tddium-notify-humbug

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person

bots/zephyr_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -31,15 +31,21 @@ import signal
 
 from .zephyr_mirror_backend import parse_args
 
+(options, args) = parse_args()
+
+sys.path[:0] = [os.path.join(options.root_path, 'api')]
+
+from types import FrameType
+from typing import Any
+
 def die(signal, frame):
+    # type: (int, FrameType) -> None
+
     # We actually want to exit, so run os._exit (so as not to be caught and restarted)
     os._exit(1)
 
 signal.signal(signal.SIGINT, die)
 
-(options, args) = parse_args()
-
-sys.path[:0] = [os.path.join(options.root_path, 'api')]
 from zulip import RandomExponentialBackoff
 
 args = [os.path.join(options.root_path, "user_root", "zephyr_mirror_backend.py")]
@@ -57,6 +63,7 @@ if options.forward_class_messages and not options.noshard:
     print("Starting parallel zephyr class mirroring bot")
     jobs = list("0123456789abcdef")
     def run_job(shard):
+        # type: (str) -> int
         subprocess.call(args + ["--shard=%s" % (shard,)])
         return 0
     for (status, job) in run_parallel(run_job, jobs, threads=16):

bots/zephyr_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -92,7 +92,7 @@ def different_paragraph(line, next_line):
             len(line) < len(words[0]))
 
 # Linewrapping algorithm based on:
-# http://gcbenison.wordpress.com/2011/07/03/a-program-to-intelligently-remove-carriage-returns-so-you-can-paste-text-without-having-it-look-awful/
+# http://gcbenison.wordpress.com/2011/07/03/a-program-to-intelligently-remove-carriage-returns-so-you-can-paste-text-without-having-it-look-awful/ #ignorelongline
 def unwrap_lines(body):
     lines = body.split("\n")
     result = ""
@@ -598,7 +598,7 @@ def zcrypt_encrypt_content(zephyr_class, instance, content):
 def forward_to_zephyr(message):
     support_heading = "Hi there! This is an automated message from Zulip."
     support_closing = """If you have any questions, please be in touch through the \
-Feedback button or at support@zulip.com."""
+Feedback button or at support@zulipchat.com."""
 
     wrapper = textwrap.TextWrapper(break_long_words=False, break_on_hyphens=False)
     wrapped_content = "\n".join("\n".join(wrapper.wrap(line))

changelog.md

@@ -1,81 +0,0 @@
-# Change Log
-
-All notable changes to this project will be documented in this file.
-
-[Unreleased]
-
-[1.3.11]
-- Moved email digest support into the default Zulip production configuration.
-- Added options for configuring Postgres, RabbitMQ, Redis, and memcached
-  in settings.py.
-- Added documentation on using Hubot to integrate with useful services
-  not yet integrated with Zulip directly (e.g. Google Hangouts).
-- Added new management command to test sending email from Zulip.
-- Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.
-- Added Nagios plugins to the main distribution.
-- Added ability for realm administrators to manage custom emoji.
-- Added guide to writing new integrations.
-- Enabled camo image proxy to fix mixed-content warnings for http images.
-- Refactored the Zulip puppet modules to be more modular.
-- Refactored the Tornado event system, fixing old memory leaks.
-- Removed many old-style /json API endpoints
-- Implemented running queue processors multithreaded in development,
-  decreasing RAM requirements for a Zulip development environment from
-  ~1GB to ~300MB.
-- Fixed rerendering the complete buddy list whenever a user came back from
-  idle, which was a significant performance issue in larger realms.
-- Fixed the disabling of desktop notifications from 1.3.7 for new users.
-- Fixed the (admin) create_user API enforcing restricted_to_domain, even
-  if that setting was disabled for the realm.
-- Fixed bugs changing certain settings in administration pages.
-- Fixed collapsing messages in narrowed views.
-- Fixed 500 errors when uploading a non-image file as an avatar.
-- Fixed Jira integration incorrectly not @-mentioning assignee.
-
-[1.3.10]
-- Added new integration for Travis CI.
-- Added settings option to control maximum file upload size.
-- Added support for running Zulip development environment in Docker.
-- Added easy configuration support for a remote postgres database.
-- Added extensive documentation on scalability, backups, and security.
-- Recent private message threads are now displayed expanded similar to
-  the pre-existing recent topics feature.
-- Made it possible to set LDAP and EMAIL_HOST passwords in
-  /etc/zulip/secrets.conf.
-- Improved the styling for the Administration page and added tabs.
-- Substantially improved loading performance on slow networks by enabling
-  GZIP compression on more assets.
-- Changed the page title in narrowed views to include the current narrow.
-- Fixed several backend performance issues affecting very large realms.
-- Fixed bugs where draft compose content might be lost when reloading site.
-- Fixed support for disabling the "zulip" notifications stream.
-- Fixed missing step in postfix_localmail installation instructions.
-- Fixed several bugs/inconveniences in the production upgrade process.
-- Fixed realm restrictions for servers with a unique, open realm.
-- Substantially cleaned up console logging from run-dev.py.
-
-[1.3.9] - 2015-11-16
-- Fixed buggy #! lines in upgrade scripts.
-
-[1.3.8] - 2015-11-15
-- Added options to the Python api for working with untrusted server certificates.
-- Added a lot of documentation on the development environment and testing.
-- Added partial support for translating the Zulip UI.
-- Migrated installing Node dependencies to use npm.
-- Fixed LDAP integration breaking autocomplete of @-mentions.
-- Fixed admin panel reactivation/deactivation of bots.
-- Fixed inaccurate documentation for downloading the desktop apps.
-- Fixed various minor bugs in production installation process.
-- Fixed security issue where recent history on private streams might
-  be visible to new users (to the Zulip team) who were invited with that
-  private stream as one of their initial streams
-  (https://github.com/zulip/zulip/issues/230).
-- Major preliminary progress towards supporting Python 3.
-
-[1.3.7] - 2015-10-19
-- Turn off desktop and audible notifications for streams by default.
-- Added support for the LDAP authentication integration creating new users.
-- Added new endpoint to support Google auth on mobile.
-- Fixed desktop notifications in modern Firefox.
-- Fixed several installation issues for both production and development environments.
-- Improved documentation for outgoing SMTP and the email mirror integration.

confirmation/__init__.py

@@ -2,23 +2,23 @@
 
 # Copyright: (c) 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 
-# Permission is hereby granted, free of charge, to any person obtaining a                                                       
-# copy of this software and associated documentation files (the                                                                 
-# "Software"), to deal in the Software without restriction, including                                                           
-# without limitation the rights to use, copy, modify, merge, publish, dis-                                                      
-# tribute, sublicense, and/or sell copies of the Software, and to permit                                                        
-# persons to whom the Software is furnished to do so, subject to the fol-                                                       
-# lowing conditions:                                                                                                            
-#                                                                                                                               
-# The above copyright notice and this permission notice shall be included                                                       
-# in all copies or substantial portions of the Software.                                                                        
-#                                                                                                                               
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS                                                       
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-                                                      
-# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT                                                        
-# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,                                                         
-# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,                                                            
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS                                                        
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
 # IN THE SOFTWARE.
 
 VERSION = (0, 9, 'pre')

confirmation/management/commands/cleanupconfirmation.py

@@ -4,6 +4,7 @@
 
 __revision__ = '$Id: cleanupconfirmation.py 5 2008-11-18 09:10:12Z jarek.zgoda $'
 
+from typing import Any
 
 from django.core.management.base import NoArgsCommand
 
@@ -14,4 +15,5 @@ class Command(NoArgsCommand):
     help = 'Delete expired confirmations from database'
 
     def handle_noargs(self, **options):
+        # type: (**Any) -> None
         Confirmation.objects.delete_expired_confirmations()

confirmation/migrations/0002_realmcreationkey.py

@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+import django.utils.timezone
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('confirmation', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='RealmCreationKey',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('creation_key', models.CharField(max_length=40, verbose_name='activation key')),
+                ('date_created', models.DateTimeField(default=django.utils.timezone.now, verbose_name='created')),
+            ],
+        ),
+    ]

confirmation/models.py

@@ -30,15 +30,33 @@ except ImportError:
 
 B16_RE = re.compile('^[a-f0-9]{40}$')
 
+def check_key_is_valid(creation_key):
+    if not RealmCreationKey.objects.filter(creation_key=creation_key).exists():
+        return False
+    days_sofar = (now() - RealmCreationKey.objects.get(creation_key=creation_key).date_created).days
+    # Realm creation link expires after settings.REALM_CREATION_LINK_VALIDITY_DAYS
+    if days_sofar <= settings.REALM_CREATION_LINK_VALIDITY_DAYS:
+        return True
+    return False
+
 def generate_key():
     return generate_random_token(40)
 
-def generate_activation_url(key):
+def generate_activation_url(key, host=None):
+    if host is None:
+        host = settings.EXTERNAL_HOST
     return u'%s%s%s' % (settings.EXTERNAL_URI_SCHEME,
-                        settings.EXTERNAL_HOST,
+                        host,
                         reverse('confirmation.views.confirm',
                                 kwargs={'confirmation_key': key}))
 
+def generate_realm_creation_url():
+    key = generate_key()
+    RealmCreationKey.objects.create(creation_key=key, date_created=now())
+    return u'%s%s%s' % (settings.EXTERNAL_URI_SCHEME,
+                        settings.EXTERNAL_HOST,
+                        reverse('zerver.views.create_realm',
+                        kwargs={'creation_key': key}))
 
 class ConfirmationManager(models.Manager):
 
@@ -55,16 +73,17 @@ class ConfirmationManager(models.Manager):
             return obj
         return False
 
-    def get_link_for_object(self, obj):
+    def get_link_for_object(self, obj, host=None):
         key = generate_key()
         self.create(content_object=obj, date_sent=now(), confirmation_key=key)
-        return generate_activation_url(key)
+        return generate_activation_url(key, host=host)
 
     def send_confirmation(self, obj, email_address, additional_context=None,
-            subject_template_path=None, body_template_path=None):
+                          subject_template_path=None, body_template_path=None,
+                          host=None):
         confirmation_key = generate_key()
         current_site = Site.objects.get_current()
-        activate_url = generate_activation_url(confirmation_key)
+        activate_url = generate_activation_url(confirmation_key, host=host)
         context = Context({
             'activate_url': activate_url,
             'current_site': current_site,
@@ -74,8 +93,12 @@ class ConfirmationManager(models.Manager):
         })
         if additional_context is not None:
             context.update(additional_context)
+        if obj.realm is not None and obj.realm.is_zephyr_mirror_realm:
+            template_name = "mituser"
+        else:
+            template_name = obj._meta.model_name
         templates = [
-            'confirmation/%s_confirmation_email_subject.txt' % obj._meta.model_name,
+            'confirmation/%s_confirmation_email_subject.txt' % (template_name,),
             'confirmation/confirmation_email_subject.txt',
         ]
         if subject_template_path:
@@ -84,7 +107,7 @@ class ConfirmationManager(models.Manager):
             template = loader.select_template(templates)
         subject = template.render(context).strip().replace(u'\n', u' ') # no newlines, please
         templates = [
-            'confirmation/%s_confirmation_email_body.txt' % obj._meta.model_name,
+            'confirmation/%s_confirmation_email_body.txt' % (template_name,),
             'confirmation/confirmation_email_body.txt',
         ]
         if body_template_path:
@@ -110,4 +133,8 @@ class Confirmation(models.Model):
         verbose_name_plural = _('confirmation emails')
 
     def __unicode__(self):
-        return _('confirmation email for %s') % self.content_object
+        return _('confirmation email for %s') % (self.content_object,)
+
+class RealmCreationKey(models.Model):
+    creation_key = models.CharField(_('activation key'), max_length=40)
+    date_created = models.DateTimeField(_('created'), default=now)

confirmation/util.py

@@ -9,4 +9,4 @@ from django.conf import settings
 def get_status_field(app_label, model_name):
     model = '%s.%s' % (app_label, model_name)
     mapping = getattr(settings, 'STATUS_FIELDS', {})
-    return mapping.get(model, 'status')
+    return mapping.get(model, 'status')

confirmation/views.py

@@ -8,11 +8,14 @@ __revision__ = '$Id: views.py 21 2008-12-05 09:21:03Z jarek.zgoda $'
 from django.shortcuts import render_to_response
 from django.template import RequestContext
 from django.conf import settings
+from django.http import HttpRequest, HttpResponse
 
 from confirmation.models import Confirmation
+from zproject.jinja2 import render_to_response
 
 
 def confirm(request, confirmation_key):
+    # type: (HttpRequest, str) -> HttpResponse
     confirmation_key = confirmation_key.lower()
     obj = Confirmation.objects.confirm(confirmation_key)
     confirmed = True
@@ -31,13 +34,12 @@ def confirm(request, confirmation_key):
         'key': confirmation_key,
         'full_name': request.GET.get("full_name", None),
         'support_email': settings.ZULIP_ADMINISTRATOR,
-        'voyager': settings.VOYAGER
+        'verbose_support_offers': settings.VERBOSE_SUPPORT_OFFERS,
     }
     templates = [
         'confirmation/confirm.html',
     ]
     if obj:
         # if we have an object, we can use specific template
-        templates.insert(0, 'confirmation/confirm_%s.html' % obj._meta.model_name)
-    return render_to_response(templates, ctx,
-        context_instance=RequestContext(request))
+        templates.insert(0, 'confirmation/confirm_%s.html' % (obj._meta.model_name,))
+    return render_to_response(templates, ctx, request=request)

contrib_bots/lib/followup.py

@@ -0,0 +1,53 @@
+# See readme.md for instructions on running this code.
+
+class FollowupHandler(object):
+    '''
+    This plugin facilitates creating follow-up tasks when
+    you are using Zulip to conduct a virtual meeting.  It
+    looks for messages starting with '@followup'.
+
+    In this example, we write follow up items to a special
+    Zulip stream called "followup," but this code could
+    be adapted to write follow up items to some kind of
+    external issue tracker as well.
+    '''
+
+    def usage(self):
+        return '''
+            This plugin will allow users to flag messages
+            as being follow-up items.  Users should preface
+            messages with "@followup".
+
+            Before running this, make sure to create a stream
+            called "followup" that your API user can send to.
+            '''
+
+    def triage_message(self, message):
+        # return True iff we want to (possibly) response to this message
+
+        original_content = message['content']
+
+        # This next line of code is defensive, as we
+        # never want to get into an infinite loop of posting follow
+        # ups for own follow ups!
+        if message['display_recipient'] == 'followup':
+            return False
+        is_follow_up = (original_content.startswith('@followup') or
+                        original_content.startswith('@follow-up'))
+
+        return is_follow_up
+
+    def handle_message(self, message, client):
+        original_content = message['content']
+        original_sender = message['sender_email']
+        new_content = original_content.replace('@followup',
+                                               'from %s:' % (original_sender,))
+
+        client.send_message(dict(
+            type='stream',
+            to='followup',
+            subject=message['sender_email'],
+            content=new_content,
+        ))
+
+handler_class = FollowupHandler

contrib_bots/lib/help.py

@@ -0,0 +1,39 @@
+# See readme.md for instructions on running this code.
+
+class HelpHandler(object):
+    def usage(self):
+        return '''
+            This plugin will give info about Zulip to
+            any user that types a message saying "help".
+
+            This is example code; ideally, you would flesh
+            this out for more useful help pertaining to
+            your Zulip instance.
+            '''
+
+    def triage_message(self, message):
+        # return True if we think the message may be of interest
+        original_content = message['content']
+
+        if message['type'] != 'stream':
+            return True
+
+        if original_content.lower().strip() != 'help':
+            return False
+
+        return True
+
+    def handle_message(self, message, client):
+        help_content = '''
+            Info on Zulip can be found here:
+            https://github.com/zulip/zulip
+            '''.strip()
+
+        client.send_message(dict(
+            type='stream',
+            to=message['display_recipient'],
+            subject=message['subject'],
+            content=help_content,
+        ))
+
+handler_class = HelpHandler

contrib_bots/lib/readme.md

@@ -0,0 +1,109 @@
+# Overview
+
+This is the documentation for an experimental new system for writing
+bots that react to messages.
+
+This directory contains library code for running Zulip
+bots that react to messages sent by users.
+
+This document explains how to run the code, and it also
+talks about the architecture for creating bots.
+
+## Design goals
+
+The goal is to have a common framework for hosting a bot that reacts
+to messages in any of the following settings:
+
+* Run as a long-running process using `call_on_each_event`
+  (implemented today).
+
+* Run via a simple web service that can be deployed to PAAS providers
+  and handles outgoing webhook requests from Zulip.
+
+* Embedded into the Zulip server (so that no hosting is required),
+  which would be done for high quality, reusable bots; we would have a
+  nice "bot store" sort of UI for browsing and activating them.
+
+## Running bots
+
+Here is an example of running the "follow-up" bot from
+inside a Zulip repo:
+
+    cd ~/zulip/contrib_bots
+    python run.py lib/followup.py --config-file ~/.zuliprc-prod
+
+Once the bot code starts running, you will see a
+message explaining how to use the bot, as well as
+some log messages.  You can use the `--quiet` option
+to suppress these messages.
+
+The bot code will run continuously until you kill them with
+control-C (or otherwise).
+
+### Configuration
+
+For this document we assume you have some prior experience
+with using the Zulip API, but here is a quick review of
+what a `.zuliprc` files looks like.  You can connect to the
+API as your own human user, or you can go into the Zulip settings
+page to create a user-owned bot.
+
+    [api]
+    email=someuser@example.com
+    key=<your api key>
+    site=https://zulip.somewhere.com
+
+## Architecture
+
+In order to make bot development easy, we separate
+out boilerplate code (loading up the Client API, etc.)
+from bot-specific code (do what makes the bot unique).
+
+All of the boilerplate code lives in `../run.py`.  The
+runner code does things like find where it can import
+the Zulip API, instantiate a client with correct
+credentials, set up the logging level, find the
+library code for the specific bot, etc.
+
+Then, for bot-specific logic, you will find `.py` files
+in the `lib` directory (i.e. the same directory as the
+document you are reading now).
+
+Each bot library simply needs to do the following:
+
+- Define a class that supports the methods `usage`,
+`triage_message`, and `handle_message`.
+- Set `handler_class` to be the name of that class.
+
+(We make this a two-step process, so that you can give
+a descriptive name to your handler class.)
+
+## Portability
+
+Creating a handler class for each bot allows your bot
+code to be more portable.  For example, if you want to
+use your bot code in some other kind of bot platform, then
+if all of your bots conform to the `handler_class` protocol,
+you can write simple adapter code to use them elsewhere.
+
+Another future direction to consider is that Zulip will
+eventually support running certain types of bots on
+the server side, to essentially implement post-send
+hooks and things of those nature.
+
+Conforming to the `handler_class` protocol will make
+it easier for Zulip admins to integrate custom bots.
+
+In particular, `run.py` already passes in instances
+of a restricted variant of the Client class to your
+library code, which helps you ensure that your bot
+does only things that would be acceptable for running
+in a server-side environment.
+
+## Other approaches
+
+If you are not interested in running your bots on the
+server, then you can still use the full Zulip API.  The
+hope, though, is that this architecture will make
+writing simple bots a quick/easy process.
+

contrib_bots/run.py

@@ -0,0 +1,100 @@
+from __future__ import print_function
+
+import importlib
+import logging
+import optparse
+import os
+import sys
+
+our_dir = os.path.dirname(os.path.abspath(__file__))
+
+# For dev setups, we can find the API in the repo itself.
+if os.path.exists(os.path.join(our_dir, '../api/zulip')):
+    sys.path.append('../api')
+
+from zulip import Client
+
+class RestrictedClient(object):
+    def __init__(self, client):
+        # Only expose a subset of our Client's functionality
+        self.send_message = client.send_message
+
+def get_lib_module(lib_fn):
+    lib_fn = os.path.abspath(lib_fn)
+    if os.path.dirname(lib_fn) != os.path.join(our_dir, 'lib'):
+        print('Sorry, we will only import code from contrib_bots/lib.')
+        sys.exit(1)
+
+    if not lib_fn.endswith('.py'):
+        print('Please use a .py extension for library files.')
+        sys.exit(1)
+
+    sys.path.append('lib')
+    base_lib_fn = os.path.basename(os.path.splitext(lib_fn)[0])
+    module_name = 'lib.' + base_lib_fn
+    module = importlib.import_module(module_name)
+    return module
+
+def run_message_handler_for_bot(lib_module, quiet, config_file):
+    # Make sure you set up your ~/.zuliprc
+    client = Client(config_file=config_file)
+    restricted_client = RestrictedClient(client)
+
+    message_handler = lib_module.handler_class()
+
+    if not quiet:
+        print(message_handler.usage())
+
+    def handle_message(message):
+        logging.info('waiting for next message')
+        if message_handler.triage_message(message=message):
+            message_handler.handle_message(
+                message=message,
+                client=restricted_client)
+
+    logging.info('starting message handling...')
+    client.call_on_each_message(handle_message)
+
+def run():
+    usage = '''
+        python run.py <lib file>
+
+        Example: python run.py lib/followup.py
+
+        (This program loads bot-related code from the
+        library code and then runs a message loop,
+        feeding messages to the library code to handle.)
+
+        Please make sure you have a current ~/.zuliprc
+        file with the credentials you want to use for
+        this bot.
+
+        See lib/readme.md for more context.
+        '''
+
+    parser = optparse.OptionParser(usage=usage)
+    parser.add_option('--quiet', '-q',
+        action='store_true',
+        help='Turn off logging output.')
+    parser.add_option('--config-file',
+        action='store',
+        help='(alternate config file to ~/.zuliprc)')
+    (options, args) = parser.parse_args()
+
+    if len(args) == 0:
+        print('You must specify a library!')
+        sys.exit(1)
+
+    lib_module = get_lib_module(lib_fn=args[0])
+
+    if not options.quiet:
+        logging.basicConfig(stream=sys.stdout, level=logging.INFO)
+
+    run_message_handler_for_bot(
+        lib_module=lib_module,
+        config_file=options.config_file,
+        quiet=options.quiet
+    )
+
+if __name__ == '__main__':
+    run()

corporate/terms.md

@@ -0,0 +1,244 @@
+# Zulip Terms of Service
+
+### Welcome to Zulip!
+
+Thanks for using our products and services ("Services"). The Services are
+provided by Zulip, Inc. ("Zulip"), located at 552 Massachusetts Ave Suite 203,
+Cambridge, MA 02139, United States.
+
+By using our Services, you are agreeing to these terms. Please read them
+carefully.
+
+The Services are not intended for use by you if you are under 13 years of
+age. By agreeing to these terms, you are representing to us that you are over
+13.
+
+### Using our Services
+
+You must follow any policies made available to you within the Services.
+
+Don't misuse our Services. For example, don't interfere with our Services or
+try to access them using a method other than the interface and the instructions
+that we provide. You may use our Services only as permitted by law, including
+applicable export and re-export control laws and regulations. We may suspend or
+stop providing our Services to you if you do not comply with our terms or
+policies or if we are investigating suspected misconduct.
+
+Using our Services does not give you ownership of any intellectual property
+rights in our Services or the content you access. You may not use content from
+our Services unless you obtain permission from its owner or are otherwise
+permitted by law. These terms do not grant you the right to use any branding or
+logos used in our Services. Don't remove, obscure, or alter any legal notices
+displayed in or along with our Services.
+
+Our Services display some content that is not Zulip's. This content is the
+sole responsibility of the entity that makes it available. We may review
+content to determine whether it is illegal or violates our policies, and we may
+remove or refuse to display content that we reasonably believe violates our
+policies or the law. But that does not necessarily mean that we review content,
+so please don't assume that we do.
+
+In connection with your use of the Services, we may send you service
+announcements, administrative messages, and other information. You may opt out
+of some of those communications.
+
+### Your Zulip Account
+
+You may need a Zulip Account in order to use some of our Services. You may
+create your own Zulip Account, or your Zulip Account may be assigned to you
+by an administrator, such as your employer or educational institution. If you
+are using a Zulip Account assigned to you by an administrator, different or
+additional terms may apply and your administrator may be able to access or
+disable your account.
+
+If you learn of any unauthorized use of your password or account, contact
+[support@zulip.com](mailto:support@zulip.com).
+
+### Privacy and Copyright Protection
+
+Zulip's [privacy policy](/privacy) explains how we treat your
+personal data and protect your privacy when you use our Services. By using our
+Services, you agree that Zulip can use such data in accordance with our
+privacy policy.
+
+We respond to notices of alleged copyright infringement and terminate
+accounts of repeat infringers according to the process set out in the U.S.
+Digital Millennium Copyright Act.
+
+Our designated agent for notice of alleged copyright infringement on the
+Services is:
+
+> Copyright Agent  
+> Zulip, Inc.  
+> 552 Massachusetts Ave Suite 203  
+> Cambridge, MA 02139  
+> [copyright@zulip.com](mailto:copyright@zulip.com)
+
+### Your Content in our Services
+
+Some of our Services allow you to submit content. You retain ownership of
+any intellectual property rights that you hold in that content. In short, what
+belongs to you stays yours.
+
+When you upload or otherwise submit content to our Services, you give Zulip
+(and those we work with) a worldwide license to use, host, store, reproduce,
+modify, create derivative works (such as those resulting from translations,
+adaptations or other changes we make so that your content works better with our
+Services), communicate, publish, perform, display and distribute such content.
+The rights you grant in this license are for the limited purpose of operating
+and improving our Services, and to develop new ones. This license continues
+even if you stop using our Services (for example, so that we can deliver a
+message that you sent to another Zulip Account before you stopped using our
+Services). Some Services may offer you ways to access and remove content that
+has been provided to that Service. Also, in some of our Services, there may be
+terms or settings that narrow the scope of our use of the content submitted in
+those Services. Make sure you have the necessary rights to grant us this
+license for any content that you submit to our Services. If you use the
+Services to share content with others, anyone you've shared content with
+(including the general public, in certain circumstances) may have access to the
+content.
+
+In order to provide the Services, our servers save a record of the messages
+received by each Zulip Account (the "Received Messsage Information" for the
+account). If you are using our Services on behalf of a business and a
+representative of that business sends [data@zulip.com](mailto:data@zulip.com)
+a request to delete all of your business' accounts with us, then within a
+commercially reasonable period of time, we will close all of your business'
+accounts with us and delete the Received Message Information for each such
+account by removing pointers to the information on our active servers and
+overwriting it over time. Notwithstanding the foregoing, deleting the Received
+Message Information for your business' accounts will not require deleting any
+information about messages that were sent or received by any Zulip Accounts that
+are not one of your business' accounts with us (such as system-wide announcement
+messages or any messages corresponding with the Zulip support team).
+
+You can find more information about how Zulip uses and stores content in
+the privacy policy. If you submit feedback or suggestions about our Services,
+we may use your feedback or suggestions without obligation to you.
+
+
+### About Software in our Services
+
+When a Service requires or includes downloadable software, this software may
+update automatically on your device once a new version or feature is available.
+Some Services may let you adjust your automatic update settings.
+
+Zulip gives you a personal, worldwide, royalty-free, non-assignable and
+non-exclusive license to use the software provided to you by Zulip as part of
+the Services. This license is for the sole purpose of enabling you to use and
+enjoy the benefit of the Services as provided by Zulip, in the manner
+permitted by these terms. You may not copy, modify, distribute, sell, or lease
+any part of our Services or included software, nor may you reverse engineer or
+attempt to extract the source code of that software, unless laws prohibit those
+restrictions or you have our written permission.
+
+Some software used in our Services may be offered under an open source
+license that we will make available to you.  There may be provisions in the
+open source license that expressly override some of these terms.
+
+
+### Modifying and Terminating our Services
+
+We are constantly changing and improving our Services. We may add or remove
+functionalities or features, and we may suspend or stop a Service
+altogether.
+
+You can stop using our Services at any time, although we'll be sorry to see
+you go. Zulip may also stop providing Services to you, or add or create new
+limits to our Services at any time.
+
+We believe that you own your data and preserving your access to such data is
+important. If we discontinue a Service, we will, if it is practical in our sole
+discretion, give you reasonable advance notice and a chance to get information
+out of that Service.
+
+
+### Our Warranties and Disclaimers
+
+We hope that you will enjoy using our Services, but there are certain things
+that we don't promise about our Services.
+
+OTHER THAN AS EXPRESSLY SET OUT IN THESE TERMS, NEITHER ZULIP NOR ITS
+SUPPLIERS OR DISTRIBUTORS MAKE ANY SPECIFIC PROMISES ABOUT THE SERVICES.  FOR
+EXAMPLE, WE DON'T MAKE ANY COMMITMENTS ABOUT THE CONTENT WITHIN THE SERVICES,
+THE SPECIFIC FUNCTION OF THE SERVICES, OR THEIR RELIABILITY, AVAILABILITY, OR
+ABILITY TO MEET YOUR NEEDS. WE PROVIDE THE SERVICES "AS IS".
+
+SOME JURISDICTIONS PROVIDE FOR CERTAIN WARRANTIES, LIKE THE IMPLIED WARRANTY
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. TO
+THE EXTENT PERMITTED BY LAW, WE EXCLUDE ALL WARRANTIES.
+
+
+### Liability for our Services
+
+WHEN PERMITTED BY LAW, ZULIP, AND ZULIP'S SUPPLIERS AND DISTRIBUTORS, WILL
+NOT BE RESPONSIBLE FOR LOST PROFITS, REVENUES, OR DATA, FINANCIAL LOSSES OR
+INDIRECT, SPECIAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES.
+
+TO THE EXTENT PERMITTED BY LAW, THE TOTAL LIABILITY OF ZULIP, AND ITS
+SUPPLIERS AND DISTRIBUTORS, FOR ANY CLAIM UNDER THESE TERMS, INCLUDING FOR ANY
+IMPLIED WARRANTIES, IS LIMITED TO THE GREATER OF FIVE DOLLARS ($5) OR THE
+AMOUNT PAID BY YOU TO ZULIP FOR THE PAST THREE MONTHS OF THE SERVICES IN
+QUESTION.
+
+IN ALL CASES, ZULIP, AND ITS SUPPLIERS AND DISTRIBUTORS, WILL NOT BE LIABLE
+FOR ANY LOSS OR DAMAGE THAT IS NOT REASONABLY FORESEEABLE.
+
+
+### Business uses of our Services
+
+If you are using our Services on behalf of a business, that business accepts
+these terms. It will hold harmless and indemnify Zulip and its affiliates,
+officers, agents, and employees from any claim, suit or action arising from or
+related to the use of the Services or violation of these terms, including any
+liability or expense arising from claims, losses, damages, suits, judgments,
+litigation costs and attorneys' fees.
+
+You agree that we, in our sole discretion, may use your trade names,
+trademarks, service marks, logos, domain names and other distinctive brand
+features in presentations, marketing materials, customer lists, financial
+reports and Web site listings (including links to your website) for the purpose
+of advertising or publicizing your use of our products or services.
+
+### Export Compliance
+
+If you are located outside of the United States or are not a U.S. person,
+you certify that you do not reside in Cuba, Iran, North Korea, Sudan, or Syria,
+and you certify the following: "We certify that this beta test software will
+only be used for beta testing purposes, and will not be rented, leased, sold,
+sublicensed, assigned, or otherwise transferred. Further, we certify that we
+will not transfer or export any product, process, or service that is the direct
+product of the beta test software."
+
+
+### About these Terms
+
+If it turns out that a particular term is not enforceable, this will not
+affect any other terms.
+
+If you do not comply with these terms, and we don't take action right away,
+this doesn't mean that we are giving up any rights that we may have (such as
+taking action in the future).
+
+These terms control the relationship between Zulip and you. They do not
+create any third party beneficiary rights.
+
+The laws of Massachusetts, U.S.A., excluding Massachusetts's conflict of
+laws rules, will apply to any disputes arising out of or relating to these
+terms or the Services. All claims arising out of or relating to these terms or
+the Services will be litigated exclusively in the applicable federal or state
+courts of Massachusetts, and you and Zulip consent to personal jurisdiction in
+those courts.
+
+Zulip reserves the right to amend or modify these terms at any time and in
+any manner by providing reasonable notice to you. You agree that reasonable
+notice may be provided by posting on Zulip's web site, email, or other written
+notice.   By continuing to access or use the Services after revisions become
+effective, you agree to be bound by the revised terms. If you do not agree to
+the new terms, please stop using the Services.
+
+These terms constitute the whole legal agreement between you and Zulip, and
+completely replace any prior agreements between you and Zulip in relation to
+the Services.
+
+Last modified: October 4, 2013

corporate/urls.py

@@ -1,15 +1,15 @@
 from django.conf.urls import patterns, url
 from django.views.generic import TemplateView, RedirectView
 
-urlpatterns = patterns('',
+i18n_urlpatterns = [
     # Zephyr/MIT
     url(r'^zephyr/$', TemplateView.as_view(template_name='corporate/zephyr.html')),
     url(r'^mit/$', TemplateView.as_view(template_name='corporate/mit.html')),
     url(r'^zephyr-mirror/$', TemplateView.as_view(template_name='corporate/zephyr-mirror.html')),
 
     # Terms of service and privacy policy
-    url(r'^terms/$',   TemplateView.as_view(template_name='corporate/terms.html')),
     url(r'^terms-enterprise/$',  TemplateView.as_view(template_name='corporate/terms-enterprise.html')),
     url(r'^privacy/$', TemplateView.as_view(template_name='corporate/privacy.html')),
+]
 
-)
+urlpatterns = patterns('', *i18n_urlpatterns)

docs/README.md

@@ -1,23 +1,29 @@
-These docs are written in rST, and are included on the zulip.org website
-as well as on each development installation.  Many of these docs
-have been ported from the internal docs of Zulip Inc.,
-and may need to be updated for use in the open source project.
+# Documentation
 
-To generate HTML docs locally from rST:
+These docs are written in [Commonmark
+Markdown](http://commonmark.org/) with a small bit of rST.  We've
+chosen Markdown because it is [easy to
+write](http://commonmark.org/help).  The docs are served in production
+at [zulip.readthedocs.io](https://zulip.readthedocs.io/en/latest/).
 
-   * `pip install sphinx`
-   * In this directory, `make html`. Output appears in a `_build/html` subdirectory.
+If you want to build the documentation locally (e.g. to test your
+changes), the dependencies are automatically installed as part of
+Zulip development environment provisioning, and you can build the
+documentation using:
 
-To create rST from MediaWiki input:
+```
+cd docs/
+make html
+```
 
-   * Use `pandoc -r mediawiki -w rst` on MediaWiki source.
-   * Use unescape.py to remove any leftover HTML entities (often inside <pre>
-     tags and the like).
+and then opening `file:///path/to/zulip/docs/_build/html/index.html` in
+your browser (you can also use e.g. `firefox
+docs/_build/html/index.html` from the root of your Zulip checkout).
 
-We can use pandoc to translate mediawiki into reStructuredText, but some things need fixing up:
-
-   * Add page titles.
-   * Review pages for formatting (especially inline code chunks) and content.
-   * Fix wiki links?
-   * Add pages to the table of contents (`index.rst`).
+You can also usually test your changes by pushing a branch to GitHub
+and looking at the content on the GitHub web UI, since GitHub renders
+Markdown.
 
+When editing dependencies for the Zulip documentation, you should edit
+`requirements/docs.txt` (which is used by ReadTheDocs to build the
+documentation quickly, without installing all of Zulip's dependencies).

docs/architecture-overview.md

@@ -0,0 +1,238 @@
+Zulip architectural overview
+============================
+
+Key Codebases
+-------------
+
+The core Zulip application is at
+[<https://github.com/zulip/zulip>](https://github.com/zulip/zulip) and
+is a web application written in Python 2.7 (soon to also support
+Python 3) and using the Django framework. That codebase includes
+server-side code and the web client, as well as Python API bindings
+and most of our integrations with other services and applications (see
+[the directory structure guide](directory-structure.html)).
+
+We maintain several separate repositories for integrations and other
+glue code: a [Hubot adapter](https://github.com/zulip/hubot-zulip);
+integrations with
+[Phabricator](https://github.com/zulip/phabricator-to-zulip),
+[Jenkins](https://github.com/zulip/zulip-jenkins-plugin),
+[Puppet](https://github.com/matthewbarr/puppet-zulip),
+[Redmine](https://github.com/zulip/zulip-redmine-plugin), and
+[Trello](https://github.com/zulip/trello-to-zulip); [node.js API
+bindings](https://github.com/zulip/zulip-node); and our [full-text
+search PostgreSQL extension](https://github.com/zulip/tsearch_extras).
+
+Our mobile clients are separate code repositories:
+[Android](https://github.com/zulip/zulip-android), [iOS
+(stable)](https://github.com/zulip/zulip-ios), and [our experimental
+React Native iOS app](https://github.com/zulip/zulip-mobile). Our
+[legacy desktop application (implemented in
+QT/WebKit)](https://github.com/zulip/zulip-desktop) and our new, alpha
+[cross-platform desktop app (implemented in
+Electron)](https://github.com/zulip/zulip-electron) are also separate
+repositories.
+
+We use [Transifex](https://www.transifex.com/zulip/zulip/) to do
+translations.
+
+In this overview we'll mainly discuss the core Zulip server and web
+application.
+
+Usage assumptions and concepts
+------------------------------
+
+Zulip is a real-time web-based chat application meant for companies and
+similar groups ranging in size from a small team to more than a thousand
+users. It features real-time notifications, message persistence and
+search, public group conversations (*streams*), invite-only streams,
+private one-on-one and group conversations, inline image previews, team
+presence/buddy list, a rich API, Markdown message support, and numerous
+integrations with other services. The maintainer team aims to support
+users who connect to Zulip using dedicated iOS, Android, Linux, Windows,
+and Mac OS X clients, as well as people using modern web browsers or
+dedicated Zulip API clients.
+
+A server can host multiple Zulip *realms* (organizations) at the same
+domain, each of which is a private chamber with its own users,
+streams, customizations, and so on. This means that one person might
+be a user of multiple Zulip realms. The administrators of a realm can
+choose whether to allow anyone to register an account and join, or
+only allow people who have been invited, or restrict registrations to
+members of particular groups (using email domain names or corporate
+single-sign-on login for verification). For more on scalability and
+security considerations, see [the security section of the production
+maintenance
+instructions](prod-maintain-secure-upgrade.html#security-model).
+
+The default Zulip home screen is like a chronologically ordered inbox;
+it displays messages, starting at the oldest message that the user
+hasn't viewed yet (for more on that logic, see [the guide to the
+pointer and unread counts](pointer.html)). The home screen displays
+the most recent messages in all the streams a user has joined (except
+for the streams they've muted), as well as private messages from other
+users, in strict chronological order. A user can *narrow* to view only
+the messages in a single stream, and can further narrow to focus on a
+*topic* (thread) within that stream. Each narrow has its own URL.
+
+Zulip's philosophy is to provide sensible defaults but give the user
+fine-grained control over their incoming information flow; a user can
+mute topics and streams, and can make fine-grained choices to reduce
+real-time notifications they find irrelevant.
+
+Components
+----------
+
+### Tornado and Django
+
+We use both the [Tornado](http://www.tornadoweb.org) and
+[Django](https://www.djangoproject.com/) Python web frameworks.
+
+Django is the main web application server; Tornado runs the
+server-to-client real-time push system. The app servers are configured
+by the Supervisor configuration (which explains how to start the server
+processes; see "Supervisor" below) and the nginx configuration (which
+explains which HTTP requests get sent to which app server).
+
+Tornado is an asynchronous server and is meant specifically to hold open
+tens of thousands of long-lived (long-polling or websocket) connections
+-- that is to say, routes that maintain a persistent connection from
+every running client. For this reason, it's responsible for event
+(message) delivery, but not much else. We try to avoid any blocking
+calls in Tornado because we don't want to delay delivery to thousands of
+other connections (as this would make Zulip very much not real-time).
+For instance, we avoid doing cache or database queries inside the
+Tornado code paths, since those blocking requests carry a very high
+performance penalty for a single-threaded, asynchronous server.
+
+The parts that are activated relatively rarely (e.g. when people type or
+click on something) are processed by the Django application server. One
+exception to this is that Zulip uses websockets through Tornado to
+minimize latency on the code path for **sending** messages.
+
+### nginx
+
+nginx is the front-end web server to all Zulip traffic; it serves static
+assets and proxies to Django and Tornado. It handles HTTP requests
+according to the rules laid down in the many config files found in
+`zulip/puppet/zulip/files/nginx/`.
+
+`zulip/puppet/zulip/files/nginx/zulip-include-frontend/app` is the most
+important of these files. It explains what happens when requests come in
+from outside.
+
+-   In production, all requests to URLs beginning with `/static/` are
+    served from the corresponding files in `/home/zulip/prod-static/`,
+    and the production build process (`tools/build-release-tarball`)
+    compiles, minifies, and installs the static assets into the
+    `prod-static/` tree form. In development, files are served directly
+    from `/static/` in the git repository.
+-   Requests to `/json/get_events`, `/api/v1/events`, and `/sockjs` are
+    sent to the Tornado server. These are requests to the real-time push
+    system, because the user's web browser sets up a long-lived TCP
+    connection with Tornado to serve as [a channel for push
+    notifications](https://en.wikipedia.org/wiki/Push_technology#Long_Polling).
+    nginx gets the hostname for the Tornado server via
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`.
+-   Requests to all other paths are sent to the Django app via the UNIX
+    socket `unix:/home/zulip/deployments/fastcgi-socket` (defined in
+    `puppet/zulip/files/nginx/zulip-include-frontend/upstreams`). We use
+    `zproject/wsgi.py` to implement FastCGI here (see
+    `django.core.wsgi`).
+
+### Supervisor
+
+We use [supervisord](http://supervisord.org/) to start server processes,
+restart them automatically if they crash, and direct logging.
+
+The config file is
+`zulip/puppet/zulip/files/supervisor/conf.d/zulip.conf`. This is where
+Tornado and Django are set up, as well as a number of background
+processes that process event queues. We use event queues for the kinds
+of tasks that are best run in the background because they are
+expensive (in terms of performance) and don't have to be synchronous
+--- e.g., sending emails or updating analytics. Also see [the queuing
+guide](queuing.html).
+
+### memcached
+
+memcached is used to cache database model objects. `zerver/lib/cache.py`
+and `zerver/lib/cache_helpers.py` manage putting things into memcached,
+and invalidating the cache when values change. The memcached
+configuration is in `puppet/zulip/files/memcached.conf`.
+
+### Redis
+
+Redis is used for a few very short-term data stores, such as in the
+basis of `zerver/lib/rate_limiter.py`, a per-user rate limiting scheme
+[example](http://blog.domaintools.com/2013/04/rate-limiting-with-redis/)),
+and the [email-to-Zulip
+integration](https://zulipchat.com/integrations/#email).
+
+Redis is configured in `zulip/puppet/zulip/files/redis` and it's a
+pretty standard configuration except for the last line, which turns off
+persistence:
+
+    # Zulip-specific configuration: disable saving to disk.
+    save ""
+
+memcached was used first and then we added Redis specifically to
+implement rate limiting. [We're discussing switching everything over to
+Redis.](https://github.com/zulip/zulip/issues/16)
+
+### RabbitMQ
+
+RabbitMQ is a queueing system. Its config files live in
+`zulip/puppet/zulip/files/rabbitmq`. Initial configuration happens in
+`zulip/scripts/setup/configure-rabbitmq`.
+
+We use RabbitMQ for queuing expensive work (e.g. sending emails
+triggered by a message, push notifications, some analytics, etc.) that
+require reliable delivery but which we don't want to do on the main
+thread. It's also used for communication between the application server
+and the Tornado push system.
+
+Two simple wrappers around `pika` (the Python RabbitMQ client) are in
+`zulip/server/lib/queue.py`. There's an asynchronous client for use in
+Tornado and a more general client for use elsewhere.
+
+`zerver/lib/event_queue.py` has helper functions for putting events into
+one queue or another. Most of the processes started by Supervisor are
+queue processors that continually pull things out of a RabbitMQ queue
+and handle them.
+
+Also see [the queuing guide](queuing.html).
+
+### PostgreSQL
+
+PostgreSQL (also known as Postgres) is the database that stores all
+persistent data, that is, data that's expected to live beyond a user's
+current session.
+
+In production, Postgres is installed with a default configuration. The
+directory that would contain configuration files
+(`puppet/zulip/files/postgresql`) has only a utility script and a custom
+list of stopwords used by a Postgresql extension.
+
+In a development environment, configuration of that postgresql
+extension is handled by `tools/postgres-init-dev-db` (invoked by
+`tools/provision.py`).  That file also manages setting up the
+development postgresql user.
+
+`tools/provision.py` also invokes `tools/do-destroy-rebuild-database`
+to create the actual database with its schema.
+
+### Nagios
+
+Nagios is an optional component used for notifications to the system
+administrator, e.g., in case of outages.
+
+`zulip/puppet/zulip/manifests/nagios.pp` installs Nagios plugins from
+puppet/`zulip/files/nagios_plugins/`.
+
+This component is intended to install Nagios plugins intended to be run
+on a Nagios server; most of the Zulip Nagios plugins are intended to be
+run on the Zulip servers themselves, and are included with the relevant
+component of the Zulip server (e.g.
+`puppet/zulip/manifests/postgres_common.pp` installs a few under
+`/usr/lib/nagios/plugins/zulip_postgres_common`).

docs/brief-install-vagrant-dev.md

@@ -0,0 +1,113 @@
+# Vagrant environment setup (in brief)
+
+Start by cloning this repository: `git clone https://github.com/zulip/zulip.git`
+
+This is the recommended approach for all platforms, and will install
+the Zulip development environment inside a VM or container and works
+on any platform that supports Vagrant.
+
+The best performing way to run the Zulip development environment is
+using an LXC container on a Linux host, but we support other platforms
+such as Mac via Virtualbox (but everything will be 2-3x slower).
+
+* If your host is Ubuntu 15.04 or newer, you can install and configure
+  the LXC Vagrant provider directly using apt:
+  ```
+  sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir
+  vagrant plugin install vagrant-lxc
+  ```
+  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
+
+* If your host is Ubuntu 14.04, you will need to [download a newer
+  version of Vagrant][vagrant-dl], and then do the following:
+  ```
+  sudo apt-get install lxc lxc-templates cgroup-lite redir
+  sudo dpkg -i vagrant*.deb # in directory where you downloaded vagrant
+  vagrant plugin install vagrant-lxc
+  ```
+  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
+
+* For other Linux hosts with a kernel above 3.12, [follow the Vagrant
+  LXC installation instructions][vagrant-lxc] to get Vagrant with LXC
+  for your platform.
+
+* If your host is OS X or older Linux, [download VirtualBox][vbox-dl],
+  [download Vagrant][vagrant-dl], and install them both.
+
+* If you're on OS X and have VMWare, it should be possible to patch
+  Vagrantfile to use the VMWare vagrant provider which should perform
+  much better than Virtualbox.  Patches to do this by default if
+  VMWare is available are welcome!
+
+* On Windows: You can use Vagrant and Virtualbox/VMWare on Windows
+  with Cygwin, similar to the Mac setup.  Be sure to create your git
+  clone using `git clone https://github.com/zulip/zulip.git -c
+  core.autocrlf=false` to avoid Windows line endings being added to
+  files (this causes weird errors).
+
+[vagrant-dl]: https://www.vagrantup.com/downloads.html
+[vagrant-lxc]: https://github.com/fgrehm/vagrant-lxc
+[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
+[avoiding-sudo]: https://github.com/fgrehm/vagrant-lxc#avoiding-sudo-passwords
+
+Once that's done, simply change to your zulip directory and run
+`vagrant up` in your terminal to install the development server.  This
+will take a long time on the first run because Vagrant needs to
+download the Ubuntu Trusty base image, but later you can run `vagrant
+destroy` and then `vagrant up` again to rebuild the environment and it
+will be much faster.
+
+Once that finishes, you can run the development server as follows:
+
+```
+vagrant ssh
+# Now inside the container
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+To get shell access to the virtual machine running the server to run
+lint, management commands, etc., use `vagrant ssh`.
+
+(A small note on tools/run-dev.py: the `--interface=''` option will
+make the development server listen on all network interfaces.  While
+this is correct for the Vagrant guest sitting behind a NAT, you
+probably don't want to use that option when using run-dev.py in other
+environments).
+
+At this point you should [read about using the development
+environment][using-dev].
+
+[using-dev]: using-dev-environment.html
+
+### Specifying a proxy
+
+If you need to use a proxy server to access the Internet, you will
+need to specify the proxy settings before running `Vagrant up`.
+First, install the Vagrant plugin `vagrant-proxyconf`:
+
+```
+vagrant plugin install vagrant-proxyconf.
+```
+
+Then create `~/.zulip-vagrant-config` and add the following lines to
+it (with the appropriate values in it for your proxy):
+
+```
+HTTP_PROXY http://proxy_host:port
+HTTPS_PROXY http://proxy_host:port
+NO_PROXY localhost,127.0.0.1,.example.com
+```
+
+Now run `vagrant up` in your terminal to install the development
+server. If you ran `vagrant up` before and failed, you'll need to run
+`vagrant destroy` first to clean up the failed installation.
+
+You can also change the port on the host machine that Vagrant uses by
+adding to your `~/.zulip-vagrant-config` file.  E.g. if you set:
+
+```
+HOST_PORT 9971
+```
+
+(and halt and restart the Vagrant guest), then you would visit
+http://localhost:9971/ to connect to your development server.

docs/changelog.md

@@ -0,0 +1,244 @@
+# Version History
+
+All notable changes to the Zulip server are documented in this file.
+
+### Unreleased
+
+### 1.4.3 - 2017-01-29
+- CVE-2017-0881: Users could subscribe to invite-only streams.
+
+### 1.4.2 - 2016-09-27
+- Upgraded Django to version 1.8.15 (with the Zulip patches applied),
+  fixing a CSRF vulnerability in Django (see
+  https://www.djangoproject.com/weblog/2016/sep/26/security-releases/),
+  and a number of other Django bugs from past Django stable releases
+  that largely affects parts of Django that are not used by Zulip.
+- Fixed buggy logrotate configuration.
+
+### 1.4.1 - 2016-09-03
+- Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0.
+- Fixed local file uploads integration being broken for new 1.4.0
+  installations.
+
+### 1.4 - 2016-08-25
+
+- Migrated Zulip's python dependencies to be installed via a virtualenv,
+  instead of the via apt.  This is a major change to how Zulip
+  is installed that we expect will simplify upgrades in the future.
+- Fixed unnecessary loading of zxcvbn password strength checker.  This
+  saves a huge fraction of the uncached network transfer for loading
+  Zulip.
+- Added support for using Ubuntu Xenial in production.
+- Added a powerful and complete realm import/export tool.
+- Added nice UI for selecting a default language to display settings.
+- Added UI for searching streams in left sidebar with hotkeys.
+- Added Semaphore, Bitbucket, and HelloWorld (example) integrations.
+- Added new webhook-based integration for Trello.
+- Added management command for creating realms through web UI.
+- Added management command to send password reset emails.
+- Added endpoint for mobile apps to query available auth backends.
+- Added LetsEncrypt documentation for getting SSL certificates.
+- Added nice rendering of unicode emoji.
+- Added support for pinning streams to the top of the left sidebar.
+- Added search box for filtering user list when creating a new stream.
+- Added realm setting to disable message editing.
+- Added realm setting to time-limit message editing.  Default is 10m.
+- Added realm setting for default language.
+- Added year to timestamps in message interstitials for old messages.
+- Added GitHub authentication (and integrated python-social-auth, so it's
+  easy to add additional social authentication methods).
+- Added TERMS_OF_SERVICE setting using markdown formatting to configure
+  the terms of service for a Zulip server.
+- Added numerous hooks to puppet modules to enable more configurations.
+- Moved several useful puppet components into the main puppet
+  manifests (setting a redis password, etc.).
+- Added automatic configuration of postgres/memcached settings based
+  on the server's available RAM.
+- Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo.
+- Added preliminary support for Python 3.  All of Zulip's test suites now
+  pass using Python 3.4.
+- Added support for `Name <email@example.com>` format when inviting users.
+- Added numerous special-purpose settings options.
+- Added a hex input field in color picker.
+- Documented new Electron beta app and mobile apps in the /apps/ page.
+- Enabled Android Google authentication support.
+- Enhanced logic for tracking origin of user uploads.
+- Improved error messages for various empty narrows.
+- Improved missed message emails to better support directly replying.
+- Increased backend test coverage of Python code to 85.5%.
+- Increased mypy static type coverage of Python code to 95%.  Also
+  fixed many string annotations to properly handle unicode.
+- Fixed major i18n-related frontend performance regression on
+  /#subscriptions page.  Saves several seconds of load time with 1k
+  streams.
+- Fixed Jinja2 migration bug when trying to register an email that
+  already has an account.
+- Fixed narrowing to a stream from other pages.
+- Fixed various frontend strings that weren't marked for translation.
+- Fixed several bugs around editing status (/me) messages.
+- Fixed queue workers not restarting after changes in development.
+- Fixed Casper tests hanging while development server is running.
+- Fixed browser autocomplete issue when adding new stream members.
+- Fixed broken create_stream and rename_stream management commands.
+- Fixed zulip-puppet-apply exit code when puppet throws errors.
+- Fixed EPMD restart being attempted on every puppet apply.
+- Fixed message cache filling; should improve perf after server restart.
+- Fixed caching race condition when changing user objects.
+- Fixed buggy puppet configuration for supervisord restarts.
+- Fixed some error handling race conditions when editing messages.
+- Fixed fastcgi_params to protect against the httpoxy attack.
+- Fixed bug preventing users with mit.edu emails from registering accounts.
+- Fixed incorrect settings docs for the email mirror.
+- Fixed APNS push notification support (had been broken by Apple changing
+  the APNS API).
+- Fixed some logic bugs in how attachments are tracked.
+- Fixed unnecessarily resource-intensive rabbitmq cron checks.
+- Fixed old deployment directories leaking indefinitely.
+- Fixed need to manually add localhost in ALLOWED_HOSTS.
+- Fixed display positioning for the color picker on subscriptions page.
+- Fixed escaping of Zulip extensions to markdown.
+- Fixed requiring a reload to see newly uploaded avatars.
+- Fixed @all warning firing even for `@all`.
+- Restyled password reset form to look nice.
+- Improved formatting in reset password links.
+- Improved alert words UI to match style of other settings.
+- Improved error experience when sending to nonexistent users.
+- Portions of integrations documentation are now automatically generated.
+- Restructured the URLs files to be more readable.
+- Upgraded almost all Python dependencies to current versions.
+- Substantially expanded and reorganized developer documentation.
+- Reorganized production documentation and moved to ReadTheDocs.
+- Reorganized .gitignore type files to be written under var/
+- Refactored substantial portions of templates to support subdomains.
+- Renamed local_settings.py symlink to prod_settings.py for clarity.
+- Renamed email-mirror management command to email_mirror.
+- Changed HTTP verb for create_user_backend to PUT.
+- Eliminated all remaining settings hardcoded for zulip.com.
+- Eliminated essentially all remaining hardcoding of mit.edu.
+- Optimized the performance of all the test suites.
+- Optimized Django memcached configuration.
+- Removed old prototype data export tool.
+- Disabled insecure RC4 cipher in nginx configuration.
+- Enabled shared SSL session cache in nginx configuration.
+- Updated header for Zulip static assets to reflect Zulip being
+  open source.
+
+### 1.3.13 - 2016-06-21
+- Added nearly complete internationalization of the Zulip UI.
+- Added warning when using @all/@everyone.
+- Added button offering to subscribe at bottom of narrows to streams
+  the user is not subscribed to.
+- Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT,
+  Transifex, and Updown.io.
+- Added menu option to mark all messages in a stream or topic as read.
+- Added new Attachment model to keep track of uploaded files.
+- Added caching of virtualenvs in development.
+- Added mypy static type annotations to about 85% of the Zulip Python codebase.
+- Added automated test of backend templates to test for regressions.
+- Added lots of detailed documentation on the Zulip development environment.
+- Added setting allowing only administrators to create new streams.
+- Added button to exit the Zulip tutorial early.
+- Added web UI for configuring default streams.
+- Added new OPEN_REALM_CREATION setting (default off), providing a UI
+  for creating additional realms on a Zulip server.
+- Fixed email_gateway_password secret not working properly.
+- Fixed missing helper scripts for RabbitMQ Nagios plugins.
+- Fixed skipping forward to latest messages ("More messages below" button).
+- Fixed netcat issue causing Zulip installation to hang on Scaleway machines.
+- Fixed rendering of /me status messages after message editing.
+- Fixed case sensitivity of right sidebar fading when compose is open.
+- Fixed error messages when composing to invalid PM recipients.
+- Fixed LDAP auth backend not working with Zulip mobile apps.
+- Fixed erroneous WWW-Authenticate headers with expired sessions.
+- Changed "coworkers" to "users" in the Zulip UI.
+- Changed add_default_stream REST API to correctly use PUT rather than PATCH.
+- Updated the Zulip emoji set (the Android Emoji) to a modern version.
+- Made numerous small improvements to the Zulip development experience.
+- Migrated backend templates to the faster Jinja2 templating system.
+- Migrated development environment setup scripts to tools/setup/.
+- Expanded test coverage for several areas of the product.
+- Simplified the API for writing new webhook integrations.
+- Removed most of the remaining JavaScript global variables.
+
+### 1.3.12 - 2016-05-10
+- CVE-2016-4426: Bot API keys were accessible to other users in the same realm.
+- CVE-2016-4427: Deactivated users could access messages if SSO was enabled.
+- Fixed a RabbitMQ configuration bug that resulted in reordered messages.
+- Added expansive test suite for authentication backends and decorators.
+- Added an option to logout_all_users to delete only sessions for deactivated users.
+
+### 1.3.11 - 2016-05-02
+- Moved email digest support into the default Zulip production configuration.
+- Added options for configuring Postgres, RabbitMQ, Redis, and memcached
+  in settings.py.
+- Added documentation on using Hubot to integrate with useful services
+  not yet integrated with Zulip directly (e.g. Google Hangouts).
+- Added new management command to test sending email from Zulip.
+- Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.
+- Added Nagios plugins to the main distribution.
+- Added ability for realm administrators to manage custom emoji.
+- Added guide to writing new integrations.
+- Enabled camo image proxy to fix mixed-content warnings for http images.
+- Refactored the Zulip puppet modules to be more modular.
+- Refactored the Tornado event system, fixing old memory leaks.
+- Removed many old-style /json API endpoints
+- Implemented running queue processors multithreaded in development,
+  decreasing RAM requirements for a Zulip development environment from
+  ~1GB to ~300MB.
+- Fixed rerendering the complete buddy list whenever a user came back from
+  idle, which was a significant performance issue in larger realms.
+- Fixed the disabling of desktop notifications from 1.3.7 for new users.
+- Fixed the (admin) create_user API enforcing restricted_to_domain, even
+  if that setting was disabled for the realm.
+- Fixed bugs changing certain settings in administration pages.
+- Fixed collapsing messages in narrowed views.
+- Fixed 500 errors when uploading a non-image file as an avatar.
+- Fixed Jira integration incorrectly not @-mentioning assignee.
+
+### 1.3.10 - 2016-01-21
+- Added new integration for Travis CI.
+- Added settings option to control maximum file upload size.
+- Added support for running Zulip development environment in Docker.
+- Added easy configuration support for a remote postgres database.
+- Added extensive documentation on scalability, backups, and security.
+- Recent private message threads are now displayed expanded similar to
+  the pre-existing recent topics feature.
+- Made it possible to set LDAP and EMAIL_HOST passwords in
+  /etc/zulip/secrets.conf.
+- Improved the styling for the Administration page and added tabs.
+- Substantially improved loading performance on slow networks by enabling
+  GZIP compression on more assets.
+- Changed the page title in narrowed views to include the current narrow.
+- Fixed several backend performance issues affecting very large realms.
+- Fixed bugs where draft compose content might be lost when reloading site.
+- Fixed support for disabling the "zulip" notifications stream.
+- Fixed missing step in postfix_localmail installation instructions.
+- Fixed several bugs/inconveniences in the production upgrade process.
+- Fixed realm restrictions for servers with a unique, open realm.
+- Substantially cleaned up console logging from run-dev.py.
+
+### 1.3.9 - 2015-11-16
+- Fixed buggy #! lines in upgrade scripts.
+
+### 1.3.8 - 2015-11-15
+- Added options to the Python api for working with untrusted server certificates.
+- Added a lot of documentation on the development environment and testing.
+- Added partial support for translating the Zulip UI.
+- Migrated installing Node dependencies to use npm.
+- Fixed LDAP integration breaking autocomplete of @-mentions.
+- Fixed admin panel reactivation/deactivation of bots.
+- Fixed inaccurate documentation for downloading the desktop apps.
+- Fixed various minor bugs in production installation process.
+- Fixed security issue where recent history on private streams might
+  be visible to new users (to the Zulip team) who were invited with that
+  private stream as one of their initial streams
+  (https://github.com/zulip/zulip/issues/230).
+- Major preliminary progress towards supporting Python 3.
+
+### 1.3.7 - 2015-10-19
+- Turn off desktop and audible notifications for streams by default.
+- Added support for the LDAP authentication integration creating new users.
+- Added new endpoint to support Google auth on mobile.
+- Fixed desktop notifications in modern Firefox.
+- Fixed several installation issues for both production and development environments.
+- Improved documentation for outgoing SMTP and the email mirror integration.

docs/code-style.md

@@ -0,0 +1,358 @@
+# Code style and conventions
+
+## Be consistent!
+
+Look at the surrounding code, or a similar part of the project, and try
+to do the same thing. If you think the other code has actively bad
+style, fix it (in a separate commit).
+
+When in doubt, send an email to <zulip-devel@googlegroups.com> with your
+question.
+
+## Lint tools
+
+You can run them all at once with
+
+    ./tools/lint-all
+
+You can set this up as a local Git commit hook with
+
+    ``tools/setup-git-repo``
+
+The Vagrant setup process runs this for you.
+
+`lint-all` runs many lint checks in parallel, including
+
+-   JavaScript ([JSLint](http://www.jslint.com/))
+
+    > `tools/jslint/check-all.js` contains a pretty fine-grained set of
+    > JSLint options, rule exceptions, and allowed global variables. If
+    > you add a new global, you'll need to add it to the list.
+
+-   Python ([Pyflakes](http://pypi.python.org/pypi/pyflakes))
+-   templates
+-   Puppet configuration
+-   custom checks (e.g. trailing whitespace and spaces-not-tabs)
+
+## Secrets
+
+Please don't put any passwords, secret access keys, etc. inline in the
+code. Instead, use the `get_secret` function in `zproject/settings.py`
+to read secrets from `/etc/zulip/secrets.conf`.
+
+## Dangerous constructs
+
+### Misuse of database queries
+
+Look out for Django code like this:
+
+    [Foo.objects.get(id=bar.x.id)
+    for bar in Bar.objects.filter(...)
+    if  bar.baz < 7]
+
+This will make one database query for each `Bar`, which is slow in
+production (but not in local testing!). Instead of a list comprehension,
+write a single query using Django's [QuerySet
+API](https://docs.djangoproject.com/en/dev/ref/models/querysets/).
+
+If you can't rewrite it as a single query, that's a sign that something
+is wrong with the database schema. So don't defer this optimization when
+performing schema changes, or else you may later find that it's
+impossible.
+
+### UserProfile.objects.get() / Client.objects.get / etc.
+
+In our Django code, never do direct `UserProfile.objects.get(email=foo)`
+database queries. Instead always use `get_user_profile_by_{email,id}`.
+There are 3 reasons for this:
+
+1.  It's guaranteed to correctly do a case-inexact lookup
+2.  It fetches the user object from remote cache, which is faster
+3.  It always fetches a UserProfile object which has been queried using
+    .selected\_related(), and thus will perform well when one later
+    accesses related models like the Realm.
+
+Similarly we have `get_client` and `get_stream` functions to fetch those
+commonly accessed objects via remote cache.
+
+### Using Django model objects as keys in sets/dicts
+
+Don't use Django model objects as keys in sets/dictionaries -- you will
+get unexpected behavior when dealing with objects obtained from
+different database queries:
+
+For example,
+`UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])`
+is False
+
+You should work with the IDs instead.
+
+### user\_profile.save()
+
+You should always pass the update\_fields keyword argument to .save()
+when modifying an existing Django model object. By default, .save() will
+overwrite every value in the column, which results in lots of race
+conditions where unrelated changes made by one thread can be
+accidentally overwritten by another thread that fetched its UserProfile
+object before the first thread wrote out its change.
+
+### Using raw saves to update important model objects
+
+In most cases, we already have a function in zephyr/lib/actions.py with
+a name like do\_activate\_user that will correctly handle lookups,
+caching, and notifying running browsers via the event system about your
+change. So please check whether such a function exists before writing
+new code to modify a model object, since your new code has a good chance
+of getting at least one of these things wrong.
+
+### `x.attr('zid')` vs. `rows.id(x)`
+
+Our message row DOM elements have a custom attribute `zid` which
+contains the numerical message ID. **Don't access this directly as**
+`x.attr('zid')` ! The result will be a string and comparisons (e.g. with
+`<=`) will give the wrong result, occasionally, just enough to make a
+bug that's impossible to track down.
+
+You should instead use the `id` function from the `rows` module, as in
+`rows.id(x)`. This returns a number. Even in cases where you do want a
+string, use the `id` function, as it will simplify future code changes.
+In most contexts in JavaScript where a string is needed, you can pass a
+number without any explicit conversion.
+
+### JavaScript var
+
+Always declare JavaScript variables using `var`:
+
+    var x = ...;
+
+In a function, `var` is necessary or else `x` will be a global variable.
+For variables declared at global scope, this has no effect, but we do it
+for consistency.
+
+JavaScript has function scope only, not block scope. This means that a
+`var` declaration inside a `for` or `if` acts the same as a `var`
+declaration at the beginning of the surrounding `function`. To avoid
+confusion, declare all variables at the top of a function.
+
+### JavaScript `for (i in myArray)`
+
+Don't use it:
+[[1]](http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays),
+[[2]](http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop),
+[[3]](http://www.jslint.com/lint.html#forin)
+
+### jQuery global state
+
+Don't mess with jQuery global state once the app has loaded. Code like
+this is very dangerous:
+
+    $.ajaxSetup({ async: false });
+    $.get(...);
+    $.ajaxSetup({ async: true });
+
+jQuery and the browser are free to run other code while the request is
+pending, which could perform other Ajax requests with the altered
+settings.
+
+Instead, switch to the more general `$.ajax`\_ function, which can take
+options like `async`.
+
+### State and logs files
+
+Do not write state and logs files inside the current working directory
+in the production environment. This will not how you expect, because the
+current working directory for the app changes every time we do a deploy.
+Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
+settings.py for an example.
+
+## JS array/object manipulation
+
+For generic functions that operate on arrays or JavaScript objects, you
+should generally use [Underscore](http://underscorejs.org/). We used to
+use jQuery's utility functions, but the Underscore equivalents are more
+consistent, better-behaved and offer more choices.
+
+A quick conversion table:
+
+       $.each → _.each (parameters to the callback reversed)
+       $.inArray → _.indexOf (parameters reversed)
+       $.grep → _.filter
+       $.map → _.map
+       $.extend → _.extend
+
+There's a subtle difference in the case of `_.extend`; it will replace
+attributes with undefined, whereas jQuery won't:
+
+       $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
+       _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
+
+Also, `_.each` does not let you break out of the iteration early by
+returning false, the way jQuery's version does. If you're doing this,
+you probably want `_.find`, `_.every`, or `_.any`, rather than 'each'.
+
+Some Underscore functions have multiple names. You should always use the
+canonical name (given in large print in the Underscore documentation),
+with the exception of `_.any`, which we prefer over the less clear
+'some'.
+
+## More arbitrary style things
+
+### General
+
+Indentation is four space characters for Python, JS, CSS, and shell
+scripts. Indentation is two space characters for HTML templates.
+
+We never use tabs anywhere in source code we write, but we have some
+third-party files which contain tabs.
+
+Keep third-party static files under the directory
+`zephyr/static/third/`, with one subdirectory per third-party project.
+
+We don't have an absolute hard limit on line length, but we should avoid
+extremely long lines. A general guideline is: refactor stuff to get it
+under 85 characters, unless that makes the code a lot uglier, in which
+case it's fine to go up to 120 or so.
+
+Whitespace guidelines:
+
+-   Put one space (or more for alignment) around binary arithmetic and
+    equality operators.
+-   Put one space around each part of the ternary operator.
+-   Put one space between keywords like `if` and `while` and their
+    associated open paren.
+-   Put one space between the closing paren for `if` and `while`-like
+    constructs and the opening curly brace. Put the curly brace on the
+    same line unless doing otherwise improves readability.
+-   Put no space before or after the open paren for function calls and
+    no space before the close paren for function calls.
+-   For the comma operator and colon operator in languages where it is
+    used for inline dictionaries, put no space before it and at least
+    one space after. Only use more than one space for alignment.
+
+### JavaScript
+
+Don't use `==` and `!=` because these operators perform type coercions,
+which can mask bugs. Always use `===` and `!==`.
+
+End every statement with a semicolon.
+
+`if` statements with no braces are allowed, if the body is simple and
+its extent is abundantly clear from context and formatting.
+
+Anonymous functions should have spaces before and after the argument
+list:
+
+    var x = function (foo, bar) { // ...
+
+When calling a function with an anonymous function as an argument, use
+this style:
+
+    $.get('foo', function (data) {
+        var x = ...;
+        // ...
+    });
+
+The inner function body is indented one level from the outer function
+call. The closing brace for the inner function and the closing
+parenthesis for the outer call are together on the same line. This style
+isn't necessarily appropriate for calls with multiple anonymous
+functions or other arguments following them.
+
+Use
+
+    $(function () { ...
+
+rather than
+
+    $(document).ready(function () { ...
+
+and combine adjacent on-ready functions, if they are logically related.
+
+The best way to build complicated DOM elements is a Mustache template
+like `zephyr/static/templates/message.handlebars`. For simpler things
+you can use jQuery DOM building APIs like so:
+
+    var new_tr = $('<tr />').attr('id', zephyr.id);
+
+Passing a HTML string to jQuery is fine for simple hardcoded things:
+
+    foo.append('<p id="selected">foo</p>');
+
+but avoid programmatically building complicated strings.
+
+We used to favor attaching behaviors in templates like so:
+
+    <p onclick="select_zephyr({{id}})">
+
+but there are some reasons to prefer attaching events using jQuery code:
+
+-   Potential huge performance gains by using delegated events where
+    possible
+-   When calling a function from an `onclick` attribute, `this` is not
+    bound to the element like you might think
+-   jQuery does event normalization
+
+Either way, avoid complicated JavaScript code inside HTML attributes;
+call a helper function instead.
+
+### HTML / CSS
+
+Don't use the `style=` attribute. Instead, define logical classes and
+put your styles in external files such as `zulip.css`.
+
+Don't use the tag name in a selector unless you have to. In other words,
+use `.foo` instead of `span.foo`. We shouldn't have to care if the tag
+type changes in the future.
+
+Don't use inline event handlers (`onclick=`, etc. attributes). Instead,
+attach a jQuery event handler
+(`$('#foo').on('click', function () {...})`) when the DOM is ready
+(inside a `$(function () {...})` block).
+
+Use this format when you have the same block applying to multiple CSS
+styles (separate lines for each selector):
+
+    selector1,
+    selector2 {
+    };
+
+### Python
+
+-   Scripts should start with `#!/usr/bin/env python` and not
+    `#/usr/bin/python` (the right Python may not be installed in
+    `/usr/bin`) or `#/usr/bin/env/python2.7` (bad for Python 3
+    compatibility).  Don't put a shebang line on a Python file unless
+    it's meaningful to run it as a script. (Some libraries can also be
+    run as scripts, e.g. to run a test suite.)
+-   The first import in a file should be
+    `from __future__ import absolute_import`, per [PEP
+    328](http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports)
+-   Put all imports together at the top of the file, absent a compelling
+    reason to do otherwise.
+-   Unpacking sequences doesn't require list brackets:
+
+        [x, y] = xs    # unnecessary
+        x, y = xs      # better
+
+-   For string formatting, use `x % (y,)` rather than `x % y`, to avoid
+    ambiguity if `y` happens to be a tuple.
+-   When selecting by id, don't use `foo.pk` when you mean `foo.id`.
+    E.g.
+
+        recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
+
+    should be written as
+
+        recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
+
+    in case we ever change the primary keys.
+
+### Tests
+
+All significant new features should come with tests. See testing.
+
+### Third party code
+
+When adding new third-party packages to our codebase, please include
+"[third]" at the beginning of the commit message. You don't necessarily
+need to do this when patching third-party code that's already in tree.

docs/code-style.rst

@@ -1,521 +0,0 @@
-==========================
-Code style and conventions
-==========================
-
-Be consistent!
-==============
-
-Look at the surrounding code, or a similar part of the project, and
-try to do the same thing. If you think the other code has actively bad
-style, fix it (in a separate commit).
-
-When in doubt, send an email to zulip-devel@googlegroups.com with your
-question.
-
-Lint tools
-==========
-
-You can run them all at once with
-
-::
-
-    ./tools/lint-all
-
-You can set this up as a local Git commit hook with
-
-::
-
-    ``tools/setup-git-repo``
-
-The Vagrant setup process runs this for you.
-
-``lint-all`` runs many lint checks in parallel, including
-
-- Javascript (`JSLint <http://www.jslint.com/>`__)
-
-    ``tools/jslint/check-all.js`` contains a pretty fine-grained set of
-    JSLint options, rule exceptions, and allowed global variables. If you
-    add a new global, you'll need to add it to the list.
-
-- Python (`Pyflakes <http://pypi.python.org/pypi/pyflakes>`__)
-- templates
-- Puppet configuration
-- custom checks (e.g. trailing whitespace and spaces-not-tabs)
-
-Secrets
-=======
-
-Please don't put any passwords, secret access keys, etc. inline in the
-code.  Instead, use the ``get_secret`` function in
-``zproject/settings.py`` to read secrets from ``/etc/zulip/secrets.conf``.
-
-Dangerous constructs
-====================
-
-Misuse of database queries
---------------------------
-
-Look out for Django code like this::
-
-   [Foo.objects.get(id=bar.x.id)
-   for bar in Bar.objects.filter(...)
-   if  bar.baz < 7]
-
-This will make one database query for each ``Bar``, which is slow in
-production (but not in local testing!). Instead of a list comprehension,
-write a single query using Django's `QuerySet
-API <https://docs.djangoproject.com/en/dev/ref/models/querysets/>`__.
-
-If you can't rewrite it as a single query, that's a sign that something
-is wrong with the database schema. So don't defer this optimization when
-performing schema changes, or else you may later find that it's
-impossible.
-
-UserProfile.objects.get() / Client.objects.get / etc.
------------------------------------------------------
-
-In our Django code, never do direct
-``UserProfile.objects.get(email=foo)`` database queries. Instead always
-use ``get_user_profile_by_{email,id}``. There are 3 reasons for this:
-
-#. It's guaranteed to correctly do a case-inexact lookup
-#. It fetches the user object from remote cache, which is faster
-#. It always fetches a UserProfile object which has been queried using
-   .selected\_related(), and thus will perform well when one later
-   accesses related models like the Realm.
-
-Similarly we have ``get_client`` and ``get_stream`` functions to fetch
-those commonly accessed objects via remote cache.
-
-Using Django model objects as keys in sets/dicts
-------------------------------------------------
-
-Don't use Django model objects as keys in sets/dictionaries -- you will
-get unexpected behavior when dealing with objects obtained from
-different database queries:
-
-For example,
-``UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])``
-is False
-
-You should work with the IDs instead.
-
-user\_profile.save()
---------------------
-
-You should always pass the update\_fields keyword argument to .save()
-when modifying an existing Django model object. By default, .save() will
-overwrite every value in the column, which results in lots of race
-conditions where unrelated changes made by one thread can be
-accidentally overwritten by another thread that fetched its UserProfile
-object before the first thread wrote out its change.
-
-Using raw saves to update important model objects
--------------------------------------------------
-
-In most cases, we already have a function in zephyr/lib/actions.py with
-a name like do\_activate\_user that will correctly handle lookups,
-caching, and notifying running browsers via the event system about your
-change. So please check whether such a function exists before writing
-new code to modify a model object, since your new code has a good chance
-of getting at least one of these things wrong.
-
-``x.attr('zid')`` vs. ``rows.id(x)``
-------------------------------------
-
-Our message row DOM elements have a custom attribute ``zid`` which
-contains the numerical message ID. **Don't access this directly as**
-``x.attr('zid')`` ! The result will be a string and comparisons (e.g.
-with ``<=``) will give the wrong result, occasionally, just enough to
-make a bug that's impossible to track down.
-
-You should instead use the ``id`` function from the ``rows`` module, as
-in ``rows.id(x)``. This returns a number. Even in cases where you do
-want a string, use the ``id`` function, as it will simplify future code
-changes. In most contexts in JavaScript where a string is needed, you
-can pass a number without any explicit conversion.
-
-Javascript var
---------------
-
-Always declare Javascript variables using ``var``::
-
-   var x = ...;
-
-In a function, ``var`` is necessary or else ``x`` will be a global
-variable. For variables declared at global scope, this has no effect,
-but we do it for consistency.
-
-Javascript has function scope only, not block scope. This means that a
-``var`` declaration inside a ``for`` or ``if`` acts the same as a
-``var`` declaration at the beginning of the surrounding ``function``. To
-avoid confusion, declare all variables at the top of a function.
-
-Javascript ``for (i in myArray)``
----------------------------------
-
-Don't use it:
-`[1] <http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays>`__,
-`[2] <http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml#for-in_loop>`__,
-`[3] <http://www.jslint.com/lint.html#forin>`__
-
-jQuery global state
--------------------
-
-Don't mess with jQuery global state once the app has loaded. Code like
-this is very dangerous::
-
-   $.ajaxSetup({ async: false });
-   $.get(...);
-   $.ajaxSetup({ async: true });
-
-jQuery and the browser are free to run other code while the request is
-pending, which could perform other Ajax requests with the altered
-settings.
-
-Instead, switch to the more general |ajax|_ function, which can take options
-like ``async``.
-
-.. |ajax| replace:: ``$.ajax``
-.. _ajax: http://api.jquery.com/jQuery.ajax
-
-State and logs files
---------------------
-
-Do not write state and logs files inside the current working directory
-in the production environment. This will not how you expect, because the
-current working directory for the app changes every time we do a deploy.
-Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
-settings.py for an example.
-
-JS array/object manipulation
-============================
-
-For generic functions that operate on arrays or JavaScript objects, you
-should generally use `Underscore <http://underscorejs.org/>`__. We used
-to use jQuery's utility functions, but the Underscore equivalents are
-more consistent, better-behaved and offer more choices.
-
-A quick conversion table::
-
-      $.each → _.each (parameters to the callback reversed)
-      $.inArray → _.indexOf (parameters reversed)
-      $.grep → _.filter
-      $.map → _.map
-      $.extend → _.extend
-
-There's a subtle difference in the case of ``_.extend``; it will replace
-attributes with undefined, whereas jQuery won't::
-
-      $.extend({foo: 2}, {foo: undefined});  // yields {foo: 2}, BUT...
-      _.extend({foo: 2}, {foo: undefined});  // yields {foo: undefined}!
-
-Also, ``_.each`` does not let you break out of the iteration early by
-returning false, the way jQuery's version does. If you're doing this,
-you probably want ``_.find``, ``_.every``, or ``_.any``, rather than
-'each'.
-
-Some Underscore functions have multiple names. You should always use the
-canonical name (given in large print in the Underscore documentation),
-with the exception of ``_.any``, which we prefer over the less clear
-'some'.
-
-More arbitrary style things
-===========================
-
-General
--------
-
-Indentation is four space characters for Python, JS, CSS, and shell
-scripts. Indentation is two space characters for HTML templates.
-
-We never use tabs anywhere in source code we write, but we have some
-third-party files which contain tabs.
-
-Keep third-party static files under the directory
-``zephyr/static/third/``, with one subdirectory per third-party project.
-
-We don't have an absolute hard limit on line length, but we should avoid
-extremely long lines. A general guideline is: refactor stuff to get it
-under 85 characters, unless that makes the code a lot uglier, in which
-case it's fine to go up to 120 or so.
-
-Whitespace guidelines:
-
--  Put one space (or more for alignment) around binary arithmetic and
-   equality operators.
--  Put one space around each part of the ternary operator.
--  Put one space between keywords like ``if`` and ``while`` and their
-   associated open paren.
--  Put one space between the closing paren for ``if`` and ``while``-like
-   constructs and the opening curly brace. Put the curly brace on the
-   same line unless doing otherwise improves readability.
--  Put no space before or after the open paren for function calls and no
-   space before the close paren for function calls.
--  For the comma operator and colon operator in languages where it is
-   used for inline dictionaries, put no space before it and at least one
-   space after. Only use more than one space for alignment.
-
-Javascript
-----------
-
-Don't use ``==`` and ``!=`` because these operators perform type
-coercions, which can mask bugs. Always use ``===`` and ``!==``.
-
-End every statement with a semicolon.
-
-``if`` statements with no braces are allowed, if the body is simple and
-its extent is abundantly clear from context and formatting.
-
-Anonymous functions should have spaces before and after the argument
-list::
-
-   var x = function (foo, bar) { // ...
-
-When calling a function with an anonymous function as an argument, use
-this style::
-
-   $.get('foo', function (data) {
-       var x = ...;
-       // ...
-   });
-
-The inner function body is indented one level from the outer function
-call. The closing brace for the inner function and the closing
-parenthesis for the outer call are together on the same line. This style
-isn't necessarily appropriate for calls with multiple anonymous
-functions or other arguments following them.
-
-Use
-
-::
-
-   $(function () { ...
-
-rather than
-
-::
-
-   $(document).ready(function () { ...
-
-and combine adjacent on-ready functions, if they are logically related.
-
-The best way to build complicated DOM elements is a Mustache template
-like ``zephyr/static/templates/message.handlebars``. For simpler things
-you can use jQuery DOM building APIs like so::
-
-   var new_tr = $('<tr />').attr('id', zephyr.id);
-
-Passing a HTML string to jQuery is fine for simple hardcoded things::
-
-   foo.append('<p id="selected">foo</p>');
-
-but avoid programmatically building complicated strings.
-
-We used to favor attaching behaviors in templates like so::
-
-    <p onclick="select_zephyr({{id}})">
-
-but there are some reasons to prefer attaching events using jQuery code:
-
--  Potential huge performance gains by using delegated events where
-   possible
--  When calling a function from an ``onclick`` attribute, ``this`` is
-   not bound to the element like you might think
--  jQuery does event normalization
-
-Either way, avoid complicated JavaScript code inside HTML attributes;
-call a helper function instead.
-
-HTML / CSS
-----------
-
-Don't use the ``style=`` attribute. Instead, define logical classes and
-put your styles in ``zulip.css``.
-
-Don't use the tag name in a selector unless you have to. In other words,
-use ``.foo`` instead of ``span.foo``. We shouldn't have to care if the
-tag type changes in the future.
-
-Don't use inline event handlers (``onclick=``, etc. attributes).
-Instead, attach a jQuery event handler
-(``$('#foo').on('click', function () {...})``) when the DOM is ready
-(inside a ``$(function () {...})`` block).
-
-Use this format when you have the same block applying to multiple CSS
-styles (separate lines for each selector)::
-
-    selector1,
-    selector2 {
-    };
-
-Python
-------
-
--  Scripts should start with ``#!/usr/bin/env python2.7`` and not
-   ``#!/usr/bin/env python2.7``. See commit ``437d4aee`` for an explanation of
-   why. Don't put such a line on a Python file unless it's meaningful to
-   run it as a script. (Some libraries can also be run as scripts, e.g.
-   to run a test suite.)
--  The first import in a file should be
-   ``from __future__ import absolute_import``, per `PEP
-   328 <http://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports>`__
--  Put all imports together at the top of the file, absent a compelling
-   reason to do otherwise.
--  Unpacking sequences doesn't require list brackets::
-
-      [x, y] = xs    # unnecessary
-      x, y = xs      # better
-
--  For string formatting, use ``x % (y,)`` rather than ``x % y``, to
-   avoid ambiguity if ``y`` happens to be a tuple.
--  When selecting by id, don't use ``foo.pk`` when you mean ``foo.id``.
-   E.g.
-
-   ::
-
-      recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
-
-   should be written as
-
-   ::
-
-      recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
-
-   in case we ever change the primary keys.
-
-Version Control
-===============
-
-Commit Discipline
------------------
-
-We follow the Git project's own commit discipline practice of "Each
-commit is a minimal coherent idea".  This discipline takes a bit of
-work, but it makes it much easier for code reviewers to spot bugs, and
-makesthe commit history a much more useful resource for developers
-trying to understand why the code works the way it does, which also
-helps a lot in preventing bugs.
-
-Coherency requirements for any commit:
-
--  It should pass tests (so test updates needed by a change should be in
-   the same commit as the original change, not a separate "fix the tests
-   that were broken by the last commit" commit).
--  It should be safe to deploy individually, or comment in detail in the
-   commit message as to why it isn't (maybe with a [manual] tag). So
-   implementing a new API endpoint in one commit and then adding the
-   security checks in a future commit should be avoided -- the security
-   checks should be there from the beginning.
--  Error handling should generally be included along with the code that
-   might trigger the error.
--  TODO comments should be in the commit that introduces the
-   issue or functionality with further work required.
-
-When you should be minimal:
-
--  Significant refactorings should be done in a separate commit from
-   functional changes.
--  Moving code from one file to another should be done in a separate
-   commits from functional changes or even refactoring within a file.
--  2 different refactorings should be done in different commits.
--  2 different features should be done in different commits.
--  If you find yourself writing a commit message that reads like a list
-   of somewhat dissimilar things that you did, you probably should have
-   just done 2 commits.
-
-When not to be overly minimal:
-
--  For completely new features, you don't necessarily need to split out
-   new commits for each little subfeature of the new feature. E.g. if
-   you're writing a new tool from scratch, it's fine to have the initial
-   tool have plenty of options/features without doing separate commits
-   for each one.  That said, reviewing a 2000-line giant blob of new
-   code isn't fun, so please be thoughtful about submitting things in
-   reviewable units.
--  Don't bother to split back end commits from front end commits, even
-   though the backend can often be coherent on its own.
-
-Other considerations:
-
--  Overly fine commits are easily squashed, but not vice versa, so err
-   toward small commits, and the code reviewer can advise on squashing.
--  If a commit you write doesn't pass tests, you should usually fix
-   that by amending the commit to fix the bug, not writing a new "fix
-   tests" commit on top of it.
--  When you fix a GitHub issue, `mark that you've fixed the issue in
-   your commit message
-   <https://help.github.com/articles/closing-issues-via-commit-messages/>`__
-   so that the issue is automatically closed when your code is merged.
-   Zulip's preferred style for this is to have the final paragraph
-   of the commit message read e.g. "Fixes: #123."
-
-Zulip expects you to structure the commits in your pull requests to
-form a clean history before we will merge them; it's best to write
-your commits following these guidelines in the first place, but if you
-don't, you can always fix your history using `git rebase -i`.
-
-It can take some practice to get used to writing your commits with a
-clean history so that you don't spend much time doing interactive
-rebases.  For example, often you'll start adding a feature, and
-discover you need to a refactoring partway through writing the
-feature.  When that happens, we recommend stashing your partial
-feature, do the refactoring, commit it, and then finish implementing
-your feature.
-
-Commit Messages
----------------
-
--  The first line of commit messages should be written in the imperative
-   and be kept relatively short while concisely explaining what the
-   commit does. For example:
-
-Bad::
-
-   bugfix
-   gather_subscriptions was broken
-   fix bug #234.
-
-Good::
-
-   Fix gather_subscriptions throwing an exception when given bad input.
-
--  Use present-tense action verbs in your commit messages.
-
-Bad::
-
-   Fixing gather_subscriptions throwing an exception when given bad input.
-   Fixed gather_subscriptions throwing an exception when given bad input.
-
-Good::
-
-   Fix gather_subscriptions throwing an exception when given bad input.
-
--  Please use a complete sentence in the summary, ending with a
-   period.
-
--  The rest of the commit message should be written in full prose and
-   explain why and how the change was made. If the commit makes
-   performance improvements, you should generally include some rough
-   benchmarks showing that it actually improves the performance.
-
--  Any paragraph content in the commit message should be line-wrapped
-   to less than 76 characters per line, so that your commit message
-   will be reasonably readable in `git log` in a normal terminal.
-
--  In your commit message, you should describe any manual testing you
-   did in addition to running the automated tests, and any aspects of
-   the commit that you think are questionable and you'd like special
-   attention applied to.
-
-Tests
------
-
-All significant new features should come with tests.
-
-Third party code
-----------------
-
-When adding new third-party packages to our codebase, please include
-"[third]" at the beginning of the commit message. You don't necessarily
-need to do this when patching third-party code that's already in tree.

docs/conf.py

@@ -15,6 +15,7 @@
 import sys
 import os
 import shlex
+if False: from typing import Any, Dict, List, Optional
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -29,16 +30,11 @@ import shlex
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = [] # type: List[str]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
 
@@ -47,7 +43,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Zulip'
-copyright = u'2015, The Zulip Team'
+copyright = u'2015-2016, The Zulip Team'
 author = u'The Zulip Team'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -55,16 +51,16 @@ author = u'The Zulip Team'
 # built documents.
 #
 # The short X.Y version.
-version = '0.1'
+version = '1.4'
 # The full version, including alpha/beta/rc tags.
-release = '0.1'
+release = '1.4.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = None # type: Optional[str]
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
@@ -210,6 +206,8 @@ html_static_path = ['_static']
 htmlhelp_basename = 'zulip-contributor-docsdoc'
 
 def setup(app):
+    # type: (Any) -> None
+
     # overrides for wide tables in RTD theme
     app.add_stylesheet('theme_overrides.css')   # path relative to _static
 
@@ -227,7 +225,7 @@ latex_elements = {
 
 # Latex figure (float) alignment
 #'figure_align': 'htbp',
-}
+} # type: Dict[str, str]
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
@@ -300,4 +298,6 @@ source_parsers = {
         '.md': CommonMarkParser,
         }
 
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
 source_suffix = ['.rst', '.md']

docs/conversion.md

@@ -0,0 +1,242 @@
+# Exporting data
+
+## Overview
+
+Occasionally Zulip administrators will need to move data from one
+server to another.
+
+There are many major operational aspects to doing a conversion. I will
+list them here, noting that several are not within the scope of this
+document:
+
+- Get new servers running.
+- Export data from the old DB.
+- Export files from S3.
+- Import files into new storage.
+- Import data into new DB.
+- Restart new servers.
+- Decommission old server.
+
+This document focuses almost entirely on the **export** piece.  Issues
+with getting Zulip itself running are totally out of scope here.  For the
+import side of things, I only touch on it implicity.  (My reasoning is
+that we *have* to get the export piece right in a timely fashion, even
+if it means we have to sort out some straggling issues on the import side
+later.)
+
+## Export
+
+We have tools that essentially export Zulip data to the file system.
+
+A good overview of the process is here:
+[management/export.py](https://github.com/zulip/zulip/blob/master/zerver/management/commands/export.py)
+
+This document supplements that explanation, but here we focus more
+on the logistics of a big conversion.  For some historical perspective,
+this document was originally drafted as part of a big Zulip cut-over.
+
+The main exporting tools in place as of summer 2016 are below:
+
+- We can export single realms (but not yet limit users within the realm).
+- We can export single users (but then we get no realm-wide data in the process).
+- We can run exports simultaneously (but have to navigate a bunch of /tmp directories).
+
+Things that we still may need:
+- We may want to export multiple realms simultaneously.
+- We may want to export multiple single users simultaneously.
+- We may want to limit users within realm exports.
+- We may want more operational robustness/convenience while doing several exports simultaenously.
+- We may want to merge multiple export files to remove duplicates.
+
+We have a few major classes of data.  They are listed below in the order
+that we process them in `do_export_realm()`:
+
+#### Public Realm Data
+
+Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream.
+
+#### Cross Realm Data
+
+Client/zerver_userprofile_cross_realm
+
+This includes Client and three bots.
+
+Client is unique in being a fairly core table that is
+not tied to UserProfile or Realm (unless you somewhat painfully tie
+it back to users in a bottom-up fashion though other tables).
+
+#### Disjoint User Data
+
+UserProfile/UserActivity/UserActivityInterval/UserPresence.
+
+#### Recipient Data
+
+Recipient/Stream/Subscription/Huddle.
+
+These tables are tied back to users, but they introduce complications
+when you try to deal with multi-user subsets.
+
+#### File-related Data
+
+Attachment
+
+This includes Attachment, and it referencs the avatar_source field of
+UserProfile.  Most importantly, of course, it requires us to grab files
+from S3.  Finally, Attachment's m2m relationship ties to Message.
+
+#### Message Data
+
+Message/UserMessage
+
+### Summary
+
+Here are the same classes of data, listed in roughly
+decreasing order of riskiness:
+
+- Message Data (sheer volume/lack of time/security)
+- File-Related Data (S3/security/lots of moving parts)
+- Recipient Data (complexity/security/cross-realm considerations)
+- Cross Realm Data (duplicate ids)
+- Disjoint User Data
+- Public Realm Data
+
+(Note the above list is essentially in reverse order of how we
+process the data, which isn't surprising for a top-down approach.)
+
+The next section of the document talks about risk factors.
+
+# Risk Mitigation
+
+## Generic considerations
+
+We have two major mechanisms for getting data:
+
+##### Top Down
+
+Get realm data, then all users in realm, then all recipients, then all messages, etc.
+
+The problem with the top down approach will be **filtering**.  Also, if
+errors arise during top-down passes, it may be time consuming to re-run
+the processes.
+
+##### Bottom Up
+
+Start with users, get their recipient data, etc.
+
+The problems with the bottom up approach will be **merging**.  Also, if
+we run multiple bottom-up passes, there is the danger of duplicating some
+work, particularly on the message side of things.
+
+### Approved Transfers
+
+We have not yet integrated the approved-transfer model, which tells us
+which users can be moved.
+
+## Risk factors broken out by data categories
+
+### Message Data
+
+- models: Message/UserMessage.
+- assets: messages-*.json, subprocesses, partial files
+
+Rows in the Message model depend on Recipient/UserProfile.
+
+Rows in the UserMessage model depend on UserProfile/Message.
+
+The biggest concern here is the **sheer volume** of data, with
+security being a close second.  (They are interrelated, as without
+security concerns, we could just bulk-export everything one time.)
+
+We currently have these measures in place for top-down processing:
+- chunking
+- multi-processing
+- messages are filtered by both sender and recipient
+
+
+### File Related Data
+
+- models: Attachment
+- assets: S3, attachment.json, uploads-temp/, image files in avatars/, assorted files in uploads/, avatars/records.json, uploads/records.json, zerver_attachment_messages
+
+When it comes to exporting attachment data, we have some minor volume issues, but the
+main concern is just that there are **lots of moving parts**:
+
+- S3 needs to be up, and we get some metadata from it as well as files.
+- We have security concerns about copying over only files that belong to users who approved the transfer.
+- This piece is just different in how we store data from all the other DB-centric pieces.
+- At import time we have to populate the m2m table (but fortunately, this is pretty low
+  risk in terms of breaking anything.)
+
+### Recipient Data
+- models: Recipient/Stream/Subscription/Huddle
+- assets: realm.json, (user,stream,huddle)_(recipient,subscription)
+
+This data is fortunately low to medium in volume.  The risk here will come
+from **model complexity** and **cross-realm concerns**.
+
+From the top down, here are the dependencies:
+
+- Recipient depends on UserProfile
+- Subscription depends on Recipient
+- Stream currently depends on Realm (but maybe it should be tied to Subscription)
+- Huddle depends on Subscription and UserProfile
+
+The biggest risk factor here is probably just the possibility that we could introduce
+some bug in our code as we try to segment Recipient into user, stream, and huddle components,
+especially if we try to handle multiple users or realms.
+I think this can be largely mitigated by the new Config approach.
+
+And then we also have some complicated Huddle logic that will be customized
+regardless.  The fiddliest part
+of the Huddle logic is creating the set of unsafe_huddle_recipient_ids.
+
+Last but not least, if we go with some hybrid of bottom-up and top-down, these tables
+are neither close to the bottom nor close to the top, so they may have the most
+fiddly edge cases when it comes to filtering and merging.
+
+Recommendation: We probably want to get a backup of all this data that is very simply
+bulk-exported from the entire DB, and we should obviously put it in a secure place.
+
+### Cross Realm Data
+- models: Client
+- assets: realm.json, three bots (notification/email/welcome), id_maps
+
+The good news here is that Client is a small table, and there are
+only three special bots.
+
+The bad news is that cross-realm data **complicates everything else**,
+and we have to avoid **database id conflicts**.
+
+If we use bottom-up approaches to load small user populations at a time, we may
+have **merging** issues here.  We will need to consolidate ids either by merging
+exports in /tmp or handle it import time.
+
+For the three bots, they live in zerver_userprofile_crossrealm, and we re-map
+their ids on the new server.
+
+Recommendation: Do not sweat the exports too much.  Deal with all the messiness at
+import time, and rely on the tables being really small.  We already have logic
+to catch Client.DoesNotExist exceptions, for example.  As for possibly missing
+messages that the welcome bot and friends have sent in the past, I am not sure
+what our risk profile is there, but I imagine it is relatively low.
+
+### Disjoint User Data
+- models: UserProfile/UserActivity/UserActivityInterval/UserPresence
+- assets: realm.json, password, api_key, avatar salt, id_maps
+
+On the DB side this data should be fairly easy to deal with.  All of these
+tables are basically disjoint by user profile id.  Our biggest
+risk is **remapped user ids** at import time, but this is mostly covered
+in the section above.
+
+We have code in place to exclude password and api_key from UserProfile
+rows.  The import process calls set_unusable_password().
+
+### Public Realm Data
+
+- models: Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream
+- asserts: realm.json
+
+All of these tables are public (per-realm), and they are keyed by
+realm id.  There is not a ton to worry about here, except possibly
+**merging** if we run multiple bottom-up jobs for a single realm.

docs/dev-env-first-time-contributors.md

@@ -0,0 +1,697 @@
+## Vagrant environment setup tutorial
+
+This section guides first-time contributors through installing the Zulip dev
+environment on Windows 10, OS X El Capitan, Ubuntu 14.04, and Ubuntu 16.04.
+
+The recommended method for installing the Zulip dev environment is to use
+Vagrant with VirtualBox on Windows and OS X, and Vagrant with LXC on
+Ubuntu. This method creates a virtual machine (for Windows and OS X)
+or a Linux container (for Ubuntu) inside which the Zulip server and
+all related services will run.
+
+Contents:
+* [Requirements](#requirements)
+* [Step 1: Install Prerequisites](#step-1-install-prerequisites)
+* [Step 2: Get Zulip code](#step-2-get-zulip-code)
+* [Step 3: Start the dev environment](#step-3-start-the-dev-environment)
+* [Step 4: Developing](#step-4-developing)
+* [Troubleshooting & Common Errors](#troubleshooting-common-errors)
+
+If you encounter errors installing the Zulip development environment,
+check [Troubleshooting & Common
+Errors](#troubleshooting-common-errors). If that doesn't help, please
+visit [the `provision` stream in the Zulip developers'
+chat](https://zulip.tabbott.net/#narrow/stream/provision) for realtime
+help, or send a note to the [Zulip-devel Google
+group](https://groups.google.com/forum/#!forum/zulip-devel) or [file
+an issue](https://github.com/zulip/zulip/issues).
+
+### Requirements
+
+Installing the Zulip dev environment requires downloading several
+hundred megabytes of dependencies. You will need an active internet
+connection throughout the entire installation processes. (See
+[Specifying a
+proxy](brief-install-vagrant-dev.html#specifying-a-proxy) if you need
+a proxy to access the internet.)
+
+
+- **All**: 2GB available RAM, Active broadband internet connection.
+- **OS X**: OS X (El Capitan recommended, untested on previous versions), Git,
+  [VirtualBox][vbox-dl], [Vagrant][vagrant-dl].
+- **Ubuntu**: 14.04 64-bit or 16.04 64-bit, Git, [Vagrant][vagrant-dl], lxc.
+- **Windows**: Windows 64-bit (Win 10 recommended; Win 7 untested), hardware
+  virtualization enabled (VT-X or AMD-V), administrator access,
+  [Cygwin][cygwin-dl], [VirtualBox][vbox-dl], [Vagrant][vagrant-dl].
+
+Don't see your system listed above? Check out:
+* [Brief installation instructions for Vagrant development
+  environment](brief-install-vagrant-dev.html)
+* [Installing manually on UNIX-based platforms](install-generic-unix-dev.html)
+
+[cygwin-dl]: http://cygwin.com/
+
+### Step 1: Install Prerequisites
+
+Jump to:
+
+* [OS X](#os-x)
+* [Ubuntu](#ubuntu)
+* [Windows](#windows-10)
+
+#### OS X
+
+1. Install [VirtualBox][vbox-dl]
+2. Install [Vagrant][vagrant-dl]
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+#### Ubuntu
+
+The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same.
+
+If you're in a hurry, you can copy and paste the following into your terminal
+after which you can jump to [Step 2: Get Zulip Code](#step-2-get-zulip-code):
+
+```
+sudo apt-get purge vagrant
+wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb
+sudo dpkg -i vagrant*.deb
+sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
+vagrant plugin install vagrant-lxc
+vagrant lxc sudoers
+```
+
+For a step-by-step explanation, read on.
+
+##### 1. Install Vagrant
+
+For both 14.04 Trusty and 16.04 Xenial, you'll need a more recent version of
+Vagrant than what's available in the official Ubuntu repositories.
+
+First uninstall any vagrant package you may have installed from the Ubuntu
+repository:
+
+```
+christie@ubuntu-desktop:~
+$ sudo apt-get purge vagrant
+```
+
+Now download and install the most recent .deb package from [Vagrant][vagrant-dl]:
+
+```
+christie@ubuntu-desktop:~
+$ wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb
+
+christie@ubuntu-desktop:~
+$ sudo dpkg -i vagrant*.deb
+```
+
+##### 2. Install remaining dependencies
+
+Now install git and lxc-related packages:
+
+```
+christie@ubuntu-desktop:~
+$ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
+```
+
+##### 3. Install the vagrant lxc plugin:
+
+```
+christie@ubuntu-desktop:~
+$ vagrant plugin install vagrant-lxc
+Installing the 'vagrant-lxc' plugin. This can take a few minutes...
+Installed the plugin 'vagrant-lxc (1.2.1)'!
+```
+
+If you encounter an error when trying to install the vagrant-lxc plugin, [see
+this](#nomethoderror-when-installing-vagrant-lxc-plugin-ubuntu-1604).
+
+##### 4. Configure sudo to be passwordless
+
+Finally, [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo]:
+
+```
+christie@ubuntu-desktop:~
+$ vagrant lxc sudoers
+[sudo] password for christie:
+```
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+[vagrant-dl]: https://www.vagrantup.com/downloads.html
+[vagrant-lxc]: https://github.com/fgrehm/vagrant-lxc
+[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
+[avoiding-sudo]: https://github.com/fgrehm/vagrant-lxc#avoiding-sudo-passwords
+
+#### Windows 10
+
+1. Install [Cygwin][cygwin-dl]. Make sure to install default required
+   packages along with **git**, **curl**, **openssh**, and **rsync**
+   binaries.
+2. Install [VirtualBox][vbox-dl]
+3. Install [Vagrant][vagrant-dl]
+
+##### Configure Cygwin
+
+In order for symlinks to work within the Ubuntu virtual machine, you must tell
+Cygwin to create them as [native Windows
+symlinks](https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks). The
+easiest way to do this is to add a line to `~/.bash_profile` setting the CYGWIN
+environment variable.
+
+Open a Cygwin window and do this:
+
+```
+christie@win10 ~
+$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
+```
+
+Next, close that Cygwin window and open another. If you `echo` $CYGWIN you
+should see:
+
+```
+christie@win10 ~
+$ echo $CYGWIN
+winsymlinks:native
+```
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+### Step 2: Get Zulip Code
+
+If you haven't already created an ssh key and added it to your Github account,
+you should do that now by following [these
+instructions](https://help.github.com/articles/generating-an-ssh-key/).
+
+1. In your browser, visit https://github.com/zulip/zulip and click the
+   `fork` button. You will need to be logged in to Github to do this.
+2. Open Terminal (OS X/Ubuntu) or Cygwin (Windows; must run as an Administrator)
+3. In Terminal/Cygwin, clone your fork:
+```
+git clone git@github.com:YOURUSERNAME/zulip.git
+```
+
+This will create a 'zulip' directory and download the Zulip code into it.
+
+Don't forget to replace YOURUSERNAME with your git username. You will see
+something like:
+
+```
+christie@win10 ~
+$ git clone git@github.com:YOURUSERNAME/zulip.git
+Cloning into 'zulip'...
+remote: Counting objects: 73571, done.
+remote: Compressing objects: 100% (2/2), done.
+remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569
+Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done.
+Resolving deltas: 100% (51448/51448), done.
+Checking connectivity... done.
+Checking out files: 100% (1912/1912), done.`
+```
+
+Now you are ready for [Step 3: Start the dev
+environment.](#step-3-start-the-dev-environment)
+
+### Step 3: Start the dev environment
+
+Change into the zulip directory and tell vagrant to start the Zulip
+dev environment with `vagrant up`.
+
+```
+christie@win10 ~
+$ cd zulip
+
+christie@win10 ~/zulip
+$ vagrant up
+```
+
+The first time you run this command it will take some time because vagrant
+does the following:
+
+- downloads the base Ubuntu 14.04 virtual machine image (for OS X and Windows)
+  or container (for Ubuntu)
+- configures this virtual machine/container for use with Zulip,
+- creates a shared directory mapping your clone of the Zulip code inside the
+  virtual machine/container at `/srv/zulip`
+- runs the `tools/provision.py` script inside the virtual machine/container, which
+  downloads all required dependencies, sets up the python environment for
+  the Zulip dev environment, and initializes a default test database.
+
+You will need an active internet connection during the entire
+processes. (See [Specifying a
+proxy](brief-install-vagrant-dev.html#specifying-a-proxy) if you need
+a proxy to access the internet.) And if you're running into any
+problems, please come chat with us [in the `provision` stream of our
+developers' chat](https://zulip.tabbott.net/#narrow/stream/provision).
+
+Once `vagrant up` has completed, connect to the dev environment with `vagrant
+ssh`:
+
+```
+christie@win10 ~/zulip
+$ vagrant ssh
+```
+
+You should see something like this on Windows and OS X:
+
+```
+Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com/
+
+  System information as of Wed May  4 21:45:43 UTC 2016
+
+  System load:  0.61              Processes:           88
+  Usage of /:   3.5% of 39.34GB   Users logged in:     0
+  Memory usage: 7%                IP address for eth0: 10.0.2.15
+  Swap usage:   0%
+
+  Graph this data and manage this system at:
+    https://landscape.canonical.com/
+
+  Get cloud support with Ubuntu Advantage Cloud Guest:
+    http://www.ubuntu.com/business/services/cloud
+
+0 packages can be updated.
+0 updates are security updates.
+```
+
+Or something as brief as this in the case of Ubuntu:
+
+```
+Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com/
+```
+
+Congrats, you're now inside the Zulip dev environment!
+
+You can confirm this by looking at the command prompt, which starts with
+`(zulip-venv)`.
+
+Next, start the Zulip server:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+As you can see above the application's root directory, where you can
+execute Django's command line utilities is:
+
+```
+/srv/zulip/
+```
+
+You will see several lines of output starting with something like:
+
+```
+2016-05-04 22:20:33,895 INFO: process_fts_updates starting
+Recompiling templates
+2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates
+done
+Validating Django models.py...
+System check identified no issues (0 silenced).
+
+Django version 1.8
+Tornado server is running at http://localhost:9993/
+Quit the server with CTRL-C.
+2016-05-04 18:20:40,716 INFO     Tornado loaded 0 event queues in 0.001s
+2016-05-04 18:20:40,722 INFO     Tornado  95.5% busy over the past  0.0 seconds
+Performing system checks...
+```
+And ending with something similar to:
+
+```
+http://localhost:9994/webpack-dev-server/
+webpack result is served from http://localhost:9991/webpack/
+content is served from /srv/zulip
+
+webpack: bundle is now VALID.
+2016-05-06 21:43:29,553 INFO     Tornado  31.6% busy over the past 10.6 seconds
+2016-05-06 21:43:35,007 INFO     Tornado  23.9% busy over the past 16.0 seconds
+```
+
+Now the Zulip server should be running and accessible. Verify this by
+navigating to [http://localhost:9991/](http://localhost:9991/) in your browser
+on your main machine.
+
+You should see something like [(this screenshot of the Zulip dev
+environment)](https://raw.githubusercontent.com/zulip/zulip/master/docs/images/zulip-dev.png).
+
+![Image of Zulip dev environment](https://raw.githubusercontent.com/zulip/zulip/master/docs/images/zulip-dev.png)
+
+The Zulip server will continue to run and send output to the terminal window.
+When you navigate to Zulip in your browser, check your terminal and you
+should see something like:
+
+```
+2016-05-04 18:21:57,547 INFO     127.0.0.1       GET     302 582ms (+start: 417ms) / (unauth via ?)
+[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0
+2016-05-04 18:21:57,568 INFO     127.0.0.1       GET     301   4ms /login (unauth via ?)
+[04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0
+2016-05-04 18:21:57,819 INFO     127.0.0.1       GET     200 209ms (db: 7ms/2q) /login/ (unauth via ?)
+```
+
+Now you're ready for [Step 4: Developing.](#step-4-developing)
+
+### Step 4: Developing
+
+#### Where to edit files
+
+You'll work by editing files on your host machine, in the directory where you
+cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++,
+etc.).
+
+When you save changes they will be synced automatically to the Zulip dev environment
+on the virtual machine/container.
+
+Each component of the Zulip development server will automatically
+restart itself or reload data appropriately when you make changes. So,
+to see your changes, all you usually have to do is reload your
+browser.  More details on how this works are available below.
+
+Don't forget to read through the [code style
+guidelines](https://zulip.readthedocs.io/en/latest/code-style.html#general) for
+details about how to configure your editor for Zulip. For example, indentation
+should be set to 4 spaces rather than tabs.
+
+#### Understanding run-dev.py debugging output
+
+It's good to have the terminal running `run-dev.py` up as you work since error
+messages including tracebacks along with every backend request will be printed
+there.
+
+See [Logging](http://zulip.readthedocs.io/en/latest/logging.html) for
+further details on the run-dev.py console output.
+
+#### Committing and pushing changes with git
+
+When you're ready to commit or push changes via git, you will do this by
+running git commands in Terminal (OS X/Ubuntu) or Cygwin (Windows) in the directory
+where you cloned Zulip on your main machine.
+
+If you're new to working with Git/Github, check out [this
+guide](https://help.github.com/articles/create-a-repo/#commit-your-first-change).
+
+#### Maintaining the dev environment
+
+If after rebasing onto a new version of the Zulip server, you receive
+new errors while starting the Zulip server or running tests, this is
+probably not because Zulip's master branch is broken.  Instead, this
+is likely because we've recently merged changes to the development
+environment provisioning process that you need to apply to your
+development environmnet.  To update your environment, you'll need to
+re-provision your vagrant machine using `vagrant provision`
+(or just `python tools/provision.py` from `/srv/zulip` inside the Vagrant
+guest); this should be pretty fast and we're working to make it faster.
+
+See also the documentation on the [testing
+page](http://zulip.readthedocs.io/en/latest/testing.html#manual-testing-local-app-web-browser)
+for how to destroy and rebuild your database if you want to clear out test data.
+
+#### Rebuilding the dev environment
+
+If you ever want to recreate your development environment again from
+scratch (e.g. to test as change you've made to the provisioning
+process, or because you think something is broken), you can do so
+using `vagrant destroy` and then `vagrant up`.  This will usually be
+much faster than the original `vagrant up` since the base image is
+already cached on your machine (it takes about 5 minutes to run with a
+fast Internet connection).
+
+#### Shutting down the dev environment for use later
+
+To shut down but preserve the dev environment so you can use it again
+later use `vagrant halt` or `vagrant suspend`.
+
+You can do this from the same Terminal/Cygwin window that is running
+run-dev.py by pressing ^C to halt the server and then typing `exit`. Or you
+can halt vagrant from another Terminal/Cygwin window.
+
+From the window where run-dev.py is running:
+
+```
+2016-05-04 18:33:13,330 INFO     127.0.0.1       GET     200  92ms /register/ (unauth via ?)
+^C
+KeyboardInterrupt
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ exit
+logout
+Connection to 127.0.0.1 closed.
+christie@win10 ~/zulip
+```
+Now you can suspend the dev environment:
+
+```
+christie@win10 ~/zulip
+$ vagrant suspend
+==> default: Saving VM state and suspending execution...
+```
+
+If `vagrant suspend` doesn't work, try `vagrant halt`:
+
+```
+christie@win10 ~/zulip
+$ vagrant halt
+==> default: Attempting graceful shutdown of VM...
+```
+
+Check out the Vagrant documentation to learn more about
+[suspend](https://www.vagrantup.com/docs/cli/suspend.html) and
+[halt](https://www.vagrantup.com/docs/cli/halt.html).
+
+#### Resuming the dev environment
+
+When you're ready to work on Zulip again, run `vagrant up`. You will also need
+to connect to the virtual machine with `vagrant ssh` and re-start the Zulip
+server:
+
+```
+christie@win10 ~/zulip
+$ vagrant up
+$ vagrant ssh
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+#### Next Steps
+
+At this point you should [read about using the development
+environment][using-dev-environment.html].
+
+### Troubleshooting & Common Errors
+
+Zulip's `vagrant` provisioning process logs useful debugging output to
+`/var/log/zulip_provision.log`; if you encounter a new issue, please
+attach a copy of that file to your bug report.
+
+#### The box 'ubuntu/trusty64' could not be found (Windows/Cygwin)
+
+If you see the following error when you run `vagrant up` on Windows:
+
+```
+The box 'ubuntu/trusty64' could not be found or
+could not be accessed in the remote catalog. If this is a private
+box on HashiCorp's Atlas, please verify you're logged in via
+`vagrant login`. Also, please double-check the name. The expanded
+URL and error message are shown below:
+URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
+```
+
+Then the version of curl that ships with Vagrant is not working on your
+machine. The fix is simple: replace it with the version from Cygwin.
+
+First, determine the location of Cygwin's curl with `which curl`:
+
+```
+christie@win10 ~/zulip
+$ which curl
+/usr/bin/curl
+```
+Now determine the location of Vagrant with `which vagrant`:
+```
+christie@win10 ~/zulip
+$ which vagrant
+/cygdrive/c/HashiCorp/Vagrant/bin/vagrant
+```
+The path **up until `/bin/vagrant`** is what you need to know. In the example above it's `/cygdrive/c/HashiCorp/Vagrant`.
+
+Finally, copy Cygwin's curl to Vagrant `embedded/bin` directory:
+```
+christie@win10 ~/zulip
+$ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
+```
+
+Now re-run `vagrant up` and vagrant should be able to fetch the required
+box file.
+
+#### os.symlink error
+
+If you receive the following error while running `vagrant up`:
+
+```
+==> default: Traceback (most recent call last):
+==> default: File "./emoji_dump.py", line 75, in <module>
+==> default:
+==> default: os.symlink('unicode/{}.png'.format(code_point), 'out/{}.png'.format(name))
+==> default: OSError
+==> default: :
+==> default: [Errno 71] Protocol error
+```
+
+Then Vagrant was not able to create a symbolic link.
+
+First, if you are using Windows, **make sure you have run Cygwin as an
+administrator**. By default, only administrators can create symbolic links on
+Windows.
+
+Second, VirtualBox does not enable symbolic links by default. Vagrant
+starting with version 1.6.0 enables symbolic links for VirtualBox shared
+folder.
+
+You can check to see that this is enabled for your virtual machine with
+`vboxmanage` command.
+
+Get the name of your virtual machine by running `vboxmanage list vms` and
+then print out the custom settings for this virtual machine with
+`vboxmanage getextradata YOURVMNAME enumerate`:
+
+```
+christie@win10 ~/zulip
+$ vboxmanage list vms
+"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
+
+christie@win10 ~/zulip
+$ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate
+Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1
+Key: supported, Value: false
+```
+
+If you see "command not found" when you try to run VBoxManage, you need to
+add the VirtualBox directory to your path. On Windows this is mostly likely
+`C:\Program Files\Oracle\VirtualBox\`.
+
+If `vboxmanage enumerate` prints nothing, or shows a value of 0 for
+VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable
+symbolic links by running this command in Terminal/Cygwin:
+
+```
+vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip 1
+```
+
+The virtual machine needs to be shut down when you run this command.
+
+#### Connection timeout on `vagrant up`
+
+If you see the following error after running `vagrant up`:
+
+```
+default: SSH address: 127.0.0.1:2222
+default: SSH username: vagrant
+default: SSH auth method: private key
+default: Error: Connection timeout. Retrying...
+default: Error: Connection timeout. Retrying...
+default: Error: Connection timeout. Retrying...
+
+```
+A likely cause is that hardware virtualization is not enabled for your
+computer. This must be done via your computer's BIOS settings. Look for a
+setting called VT-x (Intel) or (AMD-V).
+
+If this is already enabled in your BIOS, double-check that you are running a
+64-bit operating system.
+
+For further information about troubleshooting vagrant timeout errors [see
+this post](http://stackoverflow.com/questions/22575261/vagrant-stuck-connection-timeout-retrying#22575302).
+
+#### npm install error
+
+The `tools/provision.py` script may encounter an error related to `npm install`
+that looks something like:
+
+```
+==> default: + npm install
+==> default: Traceback (most recent call last):
+==> default:   File "/srv/zulip/tools/provision.py", line 195, in <module>
+==> default:
+==> default: sys.exit(main())
+==> default:   File "/srv/zulip/tools/provision.py", line 191, in main
+==> default:
+==> default: run(["npm", "install"])
+==> default:   File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run
+==> default:
+==> default: raise subprocess.CalledProcessError(rc, args)
+==> default: subprocess
+==> default: .
+==> default: CalledProcessError
+==> default: :
+==> default: Command '['npm', 'install']' returned non-zero exit status 34
+The SSH command responded with a non-zero exit status. Vagrant
+assumes that this means the command failed. The output for this command
+should be in the log above. Please read the output to determine what
+went wrong.
+```
+
+Usually this error is not fatal. Try connecting to the dev environment and
+re-trying the command from withing the virtual machine:
+
+```
+christie@win10 ~/zulip
+$ vagrant ssh
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
+$ cd /srv/zulip
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip
+$ npm install
+npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
+npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.12
+```
+
+These are just warnings so it is okay to proceed and start the Zulip server.
+
+#### NoMethodError when installing vagrant-lxc plugin (Ubuntu 16.04)
+
+If you see the following error when you try to install the vagrant-lxc plugin:
+
+```
+/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_by' for nil:NilClass (NoMethodError)
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem'
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install'
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install'
+  from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_plugin'
+  from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin'
+  from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run'
+  from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run'
+  from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in execute'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute'
+  from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute'
+  from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute'
+  from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli'
+  from /usr/bin/vagrant:173:in `<main>'
+```
+
+And you have vagrant version 1.8.1, then you need to patch vagrant manually.
+See [this post](https://github.com/mitchellh/vagrant/issues/7073) for an
+explanation of the issue, which should be fixed when Vagrant 1.8.2 is released.
+
+In the meantime, read [this
+post](http://stackoverflow.com/questions/36811863/cant-install-vagrant-plugins-in-ubuntu-16-04/36991648#36991648)
+for how to create and apply the patch.
+
+It will look something like this:
+
+```
+christie@xenial:~
+$ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch
+patching file bundler.rb
+```
+
+#### Permissions errors when running the test suite in LXC
+
+See ["Possible testing issues"](testing.html#possible-testing-issues).

docs/dev-overview.md

@@ -0,0 +1,17 @@
+# Development environment options
+
+Zulip offers a wide range of options for how to install the
+development environment:
+
+* [Detailed tutorial for Vagrant development environment](dev-env-first-time-contributors.html).  Recommended for first-time contributors.
+* [Brief installation instructions for Vagrant development environment](brief-install-vagrant-dev.html)
+* [Installing on Ubuntu 14.04 Trusty or 16.04 Xenial directly](install-ubuntu-without-vagrant-dev.html) (convenient but more work to maintain/uninstall).
+* [Installing manually on other UNIX platforms](install-generic-unix-dev.html)
+* [Using Docker (experimental)](install-docker-dev.html)
+* [Using the Development Environment](using-dev-environment.html)
+* [Testing](testing.html)
+
+If you have a slow network connection, you should probably avoid
+installing Vagrant (which is large) and either install
+[directly](install-ubuntu-without-vagrant-dev.html) or use [the manual
+install process](install-generic-unix-dev.html) instead.

docs/directory-structure.md

@@ -0,0 +1,180 @@
+# Directory structure
+
+This page documents the Zulip directory structure, where to find
+things, and how to decide where to put a file.
+
+You may also find the [new application feature
+tutorial](new-feature-tutorial.html) helpful for understanding the
+flow through these files.
+
+### Core Python files
+
+Zulip uses the [Django web
+framework](https://docs.djangoproject.com/en/1.8/), so a lot of these
+paths will be familiar to Django developers.
+
+* `zproject/urls.py` Main [Django routes file](https://docs.djangoproject.com/en/1.8/topics/http/urls/).  Defines which URLs are handled by which view functions or templates.
+
+* `zerver/models.py` Main [Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/) file.  Defines Zulip's database tables.
+
+* `zerver/lib/actions.py` Most code doing writes to user-facing database tables.
+
+* `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/).
+
+* `zerver/views/webhooks/` Webhook views for [Zulip integrations](integration-guide.html).
+
+* `zerver/tornadoviews.py` Tornado views.
+
+* `zerver/worker/queue_processors.py` [Queue workers](queuing.html).
+
+* `zerver/lib/*.py` Most library code.
+
+* `zerver/lib/bugdown/` [Backend Markdown processor](markdown.html).
+
+* `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/).
+
+-------------------------------------------------------------------
+
+### HTML Templates
+
+See [our translating docs](translating.html) for details on Zulip's
+templating systems.
+
+* `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates for the backend (for zerver app).
+
+* `static/templates/` [Handlebars](http://handlebarsjs.com/) templates for the frontend.
+
+----------------------------------------
+
+### JavaScript and other static assets
+
+* `static/js/` Zulip's own JavaScript.
+
+* `static/styles/` Zulip's own CSS.
+
+* `static/images/` Zulip's images.
+
+* `static/third/` Third-party JavaScript and CSS that has been vendored.
+
+* `node_modules/` Third-party JavaScript installed via `npm`.
+
+* `assets/` For assets not to be served to the web (e.g. the system to
+            generate our favicons).
+
+-----------------------------------------------------------------------
+
+### Tests
+
+* `zerver/tests/` Backend tests.
+
+* `frontend_tests/node_tests/` Node Frontend unit tests.
+
+* `frontend_tests/casper_tests/` Casper frontend tests.
+
+* `tools/test-*` Developer-facing test runner scripts.
+
+-----------------------------------------------------
+
+### Management commands
+
+These are distinguished from scripts, below, by needing to run a
+Django context (i.e. with database access).
+
+* `zerver/management/commands/` Management commands one might run at a
+  production deployment site (e.g. scripts to change a value or
+  deactivate a user properly).
+
+---------------------------------------------------------------
+
+### Scripts
+
+* `scripts/` Scripts that production deployments might run manually
+  (e.g., `restart-server`).
+
+* `scripts/lib/` Scripts that are needed on production deployments but
+  humans should never run directly.
+
+* `scripts/setup/` Scripts that production deployments will only run
+  once, during installation.
+
+* `tools/` Scripts used only in a Zulip development environment.
+  These are not included in production release tarballs for Zulip, so
+  that we can include scripts here one wouldn't want someone to run in
+  production accidentally (e.g. things that delete the Zulip database
+  without prompting).
+
+* `tools/setup/` Subdirectory of `tools/` for things only used during
+  the development environment setup process.
+
+* `tools/travis/` Subdirectory of `tools/` for things only used to
+  setup and run our tests in Travis CI.  Actually test suites should
+  go in `tools/`.
+
+---------------------------------------------------------
+
+### API and Bots
+
+* `api/` Zulip's Python API bindings (released separately).
+
+* `api/examples/` API examples.
+
+* `api/integrations/` Bots distributed as part of the Zulip API bundle.
+
+* `bots/` Previously Zulip internal bots. These usually need a bit of
+   work.
+
+-------------------------------------------------------------------------
+
+### Production puppet configuration
+
+This is used to deploy essentially all configuration in production.
+
+* `puppet/zulip/` For configuration for production deployments.
+
+* `puppet/zulip/manifests/voyager.pp` Main manifest for Zulip standalone deployments.
+
+-----------------------------------------------------------------------
+
+### Additional Django apps
+
+* `confirmation` Email confirmation system.
+
+* `analytics` Analytics for the Zulip server administrator (needs work to
+  be useful to normal Zulip sites).
+
+* `corporate` The old Zulip.com website.  Not included in production
+  distribution.
+
+* `zilencer` Primarily used to hold management commands that aren't
+  used in production.  Not included in production distribution.
+
+-----------------------------------------------------------------------
+
+### Jinja2 Compatibility Files
+
+* `zproject/jinja2/__init__.py` Jinja2 environment.
+
+* `zproject/jinja2/backends.py` Jinja2 backend.
+
+* `zproject/jinja2/compressors.py` Jinja2 compatible functions of
+   Django-Pipeline.
+
+-----------------------------------------------------------------------
+
+### Translation files
+
+* `locale/` Backend (Django) translations data files.
+
+* `static/locale/` Frontend translations data files.
+
+-----------------------------------------------------------------------
+
+### Documentation
+
+*  `docs/`        Source for this documentation.
+
+--------------------------------------------------------------
+
+You can consult the repository's `.gitattributes` file to see exactly
+which components are excluded from production releases (release
+tarballs are generated using `tools/build-release-tarball`).

docs/directory-structure.rst

@@ -1,95 +0,0 @@
-===================
-Directory structure
-===================
-
-This page documents the Zulip directory structure and how to decide where to
-put a file.
-
-Scripts
-=======
-
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/``       | Scripts that production deployments might run manually (e.g. ``restart-server``)  |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``bin/``           | Scripts that are needed on production deployments but humans should never run     |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``scripts/setup/`` | Tools that production deployments will only run once, during installation         |
-+--------------------+-----------------------------------------------------------------------------------+
-| ``tools/``         | Development tools                                                                 |
-+--------------------+-----------------------------------------------------------------------------------+
-
-Bots
-====
-
-+------------------------+----------------------------------------------------------------------+
-| ``api/integrations``   | Bots distributed as part of the Zulip API bundle.                    |
-+------------------------+----------------------------------------------------------------------+
-| ``bots/``              | Previously Zulip internal bots.  These usually need a bit of work.   |
-+------------------------+----------------------------------------------------------------------+
-
-Management commands
-===================
-
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-| ``zerver/management/commands/``     | Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly)   |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+
-
-Views
-=====
-
-+--------------------------------+-----------------------------------------+
-| ``zerver/tornadoviews.py``     | Tornado views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/webhooks.py``   | Webhook views                           |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/messages.py``   | message-related views                   |
-+--------------------------------+-----------------------------------------+
-| ``zerver/views/__init__.py``   | other Django views                      |
-+--------------------------------+-----------------------------------------+
-
-Static assets
-=============
-
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``assets/``   | For assets not to be served to the web (e.g. the system to generate our favicons)                             |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-| ``static/``   | For things we do want to both serve to the web and distribute to production deployments (e.g. the webpages)   |
-+---------------+---------------------------------------------------------------------------------------------------------------+
-
-Puppet
-======
-
-+--------------------+----------------------------------------------------------------------------------+
-| ``puppet/zulip``   | For configuration for production deployments                                     |
-+--------------------+----------------------------------------------------------------------------------+
-
-Templates
-=========
-
-+--------------------------+--------------------------------------------------------+
-| ``templates/zerver``     | For templates related to zerver views                  |
-+--------------------------+--------------------------------------------------------+
-| ``static/templates``     | Handlebars templates for the frontend                  |
-+--------------------------+--------------------------------------------------------+
-
-Tests
-=====
-
-+-------------------------+-----------------------------------+
-| ``zerver/tests/``       |          Backend tests            |
-+-------------------------+-----------------------------------+
-| ``frontend_tests/node`` |          Node Frontend unit tests |
-+-------------------------+-----------------------------------+
-| ``frontend_tests/tests``|          Casper frontend tests    |
-+-------------------------+-----------------------------------+
-
-Documentation
-=============
-
-+-------------+-----------------------------------------------+
-| ``docs/``   | Source for this documentation                 |
-+-------------+-----------------------------------------------+
-
-You can consult the repository's .gitattributes file to see exactly
-which components are excluded from production releases (release
-tarballs are generated using tools/build-release-tarball).

docs/front-end-build-process.md

@@ -0,0 +1,70 @@
+# Static asset pipeline
+
+This page documents additional information that may be useful when
+developing new features for Zulip that require front-end changes. For a
+more general overview, see the new-feature-tutorial. The code-style
+documentation also has relevant information about how Zulip's code is
+structured.
+
+## Primary build process
+
+Most of the existing JS in Zulip is written in IIFE-wrapped modules, one
+per file in the static/js directory. When running Zulip in development
+mode, each file is loaded seperately. In production mode (and when
+creating a release tarball using tools/build-release-tarball),
+JavaScript files are concatenated and minified.
+
+If you add a new JavaScript file, it needs to be specified in the
+JS\_SPECS dictionary defined in zproject/settings.py to be included in
+the concatenated file.
+
+## Webpack/CommonJS modules
+
+New JS written for Zulip can be written as CommonJS modules (bundled
+using [webpack](https://webpack.github.io/), though this will taken care
+of automatically whenever `run-dev.py` is running). (CommonJS is the
+same module format that Node uses, so see the [Node
+documentation](https://nodejs.org/docs/latest/api/modules.html) for
+more information on the syntax.)
+
+Benefits of using CommonJS modules over the
+[IIFE](http://benalman.com/news/2010/11/immediately-invoked-function-expression/)
+module approach:
+
+-   namespacing/module boilerplate will be added automatically in the
+    bundling process
+-   dependencies between modules are more explicit and easier to trace
+-   no separate list of JS files needs to be maintained for
+    concatenation and minification
+-   third-party libraries can be more easily installed/versioned using
+    npm
+-   running the same code in the browser and in Node for testing is
+    simplified (as both environments use the same module syntax)
+
+The entry point file for the bundle generated by webpack is
+`static/js/src/main.js`. Any modules you add will need to be required
+from this file (or one of its dependencies) in order to be included in
+the script bundle.
+
+## Adding static files
+
+To add a static file to the app (JavaScript, CSS, images, etc), first
+add it to the appropriate place under `static/`.
+
+-   Third-party files should all go in `static/third/`. Tag the commit
+    with "[third]" when adding or modifying a third-party package.
+-   Our own JS lives under `static/js`; CSS lives under `static/styles`.
+-   JavaScript and CSS files are combined and minified in production. In
+    this case all you need to do is add the filename to
+    PIPELINE['STYLESHEET'] or JS\_SPECS in `zproject/settings.py`. (If
+    you plan to only use the JS/CSS within the app proper, and not on
+    the login page or other standalone pages, put it in the 'app'
+    category.)
+
+If you want to test minified files in development, look for the
+`PIPELINE_ENABLED =` line in `zproject/settings.py` and set it to `True`
+-- or just set `DEBUG = False`.
+
+Note that `static/html/{400,5xx}.html` will only render properly if
+minification is enabled, since they hardcode the path
+`static/min/portico.css`.

docs/front-end-build-process.rst

@@ -1,71 +0,0 @@
-=======================
-Front End Build Process
-=======================
-
-This page documents additional information that may be useful when developing new features for Zulip that require front-end changes. For a more general overview, see the new feature tutorial. The code style documentation also has relevant information about how Zulip's code is structured.
-
-Primary build process
-=====================
-
-Most of the existing JS in Zulip is written in IIFE-wrapped modules,
-one per file in the `static/js` directory. When running Zulip in
-development mode, each file is loaded seperately.  In production mode
-(and when creating a release tarball using
-`tools/build-release-tarball`), JavaScript files are concatenated and
-minified.
-
-If you add a new JavaScript file, it needs to be specified in the
-`JS_SPECS` dictionary defined in `zproject/settings.py` to be included
-in the concatenated file.
-
-Webpack/CommonJS modules
-========================
-
-New JS written for Zulip can be written as CommonJS modules (bundled
-using `webpack <https://webpack.github.io/>`_, though this will taken
-care of automatically whenever ``run-dev.py`` is running). (CommonJS
-is the same module format that Node uses, so see `the Node
-documentation <https://nodejs.org/docs/latest/api/modules.html>` for
-more information on the syntax.)
-
-Benefits of using CommonJS modules over the `IIFE
-<http://benalman.com/news/2010/11/immediately-invoked-function-expression/>`_
-module approach:
-
-* namespacing/module boilerplate will be added automatically in the bundling process
-* dependencies between modules are more explicit and easier to trace
-* no separate list of JS files needs to be maintained for concatenation and minification
-* third-party libraries can be more easily installed/versioned using npm
-* running the same code in the browser and in Node for testing is
-  simplified (as both environments use the same module syntax)
-
-The entry point file for the bundle generated by webpack is
-``static/js/src/main.js``. Any modules you add will need to be
-required from this file (or one of its dependencies) in order to be
-included in the script bundle.
-
-Adding static files
-===================
-
-To add a static file to the app (JavaScript, CSS, images, etc), first
-add it to the appropriate place under ``static/``.
-
-* Third-party files should all go in ``static/third/``.  Tag the commit
-  with "[third]" when adding or modifying a third-party package.
-
-* Our own JS lives under ``static/js``; CSS lives under ``static/styles``.
-
-* JavaScript and CSS files are combined and minified in production. In
-  this case all you need to do is add the filename to PIPELINE_CSS or
-  JS_SPECS in ``zproject/settings.py``. (If you plan to only use the
-  JS/CSS within the app proper, and not on the login page or other
-  standalone pages, put it in the 'app' category.)
-
-If you want to test minified files in development, look for the
-``PIPELINE =`` line in ``zproject/settings.py`` and set it to ``True`` -- or
-just set ``DEBUG = False``.
-
-Note that ``static/html/{400,5xx}.html`` will only render properly if
-minification is enabled, since they hardcode the path
-``static/min/portico.css``.
-

docs/full-text-search.md

@@ -0,0 +1,39 @@
+# Full-text search
+
+Zulip supports full-text search, which can be combined arbitrarily
+with Zulip's full suite of narrowing operators.  By default, it only
+supports English text, but there is an experimental
+[PGroonga](http://pgroonga.github.io/) integration that provides
+full-text search for all languages.
+
+The user interface and feature set for Zulip's full-text search is
+documented in the "Search help" documentation section in the Zulip
+app's gear menu.
+
+## The default full-text search implementation
+
+Zulip's uses [PostgreSQL's built-in full-text search
+feature](http://www.postgresql.org/docs/current/static/textsearch.html),
+with a custom set of English stop words to improve the quality of the
+search results.
+
+We use a small extension,
+[tsearch_extras](https://github.com/zulip/tsearch_extras), for
+highlighting of the matching words.  There is [some discussion of
+removing this extension, at least as an
+option](https://github.com/zulip/zulip/issues/467), so that Zulip can
+be used with database-as-a-service platforms.
+
+In order to optimize the performance of delivering messages, the
+full-text search index is updated for newly sent messages in the
+background, after the message has been delivered.  This background
+updating is done by
+`puppet/zulip/files/postgresql/process_fts_updates`, which is usually
+deployed on the database server, but could be deployed on an
+application server instead.
+
+## An optional full-text search implementation
+
+See [the option PGroonga pull
+request](https://github.com/zulip/zulip/pull/700) for details on the
+status of the PGroonga integration.

docs/html_css.md

@@ -0,0 +1,72 @@
+# HTML and CSS
+
+## Zulip CSS organization
+
+The Zulip application's CSS can be found in the `static/styles/`
+directory.  Zulip uses [Bootstrap](http://getbootstrap.com/) as its
+main third-party CSS library.
+
+Zulip currently does not use any CSS preprocessors, and is organized
+into several files.  For most pages, the CSS is combined into a single
+CSS file by the [static asset pipeline](front-end-build-process.html),
+controlled by the `PIPELINE_CSS` code in `zproject/settings.py`.
+
+The CSS files are:
+
+* `portico.css` - Main CSS for logged-out pages
+* `pygments.css` - CSS for Python syntax highlighting
+* `activity.css` - CSS for the `activity` app
+* `fonts.css` - Fonts for text in the Zulip app
+* `static/styles/thirdparty-fonts.css` - Font Awesome (used for icons)
+
+The CSS for the Zulip web application UI is primarily here:
+
+* `settings.css` - CSS for the Zulip settings and administration pages
+* `zulip.css` - CSS for the rest of the Zulip logged-in app
+* `media.css` - CSS for media queries (particularly related to screen width)
+
+We are in the process of [splitting zulip.css into several more
+files](https://github.com/zulip/zulip/issues/731); help with that
+project is very welcome!
+
+## Editing Zulip CSS
+
+If you aren't experienced with doing web development and want to make
+CSS changes, we recommend reading the excellent [Chrome web inspector
+guide on editing HTML/CSS](https://developer.chrome.com/devtools/docs/dom-and-styles),
+especially the [section on
+CSS](https://developer.chrome.com/devtools/docs/dom-and-styles#styles)
+to learn about all the great tools that you can use to modify and test
+changes to CSS interactively in-browser (without even having the
+reload the page!).
+
+## CSS Style guidelines
+
+### Avoid duplicated code
+
+Without care, it's easy for a web application to end up with thousands
+of lines of duplicated CSS code, which can make it very difficult to
+understand the current styling or modify it.  We would very much like
+to avoid such a fate.  So please make an effort to reuse existing
+styling, clean up now-unused CSS, etc., to keep things maintainable.
+
+### Be consistent with existing similar UI
+
+Ideally, do this by reusing existing CSS declarations, so that any
+improvements we make to the styling can improve all similar UI
+elements.
+
+### Use clear, unique names for classes and object IDs
+
+This makes it much easier to read the code and use `git grep` to find
+where a particular class is used.
+
+## Validating CSS
+
+When changing any part of the Zulip CSS, it's important to check that
+the new CSS looks good at a wide range of screen widths, from very
+wide screen (e.g. 1920px) all the way down to narrow phone screens
+(e.g. 480px).
+
+For complex changes, it's definitely worth testing in a few different
+browsers to make sure things look the same.

docs/html_unescape.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.7
+#!/usr/bin/env python
 from __future__ import print_function
 
 # Remove HTML entity escaping left over from MediaWiki->rST conversion.