Add changelog for Zulip 1.3.9 release.

The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.

We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.

Thanks to Dan Fedele for the bug report and suggested solution!

.gitattributes

@@ -15,6 +15,6 @@
 /zproject/test_settings.py export-ignore
 /zerver/fixtures export-ignore
 /zerver/tests.py export-ignore
-/zerver/tests export-ignore
+/frontend_tests export-ignore
 /node_modules export-ignore
 /humbug export-ignore

.gitignore

@@ -13,7 +13,6 @@ stats/
 zerver/fixtures/available-migrations
 zerver/fixtures/migration-status
 zerver/fixtures/test_data1.json
-zerver/tests/frontend/test_credentials.js
 .kdev4
 zulip.kdev4
 memcached_prefix
@@ -30,7 +29,9 @@ manage.log
 event_queues.json
 .vagrant
 /zproject/dev-secrets.conf
+static/js/bundle.js
 static/third/gemoji/
 static/third/zxcvbn/
 tools/emoji_dump/bitmaps/
 tools/emoji_dump/*.ttx
+node_modules

.travis.yml

@@ -1,14 +1,20 @@
+before_install:
+   - nvm install 0.10 
 install:
-  - pip install pbs
+  - tools/travis/setup-$TEST_SUITE
-  - python provision.py --travis
 cache:
   - apt: false
+env:
+  - TEST_SUITE=frontend
+  - TEST_SUITE=backend
+  - TEST_SUITE=production
+  - TEST_SUITE=py3k
 language: python
 python:
   - "2.7"
 # command to run tests
 script:
-  - source /srv/zulip-venv/bin/activate && env PATH=$PATH:/srv/zulip-venv/bin ./tools/test-all
+  - ./tools/travis/$TEST_SUITE
 sudo: required
 services:
 - docker

README.dev.md

@@ -0,0 +1,363 @@
+
+Installing the Zulip Development environment
+============================================
+
+You will need a machine with at least 2GB of RAM available (see
+https://github.com/zulip/zulip/issues/32 for a plan for how to
+dramatically reduce this requirement).
+
+Start by cloning this repository: `git clone https://github.com/zulip/zulip.git`
+
+Using Vagrant
+-------------
+
+This is the recommended approach for all platforms, and will install
+the Zulip development environment inside a VM or container and works
+on any platform that supports Vagrant.
+
+The best performing way to run the Zulip development environment is
+using an LXC container on a Linux host, but we support other platforms
+such as Mac via Virtualbox (but everything will be 2-3x slower).
+
+* If your host is Ubuntu 15.04 or newer, you can install and configure
+  the LXC Vagrant provider directly using apt:
+  ```
+  sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir
+  vagrant plugin install vagrant-lxc
+  ```
+
+* If your host is Ubuntu 14.04, you will need to [download a newer
+  version of Vagrant](https://www.vagrantup.com/downloads.html), and
+  then do the following:
+  ```
+  sudo apt-get install lxc lxc-templates cgroup-lite redir
+  sudo dpkg -i vagrant*.deb # in directory where you downloaded vagrant
+  vagrant plugin install vagrant-lxc
+  ```
+
+* For other Linux hosts with a kernel above 3.12, [follow the Vagrant
+  LXC installation
+  instructions](https://github.com/fgrehm/vagrant-lxc) to get Vagrant
+  with LXC for your platform.
+
+* If your host is OS X or older Linux, [download VirtualBox](https://www.virtualbox.org/wiki/Downloads),
+  [download Vagrant](https://www.vagrantup.com/downloads.html), and install them both.
+
+* If you're on OS X and have VMWare, it should be possible to patch
+  Vagrantfile to use the VMWare vagrant provider which should perform
+  much better than Virtualbox.  Patches to do this by default if
+  VMWare is available are welcome!
+
+* On Windows: You can use Vagrant and Virtualbox/VMWare on Windows
+  with Cygwin, similar to the Mac setup.  Be sure to create your git
+  clone using `git clone https://github.com/zulip/zulip.git -c
+  core.autocrlf=false` to avoid Windows line endings being added to
+  files (this causes weird errors).
+
+Once that's done, simply change to your zulip directory and run
+`vagrant up` in your terminal to install the development server.  This
+will take a long time on the first run because Vagrant needs to
+download the Ubuntu Trusty base image, but later you can run `vagrant
+destroy` and then `vagrant up` again to rebuild the environment and it
+will be much faster.
+
+Once that finishes, you can run the development server as follows:
+
+```
+vagrant ssh -- -L9991:localhost:9991
+# Now inside the container
+cd /srv/zulip
+source /srv/zulip-venv/bin/activate
+./tools/run-dev.py --interface=''
+```
+
+To get shell access to the virtual machine running the server to run
+lint, management commands, etc., use `vagrant ssh`.
+
+(A small note on tools/run-dev.py: the `--interface=''` option will make
+the development server listen on all network interfaces.  While this
+is correct for the Vagrant guest sitting behind a NAT, you probably
+don't want to use that option when using run-dev.py in other environments).
+
+At this point you should [read about using the development environment](https://github.com/zulip/zulip/blob/master/README.dev.md#using-the-development-environment).
+
+
+Using provision.py without Vagrant
+----------------------------------
+
+If you'd like to install a Zulip development environment on a server
+that's already running Ubuntu 14.04 Trusty, you can do that by just
+running:
+
+```
+sudo apt-get update
+sudo apt-get install -y python-pbs
+python /srv/zulip/provision.py
+
+cd /srv/zulip
+source /srv/zulip-venv/bin/activate
+./tools/run-dev.py
+```
+
+Note that there is no supported uninstallation process without Vagrant
+(with Vagrant, you can just do `vagrant destroy` to clean up the
+development environment).
+
+By hand
+-------
+If you really want to install everything by hand, the below
+instructions should work.
+
+Install the following non-Python dependencies:
+ * libffi-dev — needed for some Python extensions
+ * postgresql 9.1 or later — our database (also install development headers)
+ * memcached (and headers)
+ * rabbitmq-server
+ * libldap2-dev
+ * python-dev
+ * redis-server — rate limiting
+ * tsearch-extras — better text search
+ * libfreetype6-dev - needed before you pip install Pillow to properly generate emoji PNGs
+
+### On Debian or Ubuntu systems:
+
+```
+sudo apt-get install libffi-dev memcached rabbitmq-server libldap2-dev python-dev redis-server postgresql-server-dev-all libmemcached-dev libfreetype6-dev
+
+# If on 12.04 or wheezy:
+sudo apt-get install postgresql-9.1
+wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
+sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
+
+# If on 14.04:
+sudo apt-get install postgresql-9.3
+wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
+sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
+
+# If on 15.04 or jessie:
+sudo apt-get install postgresql-9.4
+wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb
+sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
+```
+
+Now continue with the "All systems" instructions below.
+
+### On Fedora 22 (experimental):
+
+These instructions are experimental and may have bugs; patches welcome!
+
+```
+sudo dnf install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server postgresql-devel postgresql libmemcached-devel freetype-devel
+```
+
+Now continue with the Common to Fedora/CentOS instructions below.
+
+### On CentOS 7 Core (experimental):
+
+These instructions are experimental and may have bugs; patches welcome!
+
+```
+# Add user zulip to the system (not necessary if you configured zulip as the administrator
+# user during the install process of CentOS 7).
+useradd zulip
+
+# Create a password for zulip user
+passwd zulip
+
+# Allow zulip to sudo
+visudo
+# Add this line after line `root    ALL=(ALL)       ALL`
+zulip   ALL=(ALL)       ALL
+
+# Switch to zulip user
+su zulip
+
+# Enable EPEL 7 repo so we can install rabbitmq-server, redis and other dependencies
+sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+
+# Install dependencies
+sudo yum install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server \
+postgresql-devel postgresql libmemcached-devel wget python-pip openssl-devel freetype-devel libjpeg-turbo-devel \
+zlib-devel nodejs
+
+# We need these packages to compile tsearch-extras
+sudo yum groupinstall "Development Tools"
+
+# clone Zulip's git repo and cd into it
+cd && git clone https://github.com/zulip/zulip && cd zulip/
+
+## NEEDS TESTING: The next few DB setup items may not be required at all.
+# Initialize the postgres db
+sudo postgresql-setup initdb
+
+# Edit the postgres settings:
+sudo vi /var/lib/pgsql/data/pg_hba.conf
+
+# Change these lines:
+host    all             all             127.0.0.1/32            ident
+host    all             all             ::1/128                 ident
+# to this:
+host    all             all             127.0.0.1/32            md5
+host    all             all             ::1/128                 md5
+```
+
+Now continue with the Common to Fedora/CentOS instructions below.
+
+### Common to Fedora/CentOS instructions
+
+```
+# Build and install postgres tsearch-extras module
+wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz
+tar xvzf tsearch-extras_0.1.3.tar.gz
+cd ts2
+make
+sudo make install
+
+# Hack around missing dictionary files -- need to fix this to get
+# the proper dictionaries from what in debian is the hunspell-en-us package.
+sudo touch /usr/share/pgsql/tsearch_data/english.stop
+sudo touch /usr/share/pgsql/tsearch_data/en_us.dict
+sudo touch /usr/share/pgsql/tsearch_data/en_us.affix
+
+# Edit the postgres settings:
+sudo vi /var/lib/pgsql/data/pg_hba.conf
+
+# Add this line before the first uncommented line to enable password auth:
+host    all             all             127.0.0.1/32            md5
+
+# Start the services
+sudo systemctl start redis memcached rabbitmq-server postgresql
+
+# Enable automatic service startup after the system startup
+sudo systemctl enable redis rabbitmq-server memcached postgresql
+```
+
+Finally continue with the All Systems instructions below.
+
+### All Systems:
+
+```
+pip install -r requirements.txt
+./tools/download-zxcvbn
+./tools/emoji_dump/build_emoji
+./scripts/setup/generate_secrets.py -d
+sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.3/tsearch_data/
+./scripts/setup/configure-rabbitmq
+./tools/postgres-init-dev-db
+./tools/do-destroy-rebuild-database
+./tools/postgres-init-test-db
+./tools/do-destroy-rebuild-test-database
+```
+
+To start the development server:
+
+```
+./tools/run-dev.py
+```
+
+… and visit [http://localhost:9991/](http://localhost:9991/).
+
+
+Using the Development Environment
+=================================
+
+Once the development environment is running, you can visit
+<http://localhost:9991/> in your browser.  By default, the development
+server homepage just shows a list of the users that exist on the
+server and you can login as any of them by just clicking on a user.
+This setup saves time for the common case where you want to test
+something other than the login process; to test the login process
+you'll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION
+case of `zproject/settings.py` from zproject.backends.DevAuthBackend
+to use the auth method(s) you'd like to test.
+
+While developing, it's helpful to watch the `run-dev.py` console
+output, which will show any errors your Zulip development server
+encounters.
+
+When you make a change, here's a guide for what you need to do in
+order to see your change take effect in Development:
+
+* If you change Javascript or CSS, you'll just need to reload the
+browser window to see changes take effect.
+
+* If you change Python code used by the the main Django/Tornado server
+processes, these services are run on top of Django's [manage.py
+runserver](https://docs.djangoproject.com/en/1.8/ref/django-admin/#runserver-port-or-address-port),
+which will automatically restart the Zulip Django and Tornado servers
+whenever you save changes to Python code.  You can watch this happen
+in the `run-dev.py` console to make sure the backend has reloaded.
+
+* The Python queue workers don't automatically restart when you save
+changes (or when they stop running), so you will want to ctrl-C and
+then restart `run-dev.py` manually if you are testing changes to the
+queue workers or if a queue worker has crashed.
+
+* If you change the database schema, you'll need to use the standard
+Django migrations process to create and then run your migrations; see
+the [new feature
+tutorial](http://zulip.readthedocs.org/en/latest/new-feature-tutorial.html)
+for an example.  Additionally you should check out the [detailed
+testing docs](http://zulip.readthedocs.org/en/latest/testing.html) for
+how to run the tests properly after doing a migration.
+
+(In production, everything runs under supervisord and thus will
+restart if it crashes, and `upgrade-zulip` will take care of running
+migrations and then cleanly restaring the server for you).
+
+Running the test suite
+======================
+
+For more details, check out the [detailed testing
+docs](http://zulip.readthedocs.org/en/latest/testing.html).
+
+To run all the tests, do this:
+```
+./tools/test-all
+```
+
+For the Vagrant environment, you'll want to first enter the environment:
+```
+vagrant ssh
+source /srv/zulip-venv/bin/activate
+cd /srv/zulip
+```
+
+This runs the linter (`tools/lint-all`) plus all of our test suites;
+they can all be run separately (just read `tools/test-all` to see
+them).  You can also run individual tests which can save you a lot of
+time debugging a test failure, e.g.:
+
+```
+./tools/lint-all # Runs all the linters in parallel
+./tools/test-backend zerver.test_bugdown.BugdownTest.test_inline_youtube
+./tools/test-js-with-casper 10-navigation.js
+./tools/test-js-with-node # Runs all node tests but is very fast
+```
+
+The above setup instructions include the first-time setup of test
+databases, but you may need to rebuild the test database occasionally
+if you're working on new database migrations.  To do this, run:
+
+```
+./tools/postgres-init-test-db
+./tools/do-destroy-rebuild-test-database
+```
+
+Possible testing issues
+=======================
+
+- When running the test suite, if you get an error like this:
+
+  ```
+      sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not   exist
+      LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
+                                                                   ^
+  ```
+
+  … then you need to install tsearch-extras, described
+  above. Afterwards, re-run the `init*-db` and the
+  `do-destroy-rebuild*-database` scripts.
+
+- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000.
+  This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner

README.md

@@ -10,253 +10,121 @@ previews, group private messages, audible notifications,
 missed-message emails, desktop apps, and much more.
 
 Further information on the Zulip project and its features can be found
-at https://www.zulip.org
+at https://www.zulip.org.
-
-Contributing to Zulip
-=====================
-
-Zulip welcomes all forms of contributions!
-
-Before a pull request can be merged, you need to to sign the [Dropbox
-Contributor License Agreement](https://opensource.dropbox.com/cla/).
-
-Please run the tests (tools/test-all) before submitting your pull
-request and read our [commit message style
-guidelines](http://zulip.readthedocs.org/en/latest/code-style.html#commit-messages).
-
-Zulip has a growing collection of developer documentation including
-detailed documentation on coding style available on [Read The
-Docs](https://zulip.readthedocs.org/).
-
-Zulip also has a [development discussion mailing list](https://groups.google.com/forum/#!forum/zulip-devel)
-
-Feel free to send any questions or suggestions of areas where you'd
-love to see more documentation to the list!
-
-We recommend sending proposals for large features or refactorings to
-the zulip-devel list for discussion and advice before getting too deep
-into implementation.
-
-Please report any security issues you discover to support@zulip.com.
-
-Running Zulip in production
-===========================
-
-This is documented in https://zulip.org/server.html and [README.prod.md](README.prod.md).
 
 Installing the Zulip Development environment
 ============================================
 
-You will need a machine with at least 2GB of RAM available (see
+The Zulip development environment is the recommened option for folks
-https://github.com/zulip/zulip/issues/32 for a plan for how to
+interested in trying out Zulip.  This is documented in
-dramatically reduce this requirement).
+[README.dev.md](README.dev.md).
 
-Start by cloning this repository: `git clone https://github.com/zulip/zulip.git`
+Running Zulip in production
+===========================
 
-Using Vagrant
+Zulip in production only supports Ubuntu 14.04 right now, but work is
--------------
+ongoing on adding support for additional platforms. The installation
+process is documented in https://zulip.org/server.html and in more
+detail in [README.prod.md](README.prod.md).
 
-This is the recommended approach, and is tested on OS X 10.10 as well as Ubuntu 14.04.
+Contributing to Zulip
+=====================
 
-* The best performing way to run the Zulip development environment is
+Zulip welcomes all forms of contributions!  The page documents the
-  using an LXC container.  If your host is Ubuntu 14.04 (or newer;
+Zulip development process.
-  what matters is having support for LXC containers), you'll want to
-  install and configure the LXC Vagrant provider like this:
-  `sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir && vagrant plugin install vagrant-lxc`
 
-* If your host is OS X, [download VirtualBox](https://www.virtualbox.org/wiki/Downloads),
+* **Pull requests**. Before a pull request can be merged, you need to to sign the [Dropbox
-  [download Vagrant](https://www.vagrantup.com/downloads.html), and install them both.
+Contributor License Agreement](https://opensource.dropbox.com/cla/).
+Also, please skim our [commit message style
+guidelines](http://zulip.readthedocs.org/en/latest/code-style.html#commit-messages).
 
-Once that's done, simply change to your zulip directory and run
+* **Testing**. The Zulip automated tests all run automatically when
-`vagrant up` in your terminal to install the development server.  This
+you submit a pull request, but you can also run them all in your
-will take a long time on the first run because Vagrant needs to
+development environment following the instructions in the [testing
-download the Ubuntu Trusty base image, but later you can run `vagrant
+section](https://github.com/zulip/zulip#running-the-test-suite) below.
-destroy` and then `vagrant up` again to rebuild the environment and it
-will be much faster.
 
-Once that finishes, you can run the development server as follows:
+* **Developer Documentation**.  Zulip has a growing collection of
+developer documentation on [Read The Docs](https://zulip.readthedocs.org/).
+Recommended reading for new contributors includes the
+[directory structure](http://zulip.readthedocs.org/en/latest/directory-structure.html) and
+[new feature tutorial](http://zulip.readthedocs.org/en/latest/new-feature-tutorial.html).
 
-```
+* **Mailing list and bug tracker** Zulip has a [development discussion
-vagrant ssh -- -L9991:localhost:9991
+mailing list](https://groups.google.com/forum/#!forum/zulip-devel) and
-# Now inside the container
+uses [GitHub issues](https://github.com/zulip/zulip/issues).  Feel
-cd /srv/zulip
+free to send any questions or suggestions of areas where you'd love to
-source /srv/zulip-venv/bin/activate
+see more documentation to the list!  Please report any security issues
-./tools/run-dev.py --interface=''
+you discover to support@zulip.com.
-```
 
-You can now visit <http://localhost:9991/> in your browser.  To get
+* **App codebases** This repository is for the Zulip server and web app; the
-shell access to the virtual machine running the server, use `vagrant ssh`.
+[desktop](https://github.com/zulip/zulip-desktop),
+[Android](https://github.com/zulip/zulip-android), and
+[iOS](https://github.com/zulip/zulip-ios) apps are separate
+repositories.
 
-(A small note on tools/run-dev.py: the `--interface=''` option will make
+How to get involved with contributing to Zulip
-the development server listen on all network interfaces.  While this
+==============================================
-is correct for the Vagrant guest sitting behind a NAT, you probably
-don't want to use that option when using run-dev.py in other environments).
 
-The run-dev.py console output will show any errors your Zulip
+First, subscribe to the Zulip [development discussion mailing list](https://groups.google.com/forum/#!forum/zulip-devel).
-development server encounters.  It runs on top of Django's "manage.py
-runserver" tool, which will automatically restart the Zulip server
-whenever you save changes to Python code.
 
-However, the Zulip queue workers will not automatically restart when
+The Zulip project uses a system of labels in our [issue
-you save changes, so you will need to ctrl-C and then restart
+tracker](https://github.com/zulip/zulip/issues) to make it easy to
-`run-dev.py` manually if you are testing changes to the queue workers
+find a project if you don't have your own project idea in mind or want
-or if a queue worker has crashed.
+to get some experience with working on Zulip before embarking on a
+larger project you have in mind:
 
-Using provision.py without Vagrant
+* [Bite Size](https://github.com/zulip/zulip/labels/bite%20size):
-----------------------------------
+  Smaller projects that could be a great first contribution.
+* [Integrations](https://github.com/zulip/zulip/labels/integrations).
+  Integrate Zulip with another piece of software and contribute it
+  back to the community!  Writing an integration can be a great
+  started project.  There's some brief documentation on the best way
+  to write integrations at https://github.com/zulip/zulip/issues/70.
+* [Documentation](https://github.com/zulip/zulip/labels/documentation).
+  The Zulip project loves contributions of new documentation.
+* [Help Wanted](https://github.com/zulip/zulip/labels/help%20wanted):
+  A broader list of projects that nobody is currently working on.
+* [Platform support](https://github.com/zulip/zulip/labels/Platform%20support).
+  These are open issues about making it possible to install Zulip on a wider
+  range of platforms.
+* [Bugs](https://github.com/zulip/zulip/labels/bug). Open bugs.
+* [Feature requests](https://github.com/zulip/zulip/labels/enhancement).
+  Browsing this list can be a great way to find feature ideas to implement that
+  other Zulip users are excited about.
 
-If you'd like to install a Zulip development environment on a server
+If you're excited about helping with an open issue, just post on the
-that's already running Ubuntu 14.04 Trusty, you can do that by just
+conversation thread that you're working on it.  You're encouraged to
-running:
+ask questions on how to best implement or debug your changes -- the
+Zulip maintainers are excited to answer questions to help you stay
+unblocked and working efficiently.
 
-```
+We also welcome suggestions of features that you feel would be
-sudo apt-get update
+valuable or changes that you feel would make Zulip a better open
-sudo apt-get install -y python-pbs
+source project, and are happy to support you in adding new features or
-python /srv/zulip/provision.py
+other user experience improvements to Zulip.
 
-cd /srv/zulip
+If you have a new feature you'd like to add, we recommend you start by
-source /srv/zulip-venv/bin/activate
+opening a GitHub issue about the feature idea explaining the problem
-./tools/run-dev.py
+that you're hoping to solve and that you're excited to work on it.  A
-```
+Zulip maintainer will usually reply within a day with feedback on the
+idea, notes on any important issues or concerns, and and often tips on
+how to implement or test it.  Please feel free to ping the thread if
+you don't hear a response from the maintainers -- we try to be very
+responsive so this usually means we missed your message.
 
-By hand
+For significant changes to the visual design, user experience, data
--------
+model, or architecture, we highly recommend posting a mockup,
-If you really want to install everything by hand, the below
+screenshot, or description of what you have in mind to zulip-devel@ to
-instructions should work.
+get broad feedback before you spend too much time on implementation
+details.
 
-Install the following non-Python dependencies:
+Finally, before implementing a larger feature, we highly recommend
- * libffi-dev — needed for some Python extensions
+looking at the new feature tutorial and coding style guidelines on
- * postgresql 9.1 or later — our database (also install development headers)
+ReadTheDocs.
- * memcached (and headers)
- * rabbitmq-server
- * libldap2-dev
- * python-dev
- * redis-server — rate limiting
- * tsearch-extras — better text search
- * libfreetype6-dev - needed before you pip install Pillow to properly generate emoji PNGs
 
-On Debian or Ubuntu systems:
+Feedback on how to make this development process more efficient, fun,
-
+and friendly to new contributors is very welcome!  Just shoot an email
-```
+to the Zulip Developers list with your thoughts.
-sudo apt-get install libffi-dev memcached rabbitmq-server libldap2-dev python-dev redis-server postgresql-server-dev-all libmemcached-dev libfreetype6-dev
-
-# If on 12.04 or wheezy:
-sudo apt-get install postgresql-9.1
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
-
-# If on 14.04:
-sudo apt-get install postgresql-9.3
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.2_amd64.deb
-
-# If on 15.04 or jessie:
-sudo apt-get install postgresql-9.4
-wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb
-sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
-```
-
-Now continue with the "All systems" instructions below.
-
-On Fedora 22 (experimental):
-
-```
-sudo dnf install libffi-devel memcached rabbitmq-server openldap-devel python-devel redis postgresql-server postgresql-devel postgresql libmemcached-devel freetype-devel
-wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz
-tar xvzf tsearch-extras_0.1.3.tar.gz
-cd ts2
-make
-sudo make install
-
-# Hack around missing dictionary files -- need to fix this to get
-# the proper dictionaries from what in debian is the hunspell-en-us package.
-sudo touch /usr/share/pgsql/tsearch_data/english.stop
-sudo touch /usr/share/pgsql/tsearch_data/en_us.dict
-sudo touch /usr/share/pgsql/tsearch_data/en_us.affix
-
-# Edit the postgres settings:
-sudo vi /var/lib/pgsql/data/pg_hba.conf
-
-# Add this line before the first uncommented line to enable password auth:
-host    all             all             127.0.0.1/32            md5
-
-# Start the services
-sudo systemctl start redis memcached rabbitmq-server postgresql
-```
-
-All Systems:
-
-```
-pip install -r requirements.txt
-./tools/download-zxcvbn
-./tools/emoji_dump/build_emoji
-./scripts/setup/generate_secrets.py -d
-sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.3/tsearch_data/
-./scripts/setup/configure-rabbitmq
-./tools/postgres-init-dev-db
-./tools/do-destroy-rebuild-database
-./tools/postgres-init-test-db
-./tools/do-destroy-rebuild-test-database
-```
-
-To start the development server:
-
-```
-./tools/run-dev.py
-```
-
-… and visit [http://localhost:9991/](http://localhost:9991/).
-
-
-Running the test suite
-======================
-
-Run all tests:
-
-```
-./tools/test-all
-```
-
-This runs the linter plus all of our test suites; they can all be run
-separately (just read `tools/test-all` to see them).  You can also run
-individual tests, e.g.:
-
-```
-./tools/test-backend zerver.test_bugdown.BugdownTest.test_inline_youtube
-./tools/test-js-with-casper 10-navigation.js
-```
-
-The above instructions include the first-time setup of test databases,
-but you may need to rebuild the test database occasionally if you're
-working on new database migrations.  To do this, run:
-
-```
-./tools/postgres-init-test-db
-./tools/do-destroy-rebuild-test-database
-```
-
-Possible testing issues
-=======================
-
-- The Casper tests are flaky on the Virtualbox environment (probably
-  due to some performance-sensitive races; they work reliably in
-  Travis CI).  Until this issue is debugged, you may need to rerun
-  them to get them to pass.
-
-- When running the test suite, if you get an error like this:
-
-  ```
-      sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not   exist
-      LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
-                                                                   ^
-  ```
-
-  … then you need to install tsearch-extras, described
-  above. Afterwards, re-run the `init*-db` and the
-  `do-destroy-rebuild*-database` scripts.
-
-- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000.
-  This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner
 
 License
 =======

README.prod.md

@@ -10,10 +10,10 @@ worry about setting up SSL certificates and an authentication mechanism.
 
 Recommended requirements:
 
-* Server running Ubuntu Precise or Debian Wheezy
+* Server running Ubuntu Trusty
 * At least 2 CPUs for production use with 100+ users
-* At least 4GB of RAM for production use with 100+ users.  We strongly
+* At least 4GB of RAM for production use with 100+ users.  We **strongly
-  recommend against installing with less than 2GB of RAM, as you will
+  recommend against installing with less than 2GB of RAM**, as you will
   likely experience OOM issues.  In the future we expect Zulip's RAM
   requirements to decrease to support smaller installations (see
   https://github.com/zulip/zulip/issues/32).
@@ -114,7 +114,7 @@ The `ADMIN_DOMAIN` realm is by default configured with the following settings:
 * `mandatory_topics=False`: Users are not required to specify a topic when sending messages.
 
 If you would like to change these settings, you can do so using the
-following process as the zulip user:
+Django management python shell (as the zulip user):
 
 ```
 cd /home/zulip/deployments/current
@@ -127,7 +127,11 @@ r.save() # save to the database
 
 If you realize you set `ADMIN_DOMAIN` wrong, in addition to fixing the
 value in settings.py, you will also want to do a similar manage.py
-process to set `r.domain = newexample.com`.
+process to set `r.domain = "newexample.com"`.  If you've already
+changed `ADMIN_DOMAIN` in settings.py, you can use
+`Realm.objects.all()` in the management shell to find the list of
+realms and pass the domain of the realm that is not "zulip.com" to
+`get_realm`.
 
 Depending what authentication backend you're planning to use, you will
 need to do some additional setup documented in the `settings.py` template:
@@ -140,7 +144,7 @@ need to do some additional setup documented in the `settings.py` template:
 You should be able to login now.  If you get an error, check
 `/var/log/zulip/errors.log` for a traceback, and consult the next
 section for advice on how to debug.  If you aren't able to figure it
-out, email zulip-devel@googlegroups.com with the traceback and we'll
+out, email zulip-help@googlegroups.com with the traceback and we'll
 try to help you out!
 
 You will likely want to make your own user account an admin user,
@@ -231,8 +235,11 @@ problems and how to resolve them:
   service nginx restart
   ```
 
-If you run into additional problems, [please report them](https://github.com/zulip/zulip/issues) so that we can
+If you run into additional problems, [please report
-update these lists!
+them](https://github.com/zulip/zulip/issues) so that we can update
+these lists!  The Zulip installation scripts logs its full output to
+`/var/log/zulip/install.log`, so please include the context for any
+tracebacks from that log.
 
 
 Making your Zulip instance awesome
@@ -328,26 +335,59 @@ everything anyone might want to know about running Zulip in
 production.
 
 
-Maintaining Zulip in production
+Maintaining and upgrading Zulip in production
-===============================
+=============================================
 
-* To upgrade to a new version, download the appropriate release
+We recommend reading this entire section before doing your first
-  tarball from https://www.zulip.com/dist/releases/ to a path readable
+upgrade.
-  by the zulip user (e.g. /home/zulip), and then run as root:
+
+* To upgrade to a new version of the zulip server, download the
+  appropriate release tarball from
+  https://www.zulip.com/dist/releases/ to a path readable by the zulip
+  user (e.g. /home/zulip), and then run as root:
   ```
   /home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
   ```
+  Be sure to download to a path readable by the Zulip user (see
+  https://github.com/zulip/zulip/issues/208 for details on this
+  issue) but then run the upgrade as root.
 
   The upgrade process will shut down the service, run `apt-get
-  upgrade` and any database migrations, and then bring the service
+  upgrade`, a puppet apply, and any database migrations, and then
-  back up.  This will result in some brief downtime for the service,
+  bring the service back up.  This will result in some brief downtime
-  which should be under 30 seconds unless there is an expensive
+  for the service, which should be under 30 seconds unless there is an
-  transition involved.  Unless you have tested the upgrade in advance,
+  expensive transition involved.  Unless you have tested the upgrade
-  we recommend doing upgrades at off hours.
+  in advance, we recommend doing upgrades at off hours.
 
   You can create your own release tarballs from a copy of zulip.git
   repository using `tools/build-release-tarball`.
 
+* **Warning**: If you have modified configuration files installed by
+  Zulip (e.g. the nginx configuration), the Zulip upgrade process will
+  overwrite your configuration when it does the `puppet apply`.  You
+  can test whether this will happen assuming no upstream changes to
+  the configuration using `scripts/zulip-puppet-apply` (without the
+  `-f` option), which will do a test puppet run and output and changes
+  it would make.  Using this list, you can save a copy of any files
+  that you've modified, do the upgrade, and then restore your
+  configuration.  If you need to do this, please report the issue so
+  that we can make the Zulip puppet configuration flexible enough to
+  handle your setup.
+
+* The Zulip upgrade script automatically logs output to
+  /var/log/zulip/upgrade.log; please use those logs to include output
+  that shows all errors in any bug reports.
+
+* The Zulip upgrade process works by creating a new deployment under
+  /home/zulip/deployments/ containing a complete copy of the Zulip
+  server code, and then moving the symlinks at
+  `/home/zulip/deployments/current` and /root/zulip` as part of the
+  upgrade process.  This means that if the new version isn't working,
+  you can quickly downgrade to the old version by using
+  `/home/zulip/deployments/<date>/scripts/restart-server` to return to
+  a previous version that you've deployed (the version is specified
+  via the path to the copy of `restart-server` you call).
+
 * To update your settings, simply edit `/etc/zulip/settings.py` and then
   run `/home/zulip/deployments/current/scripts/restart-server` to
   restart the server
@@ -357,10 +397,10 @@ Maintaining Zulip in production
   security patches.
 
 * To use the Zulip API with your Zulip server, you will need to use the
-  API endpoint of e.g. `https://zulip.yourdomain.net/api`.  Our Python
+  API endpoint of e.g. `https://zulip.example.com/api`.  Our Python
   API example scripts support this via the
-  `--site=https://zulip.yourdomain.net` argument.  The API bindings
+  `--site=https://zulip.example.com` argument.  The API bindings
-  support it via putting `site=https://zulip.yourdomain.net` in your
+  support it via putting `site=https://zulip.example.com` in your
   .zuliprc.
 
   Every Zulip integration supports this sort of argument (or e.g. a
@@ -379,8 +419,8 @@ Maintaining Zulip in production
   precise configuration.
 
 
-SSO Authentication
+Remote User SSO Authentication
-==================
+==============================
 
 Zulip supports integrating with a corporate Single-Sign-On solution.
 There are a few ways to do it, but this section documents how to
@@ -412,5 +452,59 @@ place your completed configuration file at `/etc/apache2/sites-available/zulip-s
 
 (4) Run `a2ensite zulip-sso` to enable the Apache integration site.
 
-Now you should be able to visit `https://zulip.yourdomain.net/` and
+Now you should be able to visit `https://zulip.example.com/` and
 login via the SSO solution.
+
+
+### Troubleshooting Remote User SSO
+
+This system is a little finicky to networking setup (e.g. common
+issues have to do with /etc/hosts not mapping settings.EXTERNAL_HOST
+to the Apache listening on 127.0.0.1/localhost, for example).  It can
+often help while debugging to temporarily change the Apache config in
+/etc/apache2/sites-available/zulip-sso to listen on all interfaces
+rather than just 127.0.0.1 as you debug this.  It can also be helpful
+to change /etc/nginx/zulip-include/app.d/external-sso.conf to
+proxy_pass to a more explicit URL possibly not over HTTPS when
+debugging.  The following log files can be helpful when debugging this
+setup:
+
+* /var/log/zulip/{errors.log,server.log} (the usual places)
+* /var/log/nginx/access.log (nginx access logs)
+* /var/log/apache2/zulip_auth_access.log (you may want to change
+  LogLevel to "debug" in the apache config file to make this more
+  verbose)
+
+Here's a summary of how the remote user SSO system works assuming
+you're using HTTP basic auth; this summary should help with
+understanding what's going on as you try to debug:
+
+* Since you've configured /etc/zulip/settings.py to only define the
+  zproject.backends.ZulipRemoteUserBackend, zproject/settings.py
+  configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes
+  `https://zulip.example.com/` aka the homepage for the main Zulip
+  Django app running behind nginx redirect to /accounts/login/sso if
+  you're not logged in.
+
+* nginx proxies requests to /accounts/login/sso/ to an Apache instance
+  listening on localhost:8888 apache via the config in
+  /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream
+  localhost:8888 defined in /etc/nginx/zulip-include/upstreams).
+
+* The Apache zulip-sso site which you've enabled listens on
+  localhost:8888 and presents the htpasswd dialogue; you provide
+  correct login information and the request reaches a second Zulip
+  Django app instance that is running behind Apache with with
+  REMOTE_USER set.  That request is served by
+  `zerver.views.remote_user_sso`, which just checks the REMOTE_USER
+  variable and either logs in (sets a cookie) or registers the new
+  user (depending whether they have an account).
+
+* After succeeding, that redirects the user back to / on port 443
+  (hosted by nginx); the main Zulip Django app sees the cookie and
+  proceeds to load the site homepage with them logged in (just as if
+  they'd logged in normally via username/password).
+
+Again, most issues with this setup tend to be subtle issues with the
+hostname/DNS side of the configuration.  Suggestions for how to
+improve this SSO setup documentation are very welcome!

THIRDPARTY

@@ -38,10 +38,6 @@ Files: confirmation/*
 Copyright: 2008, Jarek Zgoda <jarek.zgoda@gmail.com>
 License: BSD-3-Clause
 
-Files: node_modules/handlebars/*
-Copyright: 2011 Yehuda Katz
-License: Expat
-
 Files: puppet/apt/*
 Copyright: 2011, Evolving Web Inc.
 License: Expat
@@ -131,10 +127,6 @@ Copyright: Google, Inc.
 License: Apache-2.0
 Comment: These are actually Noto Emoji, not gemoji.
 
-Files: static/third/handlebars/handlebars.runtime.js
-Copyright: 2011 Yehuda Katz
-License: Expat
-
 Files: static/third/html5-formdata/formdata.js
 Copyright: 2010 François de Metz
 License: Expat
@@ -254,12 +246,12 @@ Files: zerver/lib/ccache.py
 Copyright: 2013 David Benjamin and Alan Huang
 License: Expat
 
-Files: zerver/tests/frontend/casperjs/*
+Files: frontend_tests/casperjs/*
 Copyright: 2011-2012 Nicolas Perriault
  Joyent, Inc. and other Node contributors
 License: Expat
 
-Files: zerver/tests/frontend/casperjs/modules/vendors/*
+Files: frontend_tests/casperjs/modules/vendors/*
 Copyright: 2011, Jeremy Ashkenas
 License: Expat
 

analytics/management/commands/active_user_stats.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from django.core.management.base import BaseCommand
 

analytics/management/commands/active_user_stats_by_day.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 import datetime
 import pytz
@@ -19,6 +20,6 @@ class Command(BaseCommand):
             date = datetime.datetime.now() - datetime.timedelta(days=1)
         else:
             date = datetime.datetime.strptime(options["date"], "%Y-%m-%d")
-        print "Activity data for", date
+        print("Activity data for", date)
-        print activity_averages_during_day(date)
+        print(activity_averages_during_day(date))
-        print "Please note that the total registered user count is a total for today"
+        print("Please note that the total registered user count is a total for today")

analytics/management/commands/analyze_mit.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from optparse import make_option
 from django.core.management.base import BaseCommand
@@ -63,7 +64,7 @@ def compute_stats(log_level):
         logging.info("Top %6s | %s%%" % (size, round(top_percents[size], 1)))
 
     grand_total = sum(total_counts.values())
-    print grand_total
+    print(grand_total)
     logging.info("%15s | %s" % ("Client", "Percentage"))
     for client in total_counts.keys():
         logging.info("%15s | %s%%" % (client, round(100. * total_counts[client] / grand_total, 1)))

analytics/management/commands/analyze_user_activity.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from zerver.lib.statistics import seconds_usage_between
 
@@ -16,7 +17,7 @@ def analyze_activity(options):
     if options["realm"]:
         user_profile_query = user_profile_query.filter(realm__domain=options["realm"])
 
-    print "Per-user online duration:\n"
+    print("Per-user online duration:\n")
     total_duration = datetime.timedelta(0)
     for user_profile in user_profile_query:
         duration = seconds_usage_between(user_profile, day_start, day_end)
@@ -25,11 +26,11 @@ def analyze_activity(options):
             continue
 
         total_duration += duration
-        print "%-*s%s" % (37, user_profile.email, duration, )
+        print("%-*s%s" % (37, user_profile.email, duration, ))
 
-    print "\nTotal Duration:                      %s" % (total_duration,)
+    print("\nTotal Duration:                      %s" % (total_duration,))
-    print "\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,)
+    print("\nTotal Duration in minutes:           %s" % (total_duration.total_seconds() / 60.,))
-    print "Total Duration amortized to a month: %s" % (total_duration.total_seconds() * 30. / 60.,)
+    print("Total Duration amortized to a month: %s" % (total_duration.total_seconds() * 30. / 60.,))
 
 class Command(BaseCommand):
     help = """Report analytics of user activity on a per-user and realm basis.
@@ -42,7 +43,7 @@ It will correctly not count server-initiated reloads in the activity statistics.
 
 The duration flag can be used to control how many days to show usage duration for
 
-Usage: python manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
+Usage: python2.7 manage.py analyze_user_activity [--realm=zulip.com] [--date=2013-09-10] [--duration=1]
 
 By default, if no date is selected 2013-09-10 is used. If no realm is provided, information
 is shown for all realms"""

analytics/management/commands/client_activity.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from django.core.management.base import BaseCommand
 from django.db.models import Count
@@ -13,9 +14,9 @@ class Command(BaseCommand):
 
 Usage examples:
 
-python manage.py client_activity
+python2.7 manage.py client_activity
-python manage.py client_activity zulip.com
+python2.7 manage.py client_activity zulip.com
-python manage.py client_activity jesstess@zulip.com"""
+python2.7 manage.py client_activity jesstess@zulip.com"""
 
     def add_arguments(self, parser):
         parser.add_argument('arg', metavar='<arg>', type=str, nargs='?', default=None,
@@ -48,8 +49,8 @@ python manage.py client_activity jesstess@zulip.com"""
         counts.sort()
 
         for count in counts:
-            print "%25s %15d" % (count[1], count[0])
+            print("%25s %15d" % (count[1], count[0]))
-        print "Total:", total
+        print("Total:", total)
 
 
     def handle(self, *args, **options):
@@ -70,5 +71,5 @@ python manage.py client_activity jesstess@zulip.com"""
                     self.compute_activity(UserActivity.objects.filter(
                             user_profile__realm=realm))
                 except Realm.DoesNotExist:
-                    print "Unknown user or domain %s" % (arg,)
+                    print("Unknown user or domain %s" % (arg,))
                     exit(1)

analytics/management/commands/realm_stats.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 import datetime
 import pytz
@@ -65,52 +66,51 @@ class Command(BaseCommand):
             fraction = 0.0
         else:
             fraction = numerator / float(denominator)
-        print "%.2f%% of" % (fraction * 100,), text
+        print("%.2f%% of" % (fraction * 100,), text)
 
     def handle(self, *args, **options):
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
+            except Realm.DoesNotExist as e:
-                print e
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
+            print(realm.domain)
 
             user_profiles = UserProfile.objects.filter(realm=realm, is_active=True)
             active_users = self.active_users(realm)
             num_active = len(active_users)
 
-            print "%d active users (%d total)" % (num_active, len(user_profiles))
+            print("%d active users (%d total)" % (num_active, len(user_profiles)))
             streams = Stream.objects.filter(realm=realm).extra(
                 tables=['zerver_subscription', 'zerver_recipient'],
                 where=['zerver_subscription.recipient_id = zerver_recipient.id',
                        'zerver_recipient.type = 2',
                        'zerver_recipient.type_id = zerver_stream.id',
                        'zerver_subscription.active = true']).annotate(count=Count("name"))
-            print "%d streams" % (streams.count(),)
+            print("%d streams" % (streams.count(),))
 
             for days_ago in (1, 7, 30):
-                print "In last %d days, users sent:" % (days_ago,)
+                print("In last %d days, users sent:" % (days_ago,))
                 sender_quantities = [self.messages_sent_by(user, days_ago) for user in user_profiles]
                 for quantity in sorted(sender_quantities, reverse=True):
-                    print quantity,
+                    print(quantity, end=' ')
-                print ""
+                print("")
 
-                print "%d stream messages" % (self.stream_messages(realm, days_ago),)
+                print("%d stream messages" % (self.stream_messages(realm, days_ago),))
-                print "%d one-on-one private messages" % (self.private_messages(realm, days_ago),)
+                print("%d one-on-one private messages" % (self.private_messages(realm, days_ago),))
-                print "%d messages sent via the API" % (self.api_messages(realm, days_ago),)
+                print("%d messages sent via the API" % (self.api_messages(realm, days_ago),))
-                print "%d group private messages" % (self.group_private_messages(realm, days_ago),)
+                print("%d group private messages" % (self.group_private_messages(realm, days_ago),))
 
-            num_notifications_enabled = len(filter(lambda x: x.enable_desktop_notifications == True,
+            num_notifications_enabled = len([x for x in active_users if x.enable_desktop_notifications == True])
-                                                   active_users))
             self.report_percentage(num_notifications_enabled, num_active,
                                    "active users have desktop notifications enabled")
 
-            num_enter_sends = len(filter(lambda x: x.enter_sends, active_users))
+            num_enter_sends = len([x for x in active_users if x.enter_sends])
             self.report_percentage(num_enter_sends, num_active,
                                    "active users have enter-sends")
 
@@ -124,8 +124,8 @@ class Command(BaseCommand):
             starrers = UserMessage.objects.filter(user_profile__in=user_profiles,
                                                   flags=UserMessage.flags.starred).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users have starred %d messages" % (
+            print("%d users have starred %d messages" % (
-                len(starrers), sum([elt["count"] for elt in starrers]))
+                len(starrers), sum([elt["count"] for elt in starrers])))
 
             active_user_subs = Subscription.objects.filter(
                 user_profile__in=user_profiles, active=True)
@@ -133,20 +133,20 @@ class Command(BaseCommand):
             # Streams not in home view
             non_home_view = active_user_subs.filter(in_home_view=False).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users have %d streams not in home view" % (
+            print("%d users have %d streams not in home view" % (
-                len(non_home_view), sum([elt["count"] for elt in non_home_view]))
+                len(non_home_view), sum([elt["count"] for elt in non_home_view])))
 
             # Code block markup
             markup_messages = human_messages.filter(
                 sender__realm=realm, content__contains="~~~").values(
                 "sender").annotate(count=Count("sender"))
-            print "%d users have used code block markup on %s messages" % (
+            print("%d users have used code block markup on %s messages" % (
-                len(markup_messages), sum([elt["count"] for elt in markup_messages]))
+                len(markup_messages), sum([elt["count"] for elt in markup_messages])))
 
             # Notifications for stream messages
             notifications = active_user_subs.filter(notifications=True).values(
                 "user_profile").annotate(count=Count("user_profile"))
-            print "%d users receive desktop notifications for %d streams" % (
+            print("%d users receive desktop notifications for %d streams" % (
-                len(notifications), sum([elt["count"] for elt in notifications]))
+                len(notifications), sum([elt["count"] for elt in notifications])))
 
-            print ""
+            print("")

analytics/management/commands/stream_stats.py

@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 from django.core.management.base import BaseCommand
 from django.db.models import Q
@@ -15,26 +16,26 @@ class Command(BaseCommand):
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
+            except Realm.DoesNotExist as e:
-                print e
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
+            print(realm.domain)
-            print "------------"
+            print("------------")
-            print "%25s %15s %10s" % ("stream", "subscribers", "messages")
+            print("%25s %15s %10s" % ("stream", "subscribers", "messages"))
             streams = Stream.objects.filter(realm=realm).exclude(Q(name__istartswith="tutorial-"))
             invite_only_count = 0
             for stream in streams:
                 if stream.invite_only:
                     invite_only_count += 1
                     continue
-                print "%25s" % (stream.name,),
+                print("%25s" % (stream.name,), end=' ')
                 recipient = Recipient.objects.filter(type=Recipient.STREAM, type_id=stream.id)
-                print "%10d" % (len(Subscription.objects.filter(recipient=recipient, active=True)),),
+                print("%10d" % (len(Subscription.objects.filter(recipient=recipient, active=True)),), end=' ')
                 num_messages = len(Message.objects.filter(recipient=recipient))
-                print "%12d" % (num_messages,)
+                print("%12d" % (num_messages,))
-            print "%d invite-only streams" % (invite_only_count,)
+            print("%d invite-only streams" % (invite_only_count,))
-            print ""
+            print("")

analytics/management/commands/user_stats.py

@@ -1,10 +1,12 @@
 from __future__ import absolute_import
+from __future__ import print_function
 
 import datetime
 import pytz
 
 from django.core.management.base import BaseCommand
 from zerver.models import UserProfile, Realm, Stream, Message, get_realm
+from six.moves import range
 
 class Command(BaseCommand):
     help = "Generate statistics on user activity."
@@ -22,20 +24,20 @@ class Command(BaseCommand):
         if options['realms']:
             try:
                 realms = [get_realm(domain) for domain in options['realms']]
-            except Realm.DoesNotExist, e:
+            except Realm.DoesNotExist as e:
-                print e
+                print(e)
                 exit(1)
         else:
             realms = Realm.objects.all()
 
         for realm in realms:
-            print realm.domain
+            print(realm.domain)
             user_profiles = UserProfile.objects.filter(realm=realm, is_active=True)
-            print "%d users" % (len(user_profiles),)
+            print("%d users" % (len(user_profiles),))
-            print "%d streams" % (len(Stream.objects.filter(realm=realm)),)
+            print("%d streams" % (len(Stream.objects.filter(realm=realm)),))
 
             for user_profile in user_profiles:
-                print "%35s" % (user_profile.email,),
+                print("%35s" % (user_profile.email,), end=' ')
                 for week in range(10):
-                    print "%5d" % (self.messages_sent_by(user_profile, week)),
+                    print("%5d" % (self.messages_sent_by(user_profile, week)), end=' ')
-                print ""
+                print("")

analytics/views.py

@@ -1,3 +1,4 @@
+from __future__ import absolute_import
 from django.db import connection
 from django.template import RequestContext, loader
 from django.utils.html import mark_safe
@@ -15,6 +16,10 @@ import itertools
 import time
 import re
 import pytz
+from six.moves import filter
+from six.moves import map
+from six.moves import range
+from six.moves import zip
 eastern_tz = pytz.timezone('US/Eastern')
 
 def make_table(title, cols, rows, has_row_class=False):
@@ -22,7 +27,7 @@ def make_table(title, cols, rows, has_row_class=False):
     if not has_row_class:
         def fix_row(row):
             return dict(cells=row, row_class=None)
-        rows = map(fix_row, rows)
+        rows = list(map(fix_row, rows))
 
     data = dict(title=title, cols=cols, rows=rows)
 
@@ -37,7 +42,7 @@ def dictfetchall(cursor):
     "Returns all rows from a cursor as a dict"
     desc = cursor.description
     return [
-        dict(zip([col[0] for col in desc], row))
+        dict(list(zip([col[0] for col in desc], row)))
         for row in cursor.fetchall()
     ]
 
@@ -226,7 +231,7 @@ def realm_summary_table(realm_minutes):
     def meets_goal(row):
         return row['active_user_count'] >= 5
 
-    num_active_sites = len(filter(meets_goal, rows))
+    num_active_sites = len(list(filter(meets_goal, rows)))
 
     # create totals
     total_active_user_count = 0
@@ -375,7 +380,7 @@ def ad_hoc_queries():
         cursor = connection.cursor()
         cursor.execute(query)
         rows = cursor.fetchall()
-        rows = map(list, rows)
+        rows = list(map(list, rows))
         cursor.close()
 
         def fix_rows(i, fixup_func):
@@ -609,7 +614,7 @@ def raw_user_activity_table(records):
                 format_date_for_activity_reports(record.last_visit)
         ]
 
-    rows = map(row, records)
+    rows = list(map(row, records))
     title = 'Raw Data'
     return make_table(title, cols, rows)
 

api/README.md

@@ -31,11 +31,24 @@ file is as follows:
     key=<api key from the web interface>
     email=<your email address>
     site=<your Zulip server's URI>
+    insecure=<true or false, true means do not verify the server certificate>
+    cert_bundle=<path to a file containing CA or server certificates to trust>
+
+If omitted, these settings have the following defaults:
+
+    site=https://api.zulip.com
+    insecure=false
+    cert_bundle=<the default CA bundle trusted by Python>
 
 Alternatively, you may explicitly use "--user" and "--api-key" in our
 examples, which is especially useful if you are running several bots
-which share a home directory.  There is also a "--site" option for
+which share a home directory.
-setting the Zulip server on the command line.
+
+The command line equivalents for other configuration options are:
+
+    --site=<your Zulip server's URI>
+    --insecure
+    --cert-bundle=<file>
 
 You can obtain your Zulip API key, create bots, and manage bots all
 from your Zulip [settings page](https://zulip.com/#settings).
@@ -101,3 +114,46 @@ Alternatively, if you don't want to use your ~/.zuliprc file:
         --api-key a0b1c2d3e4f5a6b7c8d9e0f1a2b3c4d5 \
         hamlet@example.com cordelia@example.com -m \
         "Conscience doth make cowards of us all."
+
+#### Working with an untrusted server certificate
+
+If your server has either a self-signed certificate, or a certificate signed
+by a CA that you don't wish to globally trust then by default the API will
+fail with an SSL verification error.
+
+You can add `insecure=true` to your .zuliprc file.
+
+    [api]
+    site=https://zulip.example.com
+    insecure=true
+
+This disables verification of the server certificate, so connections are
+encrypted but unauthenticated. This is not secure, but may be good enough
+for a development environment.
+
+
+You can explicitly trust the server certificate using `cert_bundle=<filename>`
+in your .zuliprc file.
+
+    [api]
+    site=https://zulip.example.com
+    cert_bundle=/home/bots/certs/zulip.example.com.crt
+
+You can also explicitly trust a different set of Certificate Authorities from
+the default bundle that is trusted by Python. For example to trust a company
+internal CA.
+
+    [api]
+    site=https://zulip.example.com
+    cert_bundle=/home/bots/certs/example.com.ca-bundle
+
+Save the server certificate (or the CA certificate) in its own file,
+converting to PEM format first if necessary.
+Verify that the certificate you have saved is the same as the one on the
+server.
+
+The `cert_bundle` option trusts the server / CA certificate only for
+interaction with the zulip site, and is relatively secure.
+
+Note that a certificate bundle is merely one or more certificates combined
+into a single file.

api/bin/zulip-send

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 # zulip-send -- Sends a message to the specified recipients.
 

api/examples/create-user

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012-2014 Zulip, Inc.

api/examples/edit-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/get-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/list-members

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2014 Zulip, Inc.

api/examples/list-subscriptions

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-events

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-messages

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/print-next-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/send-message

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/subscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/examples/unsubscribe

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
 # Copyright © 2012 Zulip, Inc.

api/integrations/asana/zulip_asana_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/asana/zulip_asana_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Asana integration for Zulip

api/integrations/basecamp/zulip_basecamp_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/basecamp/zulip_basecamp_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Basecamp activity

api/integrations/codebase/zulip_codebase_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/codebase/zulip_codebase_mirror

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Zulip mirror of Codebase HQ activity

api/integrations/git/post-receive

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-receive hook.

api/integrations/git/zulip_git_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/hg/zulip-changegroup.py

@@ -1,8 +1,8 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Zulip hook for Mercurial changeset pushes.
-# Copyright © 2012-2014 Zulip, Inc.
+# Copyright © 2012-2014 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -25,8 +25,10 @@
 #
 # This hook is called when changesets are pushed to the master repository (ie
 # `hg push`). See https://zulip.com/integrations for installation instructions.
+from __future__ import absolute_import
 
 import zulip
+from six.moves import range
 
 VERSION = "0.9"
 

api/integrations/nagios/nagios-notify-zulip

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import optparse
 import zulip
 

api/integrations/perforce/zulip_change-commit.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2012-2014 Zulip, Inc.
@@ -33,6 +33,7 @@ For example:
   1234 //depot/security/src/
 
 '''
+from __future__ import print_function
 
 import os
 import sys
@@ -59,12 +60,12 @@ try:
     changelist = int(sys.argv[1])
     changeroot = sys.argv[2]
 except IndexError:
-    print >> sys.stderr, "Wrong number of arguments.\n\n",
+    print("Wrong number of arguments.\n\n", end=' ', file=sys.stderr)
-    print >> sys.stderr,  __doc__
+    print(__doc__, file=sys.stderr)
     sys.exit(-1)
 except ValueError:
-    print >> sys.stderr, "First argument must be an integer.\n\n",
+    print("First argument must be an integer.\n\n", end=' ', file=sys.stderr)
-    print >> sys.stderr, __doc__
+    print(__doc__, file=sys.stderr)
     sys.exit(-1)
 
 metadata = git_p4.p4_describe(changelist)

api/integrations/perforce/zulip_perforce_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/rss/rss-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # RSS integration for Zulip

api/integrations/svn/post-commit

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Zulip notification post-commit hook.

api/integrations/svn/zulip_svn_config.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Copyright © 2014 Zulip, Inc.

api/integrations/trac/zulip_trac.py

@@ -107,7 +107,7 @@ class ZulipPlugin(Component):
         field_changes = []
         for key in old_values.keys():
             if key == "description":
-                content += '- Changed %s from %s to %s' % (key, markdown_block(old_values.get(key)),
+                content += '- Changed %s from %s\n\nto %s' % (key, markdown_block(old_values.get(key)),
                                                               markdown_block(ticket.values.get(key)))
             elif old_values.get(key) == "":
                 field_changes.append('%s: => **%s**' % (key, ticket.values.get(key)))

api/integrations/twitter/twitter-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Twitter integration for Zulip

api/integrations/twitter/twitter-search-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 #
 # Twitter search integration for Zulip

api/setup.py

@@ -1,6 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # -*- coding: utf-8 -*-
 
+from __future__ import print_function
 import os
 import sys
 
@@ -9,8 +10,8 @@ import itertools
 def version():
     version_py = os.path.join(os.path.dirname(__file__), "zulip", "__init__.py")
     with open(version_py) as in_handle:
-        version_line = itertools.dropwhile(lambda x: not x.startswith("__version__"),
+        version_line = next(itertools.dropwhile(lambda x: not x.startswith("__version__"),
-                                           in_handle).next()
+                                           in_handle))
     version = version_line.split('=')[-1].strip().replace('"', '')
     return version
 
@@ -60,13 +61,13 @@ except ImportError:
     try:
         import simplejson
     except ImportError:
-        print >>sys.stderr, "simplejson is not installed"
+        print("simplejson is not installed", file=sys.stderr)
         sys.exit(1)
     try:
         import requests
         assert(LooseVersion(requests.__version__) >= LooseVersion('0.12.1'))
     except (ImportError, AssertionError):
-        print >>sys.stderr, "requests >=0.12.1 is not installed"
+        print("requests >=0.12.1 is not installed", file=sys.stderr)
         sys.exit(1)
 
 

api/zulip/__init__.py

@@ -20,6 +20,8 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
+from __future__ import print_function
+from __future__ import absolute_import
 import simplejson
 import requests
 import time
@@ -33,8 +35,9 @@ import urllib
 import random
 from distutils.version import LooseVersion
 
-from ConfigParser import SafeConfigParser
+from six.moves.configparser import SafeConfigParser
 import logging
+import six
 
 
 __version__ = "0.2.4"
@@ -87,7 +90,7 @@ class RandomExponentialBackoff(CountingBackoff):
         try:
             logger.warning(message)
         except NameError:
-            print message
+            print(message)
         time.sleep(delay)
 
 def _default_client():
@@ -117,6 +120,19 @@ def generate_option_group(parser, prefix=''):
                      default=None,
                      dest="zulip_client",
                      help=optparse.SUPPRESS_HELP)
+    group.add_option('--insecure',
+                     action='store_true',
+                     dest='insecure',
+                     help='''Do not verify the server certificate.
+                          The https connection will not be secure.''')
+    group.add_option('--cert-bundle',
+                     action='store',
+                     dest='cert_bundle',
+                     help='''Specify a file containing either the
+                          server certificate, or a set of trusted
+                          CA certificates. This will be used to
+                          verify the server's identity. All
+                          certificates should be PEM encoded.''')
     return group
 
 def init_from_options(options, client=None):
@@ -126,7 +142,8 @@ def init_from_options(options, client=None):
         client = _default_client()
     return Client(email=options.zulip_email, api_key=options.zulip_api_key,
                   config_file=options.zulip_config_file, verbose=options.verbose,
-                  site=options.zulip_site, client=client)
+                  site=options.zulip_site, client=client,
+                  cert_bundle=options.cert_bundle, insecure=options.insecure)
 
 def get_default_config_filename():
     config_file = os.path.join(os.environ["HOME"], ".zuliprc")
@@ -138,15 +155,14 @@ def get_default_config_filename():
 class Client(object):
     def __init__(self, email=None, api_key=None, config_file=None,
                  verbose=False, retry_on_errors=True,
-                 site=None, client=None):
+                 site=None, client=None,
+                 cert_bundle=None, insecure=None):
         if client is None:
             client = _default_client()
-        if None in (api_key, email):
+
         if config_file is None:
             config_file = get_default_config_filename()
-            if not os.path.exists(config_file):
+        if os.path.exists(config_file):
-                raise RuntimeError("api_key or email not specified and %s does not exist"
-                                   % (config_file,))
             config = SafeConfigParser()
             with file(config_file, 'r') as f:
                 config.readfp(f, config_file)
@@ -156,6 +172,22 @@ class Client(object):
                 email = config.get("api", "email")
             if site is None and config.has_option("api", "site"):
                 site = config.get("api", "site")
+            if cert_bundle is None and config.has_option("api", "cert_bundle"):
+                cert_bundle = config.get("api", "cert_bundle")
+            if insecure is None and config.has_option("api", "insecure"):
+                # Be quite strict about what is accepted so that users don't
+                # disable security unintentionally.
+                insecure_setting = config.get("api", "insecure").lower()
+                if insecure_setting == "true":
+                    insecure = True
+                elif insecure_setting == "false":
+                    insecure = False
+                else:
+                    raise RuntimeError("insecure is set to '%s', it must be 'true' or 'false' if it is used in %s"
+                                       % (insecure_setting, config_file))
+        elif None in (api_key, email):
+            raise RuntimeError("api_key or email not specified and %s does not exist"
+                               % (config_file,))
 
         self.api_key = api_key
         self.email = email
@@ -175,6 +207,17 @@ class Client(object):
         self.retry_on_errors = retry_on_errors
         self.client_name = client
 
+        if insecure:
+            self.tls_verification=False
+        elif cert_bundle is not None:
+            if not os.path.isfile(cert_bundle):
+                raise RuntimeError("tls bundle '%s' does not exist"
+                                   %(cert_bundle,))
+            self.tls_verification=cert_bundle
+        else:
+            # Default behavior: verify against system CA certificates
+            self.tls_verification=True
+
     def get_user_agent(self):
         vendor = ''
         vendor_version = ''
@@ -203,7 +246,7 @@ class Client(object):
         request = {}
 
         for (key, val) in orig_request.iteritems():
-            if not (isinstance(val, str) or isinstance(val, unicode)):
+            if not (isinstance(val, str) or isinstance(val, six.text_type)):
                 request[key] = simplejson.dumps(val)
             else:
                 request[key] = val
@@ -233,9 +276,9 @@ class Client(object):
         def end_error_retry(succeeded):
             if query_state["had_error_retry"] and self.verbose:
                 if succeeded:
-                    print "Success!"
+                    print("Success!")
                 else:
-                    print "Failed!"
+                    print("Failed!")
 
         while True:
             try:
@@ -249,7 +292,7 @@ class Client(object):
                         urlparse.urljoin(self.base_url, url),
                         auth=requests.auth.HTTPBasicAuth(self.email,
                                                          self.api_key),
-                        verify=True, timeout=90,
+                        verify=self.tls_verification, timeout=90,
                         headers={"User-agent": self.get_user_agent()},
                         **kwargs)
 
@@ -311,7 +354,7 @@ class Client(object):
             else:
                 req_url = url
             return self.do_api_query(request, API_VERSTRING + req_url, method=method, **query_kwargs)
-        call.func_name = name
+        call.__name__ = name
         setattr(cls, name, call)
 
     def call_on_each_event(self, callback, event_types=None, narrow=[]):
@@ -324,7 +367,7 @@ class Client(object):
 
                 if 'error' in res.get('result'):
                     if self.verbose:
-                        print "Server returned error:\n%s" % res['msg']
+                        print("Server returned error:\n%s" % res['msg'])
                     time.sleep(1)
                 else:
                     return (res['queue_id'], res['last_event_id'])
@@ -338,13 +381,13 @@ class Client(object):
             if 'error' in res.get('result'):
                 if res["result"] == "http-error":
                     if self.verbose:
-                        print "HTTP error fetching events -- probably a server restart"
+                        print("HTTP error fetching events -- probably a server restart")
                 elif res["result"] == "connection-error":
                     if self.verbose:
-                        print "Connection error fetching events -- probably server is temporarily down?"
+                        print("Connection error fetching events -- probably server is temporarily down?")
                 else:
                     if self.verbose:
-                        print "Server returned error:\n%s" % res["msg"]
+                        print("Server returned error:\n%s" % res["msg"])
                     if res["msg"].startswith("Bad event queue id:"):
                         # Our event queue went away, probably because
                         # we were asleep or the server restarted

assets/favicon/generate

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import xml.etree.ElementTree as ET
 import subprocess
 

bin/get-django-setting

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 from __future__ import absolute_import
 
 import os

bin/log-management-command

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import sys
 import logging
 import os

bots/check-mirroring

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import sys
 import time
 import optparse

bots/check-rabbitmq-consumers

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 
 import sys
 import time

bots/check-rabbitmq-queue

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 
 import sys
 import re

bots/cron_file_helper.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import time
 
 def nagios_from_file(results_file):

bots/gcal-bot

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import sys
 import time
 import datetime

bots/irc-mirror.py

@@ -1,4 +1,4 @@
-#! /usr/bin/env python
+#! /usr/bin/env python2.7
 #
 # EXPERIMENTAL
 # IRC <=> Zulip mirroring bot
@@ -6,6 +6,7 @@
 # Setup: First, you need to install python-irc version 8.5.3
 # (https://bitbucket.org/jaraco/irc)
 
+from __future__ import print_function
 import irc.bot
 import irc.strings
 from irc.client import ip_numstr_to_quad, ip_quad_to_numstr
@@ -53,12 +54,12 @@ class IRCBot(irc.bot.SingleServerIRCBot):
             return
 
         # Forward the PM to Zulip
-        print zulip_client.send_message({
+        print(zulip_client.send_message({
                 "sender": sender,
                 "type": "private",
                 "to": "username@example.com",
                 "content": content,
-                })
+                }))
 
     def on_pubmsg(self, c, e):
         content = e.arguments[0]
@@ -68,14 +69,14 @@ class IRCBot(irc.bot.SingleServerIRCBot):
             return
 
         # Forward the stream message to Zulip
-        print zulip_client.send_message({
+        print(zulip_client.send_message({
                 "forged": "yes",
                 "sender": sender,
                 "type": "stream",
                 "to": stream,
                 "subject": "IRC",
                 "content": content,
-                })
+                }))
 
     def on_dccmsg(self, c, e):
         c.privmsg("You said: " + e.arguments[0])
@@ -92,11 +93,11 @@ class IRCBot(irc.bot.SingleServerIRCBot):
                 return
             self.dcc_connect(address, port)
 
-usage = """python irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
+usage = """python2.7 irc-mirror.py --server=IRC_SERVER --channel=<CHANNEL> --nick-prefix=<NICK> [optional args]
 
 Example:
 
-python irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
+python2.7 irc-mirror.py --irc-server=127.0.0.1 --channel='#test' --nick-prefix=username
   --site=https://zulip.example.com --user=irc-bot@example.com
   --api-key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 

bots/jabber_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # Copyright (C) 2014 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -21,6 +21,7 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+from __future__ import print_function
 import sys
 import subprocess
 import os
@@ -39,7 +40,7 @@ args.extend(sys.argv[1:])
 
 backoff = RandomExponentialBackoff(timeout_success_equivalent=300)
 while backoff.keep_going():
-    print "Starting Jabber mirroring bot"
+    print("Starting Jabber mirroring bot")
     try:
         ret = subprocess.call(args)
     except:
@@ -51,9 +52,9 @@ while backoff.keep_going():
 
     backoff.fail()
 
-print ""
+print("")
-print ""
+print("")
-print "ERROR: The Jabber mirroring bot is unable to continue mirroring Jabber."
+print("ERROR: The Jabber mirroring bot is unable to continue mirroring Jabber.")
-print "Please contact zulip-devel@googlegroups.com if you need assistance."
+print("Please contact zulip-devel@googlegroups.com if you need assistance.")
-print ""
+print("")
 sys.exit(1)

bots/jabber_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python2.7
 #
 # Copyright (C) 2013 Permabit, Inc.
 # Copyright (C) 2013--2014 Zulip, Inc.
@@ -44,7 +44,7 @@ import optparse
 
 from sleekxmpp import ClientXMPP, InvalidJID, JID
 from sleekxmpp.exceptions import IqError, IqTimeout
-from ConfigParser import SafeConfigParser
+from six.moves.configparser import SafeConfigParser
 import os, sys, zulip, getpass
 import re
 

bots/log2zulip

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python2.7
 import subprocess
 import os
 import sys

bots/process_ccache

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python2.7
 import sys
 import subprocess
 import base64

bots/summarize_stream.py

@@ -1,3 +1,4 @@
+from __future__ import print_function
 # This is hacky code to analyze data on our support stream.  The main
 # reusable bits are get_recent_messages and get_words.
 
@@ -31,7 +32,7 @@ def analyze_messages(msgs, word_count, email_count):
         if False:
             if ' ack' in msg['content']:
                 name = msg['sender_full_name'].split()[0]
-                print 'ACK', name
+                print('ACK', name)
         m = re.search('ticket (Z....).*email: (\S+).*~~~(.*)', msg['content'], re.M | re.S)
         if m:
             ticket, email, req = m.groups()
@@ -40,9 +41,9 @@ def analyze_messages(msgs, word_count, email_count):
                 word_count[word] += 1
             email_count[email] += 1
         if False:
-            print
+            print()
             for k, v in msg.items():
-                print '%-20s: %s' % (k, v)
+                print('%-20s: %s' % (k, v))
 
 def generate_support_stats():
     client = zulip.Client()
@@ -64,16 +65,16 @@ def generate_support_stats():
 
     if True:
         words = word_count.keys()
-        words = filter(lambda w: word_count[w] >= 10, words)
+        words = [w for w in words if word_count[w] >= 10]
-        words = filter(lambda w: len(w) >= 5, words)
+        words = [w for w in words if len(w) >= 5]
         words = sorted(words, key=lambda w: word_count[w], reverse=True)
         for word in words:
-            print word, word_count[word]
+            print(word, word_count[word])
 
     if False:
         emails = email_count.keys()
         emails = sorted(emails, key=lambda w: email_count[w], reverse=True)
         for email in emails:
-            print email, email_count[email]
+            print(email, email_count[email])
 
 generate_support_stats()

bots/sync-public-streams

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 import sys
 import os
 import logging

bots/tddium-notify-humbug

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person

bots/zephyr_mirror.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -21,13 +21,15 @@
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+from __future__ import absolute_import
+from __future__ import print_function
 import sys
 import subprocess
 import os
 import traceback
 import signal
 
-from zephyr_mirror_backend import parse_args
+from .zephyr_mirror_backend import parse_args
 
 def die(signal, frame):
     # We actually want to exit, so run os._exit (so as not to be caught and restarted)
@@ -52,30 +54,30 @@ if options.forward_class_messages and not options.noshard:
     if options.on_startup_command is not None:
         subprocess.call([options.on_startup_command])
     from zerver.lib.parallel import run_parallel
-    print "Starting parallel zephyr class mirroring bot"
+    print("Starting parallel zephyr class mirroring bot")
     jobs = list("0123456789abcdef")
     def run_job(shard):
         subprocess.call(args + ["--shard=%s" % (shard,)])
         return 0
     for (status, job) in run_parallel(run_job, jobs, threads=16):
-        print "A mirroring shard died!"
+        print("A mirroring shard died!")
         pass
     sys.exit(0)
 
 backoff = RandomExponentialBackoff(timeout_success_equivalent=300)
 while backoff.keep_going():
-    print "Starting zephyr mirroring bot"
+    print("Starting zephyr mirroring bot")
     try:
         subprocess.call(args)
     except:
         traceback.print_exc()
     backoff.fail()
 
-print ""
+print("")
-print ""
+print("")
-print "ERROR: The Zephyr mirroring bot is unable to continue mirroring Zephyrs."
+print("ERROR: The Zephyr mirroring bot is unable to continue mirroring Zephyrs.")
-print "This is often caused by failing to maintain unexpired Kerberos tickets"
+print("This is often caused by failing to maintain unexpired Kerberos tickets")
-print "or AFS tokens.  See https://zulip.com/zephyr for documentation on how to"
+print("or AFS tokens.  See https://zulip.com/zephyr for documentation on how to")
-print "maintain unexpired Kerberos tickets and AFS tokens."
+print("maintain unexpired Kerberos tickets and AFS tokens.")
-print ""
+print("")
 sys.exit(1)

bots/zephyr_mirror_backend.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 # Copyright (C) 2012 Zulip, Inc.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -20,8 +20,11 @@
 # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
+from __future__ import absolute_import
 
 import sys
+from six.moves import map
+from six.moves import range
 try:
     import simplejson
 except ImportError:
@@ -41,8 +44,8 @@ import select
 
 DEFAULT_SITE = "https://api.zulip.com"
 
-class States:
+class States(object):
-    Startup, ZulipToZephyr, ZephyrToZulip, ChildSending = range(4)
+    Startup, ZulipToZephyr, ZephyrToZulip, ChildSending = list(range(4))
 CURRENT_STATE = States.Startup
 
 def to_zulip_username(zephyr_username):

changelog.md

@@ -2,9 +2,27 @@
 
 All notable changes to this project will be documented in this file.
 
-## [Unreleased]
+[Unreleased]
 
-###
+[1.3.9] - 2015-11-16
+- Fixed buggy #! lines in upgrade scripts.
+
+[1.3.8] - 2015-11-15
+- Added options to the Python api for working with untrusted server certificates.
+- Added a lot of documentation on the development environment and testing.
+- Added partial support for translating the Zulip UI.
+- Migrated installing Node dependencies to use npm.
+- Fixed LDAP integration breaking autocomplete of @-mentions.
+- Fixed admin panel reactivation/deactivation of bots.
+- Fixed inaccurate documentation for downloading the desktop apps.
+- Fixed various minor bugs in production installation process.
+- Fixed security issue where recent history on private streams might
+  be visible to new users (to the Zulip team) who were invited with that
+  private stream as one of their initial streams
+  (https://github.com/zulip/zulip/issues/230).
+- Major preliminary progress towards supporting Python 3.
+
+[1.3.7] - 2015-10-19
 - Turn off desktop and audible notifications for streams by default.
 - Added support for the LDAP authentication integration creating new users.
 - Added new endpoint to support Google auth on mobile.

confirmation/models.py

@@ -105,7 +105,7 @@ class Confirmation(models.Model):
 
     objects = ConfirmationManager()
 
-    class Meta:
+    class Meta(object):
         verbose_name = _('confirmation email')
         verbose_name_plural = _('confirmation emails')
 

docs/code-style.rst

@@ -352,8 +352,8 @@ styles (separate lines for each selector)::
 Python
 ------
 
--  Scripts should start with ``#!/usr/bin/env python`` and not
+-  Scripts should start with ``#!/usr/bin/env python2.7`` and not
-   ``#!/usr/bin/python``. See commit ``437d4aee`` for an explanation of
+   ``#!/usr/bin/env python2.7``. See commit ``437d4aee`` for an explanation of
    why. Don't put such a line on a Python file unless it's meaningful to
    run it as a script. (Some libraries can also be run as scripts, e.g.
    to run a test suite.)

docs/directory-structure.rst

@@ -76,11 +76,11 @@ Tests
 =====
 
 +------------------------+-----------------------------------+
-| ``zerver/test*.py``             | Backend tests            |       |
+| ``zerver/test*.py``             | Backend tests            |
 +------------------------+-----------------------------------+
-| ``zerver/tests/frontend/node``  | Node Frontend unit tests |
+| ``frontend_tests/node``         | Node Frontend unit tests |
 +------------------------+-----------------------------------+
-| ``zerver/tests/frontend/tests`` | Casper frontend tests    |
+| ``frontend_tests/tests``        | Casper frontend tests    |
 +------------------------+-----------------------------------+
 
 Documentation

docs/html_unescape.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python2.7
 
 # Remove HTML entity escaping left over from MediaWiki->rST conversion.
 

docs/testing.rst

@@ -56,10 +56,10 @@ Backend Django tests
 These live in ``zerver/tests.py`` and ``zerver/test_*.py``. Run them
 with ``tools/test-backend``.
 
-Web frontend black-box tests
+Web frontend black-box casperjs tests
-----------------------------
+-------------------------------------
 
-These live in ``zerver/tests/frontend/tests/``. This is a "black box"
+These live in ``frontend_tests/casper_tests/``. This is a "black box"
 test; we load the frontend in a real (headless) browser, from a real dev
 server, and simulate UI interactions like sending messages, narrowing,
 etc.
@@ -67,8 +67,63 @@ etc.
 Since this is interacting with a real dev server, it can catch backend
 bugs as well.
 
-You can run this with ``./zerver/tests/frontend/run``. You will need
+You can run this with ``./tools/test-js-with-casper`` or as
-`PhantomJS <http://phantomjs.org/>`__ 1.7.0 or later.
+``./tools/test-js-with-casper 05-settings.js`` to run a single test
+file from ``frontend_tests/casper_tests/``.
+
+Writing Casper tests
+~~~~~~~~~~~~~~~~~~~~
+
+Probably the easiest way to learn how to write Casper tests is to
+study some of the existing test files.  There are a few tips that can
+be useful for writing Casper tests in addition to the debugging notes
+below:
+
+- Run just the file containing your new tests as described above to
+  have a fast debugging cycle.
+- With frontend tests in general, it's very important to write your
+  code to wait for the right events.  Before essentially every action
+  you take on the page, you'll want to use ``waitForSelector``,
+  ``waitUntilVisible``, or a similar function to make sure the page or
+  elemant is ready before you interact with it. For instance, if you
+  want to click a button that you can select via ``#btn-submit``, and
+  then check that it causes ``success-elt`` to appear, you'll want to
+  write something like:
+
+  ::
+
+    casper.waitForSelector("#btn-submit", function () {
+       casper.click('#btn-submit')
+       casper.test.assertExists("#success-elt");
+     });
+
+  This will ensure that the element is present before the interaction
+  is attempted.  The various wait functions supported in Casper are
+  documented in the Casper here:
+  http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector
+  and the various assert statements available are documented here:
+  http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype
+- Casper uses CSS3 selectors; you can often save time by testing and
+  debugigng your selectors on the relevant page of the Zulip
+  development app in the Chrome javascript console by using
+  e.g. ``$$("#settings-dropdown")``.
+- The test suite uses a smaller set of default user accounts and other
+  data initialized in the database than the development environment;
+  to see what differs check out the section related to
+  ``options["test_suite"]`` in
+  ``zilencer/management/commands/populate_db.py``.
+- Casper effectively runs your test file in two phases -- first it
+  runs the code in the test file, which for most test files will just
+  collect a series of steps (each being a ``casper.then`` or
+  ``casper.wait...`` call).  Then, usually at the end of the test
+  file, you'll have a ``casper.run`` call which actually runs that
+  series of steps.  This means that if you If you write code in your
+  test file outside a ``casper.then`` or ``casper.wait...`` method, it
+  will actually run before all the Casper test steps that are declared
+  in the file, which can lead to confusing failures where the new code
+  you write in between two ``casper.then`` blocks actually runs before
+  either of them.  See this for more details about how Casper works:
+  http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work
 
 Debugging Casper.JS
 ~~~~~~~~~~~~~~~~~~~
@@ -78,7 +133,7 @@ is not perfect. Here are some steps for using it and gotchas you might
 want to know.
 
 To turn on remote debugging, pass ``--remote-debug`` to the
-``./zerver/frontend/tests/run`` script. This will run the tests with
+``./frontend_tests/tests/run`` script. This will run the tests with
 port ``7777`` open for remote debugging. You can now connect to
 ``localhost:7777`` in a Webkit browser. Somewhat recent versions of
 Chrome or Safari might be required.
@@ -97,6 +152,23 @@ you set a breakpoint and it is hit, the inspector will pause and you can
 do your normal JS debugging. You can also put breakpoints in the Zulip
 webpage itself if you wish to inspect the state of the Zulip frontend.
 
+If you need to use print debugging in casper, you can do using
+``casper.log``; see http://docs.casperjs.org/en/latest/logging.html
+for details.
+
+An additional debugging technique is to enable verbose mode in the
+Casper tests; you can do this by adding to the top of the relevant
+test file the following:
+
+  ::
+
+     var casper = require('casper').create({
+        verbose: true,
+        logLevel: "debug"
+     });
+
+This can sometimes give insight into exactly what's happening.
+
 Web frontend unit tests
 -----------------------
 
@@ -114,7 +186,7 @@ bottom of ``foobar.js``:
 This makes ``foobar.js`` follow the CommonJS module pattern, so it can
 be required in Node.js, which runs our tests.
 
-Now create ``zerver/tests/frontend/node/foobar.js``. At the top, require
+Now create ``frontend_tests/node_tests/foobar.js``. At the top, require
 the `Node.js assert module <http://nodejs.org/api/assert.html>`__, and
 the module you're testing, like so:
 
@@ -140,12 +212,12 @@ asserts, the *actual* value comes first, the *expected* value second.
      }());
 
 The test runner (index.js) automatically runs all .js files in the
-zerver/tests/frontend/node directory.
+frontend_tests/node directory.
 
 .. _handling-dependencies:
 
-Handling dependencies in tests
+Handling dependencies in unit tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The following scheme helps avoid tests leaking globals between each
 other.

frontend_tests/casper_lib/.gitignore

@@ -0,0 +1,2 @@
+/server.log
+/test_credentials.js

frontend_tests/casper_lib/common.js

@@ -2,7 +2,7 @@ var common = (function () {
 
 var exports = {};
 
-var test_credentials = require('../test_credentials.js').test_credentials;
+var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
 
 function timestamp() {
     return new Date().getTime();

frontend_tests/casper_tests/00-login.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 // Start of test script.
 casper.start('http://localhost:9981/', common.initialize_casper);

frontend_tests/casper_tests/01-site.js

@@ -10,7 +10,7 @@
 // For example, utils.dump() prints an Object with nice formatting.
 var utils = require('utils');
 
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/02-narrow.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/03-compose.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/04-subscriptions.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/05-settings.js

@@ -1,5 +1,5 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
-var test_credentials = require('../test_credentials.js').test_credentials;
+var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/07-stars.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 function star_count() {
     return casper.evaluate(function () {

frontend_tests/casper_tests/09-edit.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 

frontend_tests/casper_tests/10-navigation.js

@@ -1,4 +1,4 @@
-var common = require('../common.js').common;
+var common = require('../casper_lib/common.js').common;
 
 // Test basic tab navigation.
 

frontend_tests/casper_tests/11-admin.js

@@ -0,0 +1,77 @@
+var common = require('../casper_lib/common.js').common;
+var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
+
+common.start_and_log_in();
+
+casper.then(function () {
+    casper.test.info('Administration page');
+    casper.click('a[href^="#administration"]');
+    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#administration/, 'URL suggests we are on administration page');
+    casper.test.assertExists('#administration.tab-pane.active', 'Administration page is active');
+});
+
+// Test user deactivation and reactivation
+casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
+    casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
+    casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
+    casper.click('#do_deactivate_user_button');
+});
+
+casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"].deactivated_user', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
+    casper.click('.user_row[id="user_cordelia@zulip.com"] .reactivate');
+});
+
+casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]:not(.deactivated_user)', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+});
+
+// Test Deactivated users section of admin page
+casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
+    casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
+    casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
+    casper.click('#do_deactivate_user_button');
+});
+
+casper.then(function () {
+    // Leave the page and return
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#subscriptions"]');
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+
+    casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
+        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
+        casper.click('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] .reactivate');
+    });
+
+    casper.waitForSelector('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]:not(.deactivated_user)', function () {
+        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    });
+});
+
+// Test bot deactivation and reactivation
+casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
+    casper.click('.user_row[id="user_new-user-bot@zulip.com"] .deactivate');
+});
+
+casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"].deactivated_user', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Reactivate');
+    casper.click('.user_row[id="user_new-user-bot@zulip.com"] .reactivate');
+});
+casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]:not(.deactivated_user)', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
+});
+
+// TODO: Test stream deletion
+
+common.then_log_out();
+
+casper.run(function () {
+    casper.test.done();
+});

frontend_tests/casperjs/bin/casperjs

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7
 
 import json
 import os