Update changelog for Zulip 1.4.3 release.

A bug in Zulip's implementation of the "stream exists" endpoint meant
that any user of a Zulip server could subscribe to an invite-only
stream without needing to be invited by using the "autosubscribe"
argument.

Thanks to Rafid Aslam for discovering this issue.

.gitignore

@@ -1,48 +1,26 @@
 *.pyc
 *~
-/all_messages_log.*
-/event_log/*
-/digest.log*
-/errors.log*
-/manage.log*
-/server.log*
-/workers.log*
-/email-deliverer.log
-/email-mirror.log
-/sync_ldap_user_data.log
-/update-prod-static.log
-frontend_tests/casper_tests/server.log
-frontend_tests/casper_lib/test_credentials.js
-memcached_prefix
 /prod-static
 /errors/*
 *.sw[po]
 *.DS_Store
-event_queues.pickle
 stats/
-zerver/fixtures/available-migrations
-zerver/fixtures/migration-status
-zerver/fixtures/test_data1.json
 .kdev4
+.idea
 zulip.kdev4
-remote_cache_prefix
 coverage/
+.coverage
 /queue_error
-.test-js-with-node.html
 .kateproject.d/
 .kateproject
 *.kate-swp
-event_queues.json
 .vagrant
 /zproject/dev-secrets.conf
 static/js/bundle.js
 static/third/gemoji/
 static/third/zxcvbn/
-tools/setup/emoji_dump/bitmaps/
-tools/setup/emoji_dump/*.ttx
-tools/phantomjs
+static/locale/language_options.json
 node_modules
 npm-debug.log
-uploads/
-test_uploads/
 *.mo
+var/*

.travis.yml

@@ -1,3 +1,4 @@
+dist: trusty
 before_install:
    - nvm install 0.10
 install:
@@ -16,20 +17,24 @@ env:
     - COVERALLS_PARALLEL=true
     - COVERALLS_SERVICE_NAME=travis-pro
     - COVERALLS_REPO_TOKEN=hnXUEBKsORKHc8xIENGs9JjktlTb2HKlG
+    - BOTO_CONFIG=/tmp/nowhere
   matrix:
     - TEST_SUITE=frontend
     - TEST_SUITE=backend
-    - TEST_SUITE=production
-    - TEST_SUITE=py3k
 language: python
 python:
   - "2.7"
+  - "3.4"
 matrix:
   include:
     - python: "3.4"
-      env: TEST_SUITE=mypy
+      env: TEST_SUITE=static-analysis
+    - python: "2.7"
+      env: TEST_SUITE=production
+      sudo: required
 # command to run tests
 script:
+  - unset GEM_PATH
   - ./tools/travis/$TEST_SUITE
 sudo: required
 services:

.tx/config

@@ -2,10 +2,10 @@
 host = https://www.transifex.com
 
 [zulip.djangopo]
-source_file = locale/en/LC_MESSAGES/django.po
+source_file = static/locale/en/LC_MESSAGES/django.po
 source_lang = en
 type = PO
-file_filter = locale/<lang>/LC_MESSAGES/django.po
+file_filter = static/locale/<lang>/LC_MESSAGES/django.po
 lang_map = zh-Hans: zh_CN
 
 [zulip.translationsjson]

README.md

@@ -1,4 +1,5 @@
 **[Zulip overview](#zulip-overview)** |
+**[Community](#community)** |
 **[Installing for dev](#installing-the-zulip-development-environment)** |
 **[Installing for production](#running-zulip-in-production)** |
 **[Ways to contribute](#ways-to-contribute)** |
@@ -20,18 +21,41 @@ at https://www.zulip.org.
 
 [![Build Status](https://travis-ci.org/zulip/zulip.svg?branch=master)](https://travis-ci.org/zulip/zulip) [![Coverage Status](https://coveralls.io/repos/github/zulip/zulip/badge.svg?branch=master)](https://coveralls.io/github/zulip/zulip?branch=master)
 
+## Community
+
+There are several places online where folks discuss Zulip.
+
+One of those places is our [public Zulip instance](https://zulip.tabbott.net/).
+You can go through the simple signup process at that link, and then you
+will soon be talking to core Zulip developers and other users.  To get
+help in real time, you will have the best luck finding core developers
+roughly between 16:00 UTC and 23:59 UTC.  Most questions get answered
+within a day.
+
+We have a [Google mailing list](https://groups.google.com/forum/#!forum/zulip-devel)
+that is currently pretty low traffic.  It is where we do things like
+announce public meetings or major releases.  You can also use it to
+ask questions about features or possible bugs.
+
+Last but not least, we use [GitHub](https://github.com/zulip/zulip) to
+track Zulip-related issues (and store our code, of course).
+Anybody with a Github account should be able to create Issues there
+pertaining to bugs or enhancement requests.  We also use Pull
+Requests as our primary mechanism to receive code contributions.
+
 ## Installing the Zulip Development environment
 
 The Zulip development environment is the recommended option for folks
-interested in trying out Zulip.  This is documented in
-[README.dev.md](https://github.com/zulip/zulip/blob/master/README.dev.md).
+interested in trying out Zulip.  This is documented in [the developer
+installation guide][dev-install].
 
 ## Running Zulip in production
 
 Zulip in production only supports Ubuntu 14.04 right now, but work is
 ongoing on adding support for additional platforms. The installation
 process is documented at https://zulip.org/server.html and in more
-detail in [README.prod.md](https://github.com/zulip/zulip/blob/master/README.prod.md).
+detail in [the
+documentation](https://zulip.readthedocs.io/en/latest/prod-install.html).
 
 ## Ways to contribute
 
@@ -45,7 +69,9 @@ please skim our [commit message style guidelines][doc-commit-style].
 * **Testing**. The Zulip automated tests all run automatically when
 you submit a pull request, but you can also run them all in your
 development environment following the instructions in the [testing
-docs][doc-test].
+docs][doc-test]. You can also try out [our new desktop
+client][electron], which is in alpha; we'd appreciate testing and
+[feedback](https://github.com/zulip/zulip-electron/issues/new).
 
 * **Developer Documentation**.  Zulip has a growing collection of
 developer documentation on [Read The Docs][doc].  Recommended reading
@@ -63,8 +89,9 @@ zulip-security@googlegroups.com.
 
 * **App codebases**. This repository is for the Zulip server and web
 app (including most integrations); the [desktop][], [Android][], and
-[iOS][] apps, are separate repositories, as is [our experimental React
-Native iOS app][ios-exp].
+[iOS][] apps, are separate repositories, as are our [experimental
+React Native iOS app][ios-exp] and [alpha Electron desktop
+app][electron].
 
 * **Glue code**. We maintain a [Hubot adapter][hubot-adapter] and several
 integrations ([Phabricator][phab], [Jenkins][], [Puppet][], [Redmine][],
@@ -73,15 +100,17 @@ PostgreSQL extension][tsearch], as separate repos.
 
 * **Translations**.  Zulip is in the process of being translated into
 10+ languages, and we love contributions to our translations.  See our
-[translating documentation](transifex) if you're interested in
+[translating documentation][transifex] if you're interested in
 contributing!
 
 [cla]: https://opensource.dropbox.com/cla/
+[dev-install]: https://zulip.readthedocs.io/en/latest/dev-overview.html
 [doc]: https://zulip.readthedocs.io/
 [doc-commit-style]: http://zulip.readthedocs.io/en/latest/code-style.html#commit-messages
 [doc-dirstruct]: http://zulip.readthedocs.io/en/latest/directory-structure.html
 [doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
-[doc-test]: https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
+[doc-test]: http://zulip.readthedocs.io/en/latest/testing.html
+[electron]: https://github.com/zulip/zulip-electron
 [gg-devel]: https://groups.google.com/forum/#!forum/zulip-devel
 [gh-issues]: https://github.com/zulip/zulip/issues
 [desktop]: https://github.com/zulip/zulip-desktop
@@ -98,7 +127,7 @@ contributing!
 [redmine]: https://github.com/zulip/zulip-redmine-plugin
 [trello]: https://github.com/zulip/trello-to-zulip
 [tsearch]: https://github.com/zulip/tsearch_extras
-[transifex]: https://www.transifex.com/zulip/zulip/
+[transifex]: https://zulip.readthedocs.io/en/latest/translating.html#testing-translations
 [z-org]: https://github.com/zulip/zulip.github.io
 
 ## How to get involved with contributing to Zulip

THIRDPARTY

@@ -42,13 +42,6 @@ Files: puppet/apt/*
 Copyright: 2011, Evolving Web Inc.
 License: Expat
 
-Files: puppet/common/*
-Copyright: 2007, David Schmitt
-License: BSD-3-Clause
-Comment: https://github.com/DavidS/puppet-common
- Distribution includes a file `lib/puppet/parser/functions/ip_to_cron.rb` which
- we removed due to unclear license
-
 Files: puppet/stdlib/*
 Copyright: 2011, Krzysztof Wilczynski
  2011, Puppet Labs Inc
@@ -182,6 +175,10 @@ Files: static/third/marked/*
 Copyright: 2011-2013, Christopher Jeffrey
 License: Expat
 
+Files: static/third/string-prototype-codepointat/*
+Copyright: 2014 Mathias Bynens
+License: Expat
+
 Files: static/third/sockjs/sockjs-0.3.4.js
 Copyright: 2011-2012 VMware, Inc.
  2012 Douglas Crockford

Vagrantfile

@@ -3,7 +3,7 @@
 VAGRANTFILE_API_VERSION = "2"
 
 def command?(name)
-  `which #{name}`
+  `which #{name} > /dev/null 2>&1`
   $?.success?
 end
 
@@ -13,16 +13,15 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.box = "fgrehm/trusty64-lxc"
 
   # The Zulip development environment runs on 9991 on the guest.
-  config.vm.network "forwarded_port", guest: 9991, host: 9991, host_ip: "127.0.0.1"
+  host_port = 9991
+  http_proxy = https_proxy = no_proxy = ""
 
   config.vm.synced_folder ".", "/vagrant", disabled: true
   config.vm.synced_folder ".", "/srv/zulip"
 
-  proxy_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
-  if File.file?(proxy_config_file)
-    http_proxy = https_proxy = no_proxy = ""
-
-    IO.foreach(proxy_config_file) do |line|
+  vagrant_config_file = ENV['HOME'] + "/.zulip-vagrant-config"
+  if File.file?(vagrant_config_file)
+    IO.foreach(vagrant_config_file) do |line|
       line.chomp!
       key, value = line.split(nil, 2)
       case key
@@ -30,19 +29,22 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
       when "HTTP_PROXY"; http_proxy = value
       when "HTTPS_PROXY"; https_proxy = value
       when "NO_PROXY"; no_proxy = value
+      when "HOST_PORT"; host_port = value.to_i
       end
     end
+  end
 
-    if Vagrant.has_plugin?("vagrant-proxyconf")
-      if http_proxy != ""
-        config.proxy.http = http_proxy
-      end
-      if https_proxy != ""
-        config.proxy.https = https_proxy
-      end
-      if https_proxy != ""
-        config.proxy.no_proxy = no_proxy
-      end
+  config.vm.network "forwarded_port", guest: 9991, host: host_port, host_ip: "127.0.0.1"
+
+  if Vagrant.has_plugin?("vagrant-proxyconf")
+    if http_proxy != ""
+      config.proxy.http = http_proxy
+    end
+    if https_proxy != ""
+      config.proxy.https = https_proxy
+    end
+    if https_proxy != ""
+      config.proxy.no_proxy = no_proxy
     end
   end
 
@@ -63,15 +65,16 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
   config.vm.provider "virtualbox" do |vb, override|
     override.vm.box = "ubuntu/trusty64"
-    # It's possible we can get away with just 1GB; more testing needed
-    vb.memory = 1280
+    # It's possible we can get away with just 1.5GB; more testing needed
+    vb.memory = 2048
   end
 
 $provision_script = <<SCRIPT
 set -x
 set -e
+set -o pipefail
 ln -nsf /srv/zulip ~/zulip
-/usr/bin/python /srv/zulip/provision.py
+/usr/bin/python /srv/zulip/tools/provision.py | sudo tee -a /var/log/zulip_provision.log
 SCRIPT
 
   config.vm.provision "shell",

analytics/management/commands/analyze_user_activity.py

@@ -55,7 +55,8 @@ is shown for all realms"""
     option_list = BaseCommand.option_list + (
         make_option('--realm', action='store'),
         make_option('--date', action='store', default="2013-09-06"),
-        make_option('--duration', action='store', default=1, type=int, help="How many days to show usage information for"),
+        make_option('--duration', action='store', default=1, type=int,
+                    help="How many days to show usage information for"),
         )
 
     def handle(self, *args, **options):

api/integrations/codebase/zulip_codebase_mirror

@@ -208,7 +208,8 @@ def handle_event(event):
             subject = "Discussion: %s" % (subj,)
 
             if category:
-                content = "%s started a new discussion in %s:\n\n~~~ quote\n%s\n~~~" % (actor_name, category, comment_content)
+                format_str = "%s started a new discussion in %s:\n\n~~~ quote\n%s\n~~~"
+                content = format_str % (actor_name, category, comment_content)
             else:
                 content = "%s posted:\n\n~~~ quote\n%s\n~~~" % (actor_name, comment_content)
 
@@ -223,7 +224,8 @@ def handle_event(event):
 
         start_ref_url = make_url("projects/%s/repositories/%s/commit/%s" % (project_link, repo_link, start_ref))
         end_ref_url = make_url("projects/%s/repositories/%s/commit/%s" % (project_link, repo_link, end_ref))
-        between_url = make_url("projects/%s/repositories/%s/compare/%s...%s" % (project_link, repo_link, start_ref, end_ref))
+        between_url = make_url("projects/%s/repositories/%s/compare/%s...%s" % (
+            project_link, repo_link, start_ref, end_ref))
 
         subject = "Deployment to %s" % (environment,)
 

api/integrations/hg/zulip-changegroup.py

@@ -24,7 +24,7 @@
 #
 #
 # This hook is called when changesets are pushed to the master repository (ie
-# `hg push`). See https://zulip.com/integrations for installation instructions.
+# `hg push`). See https://zulipchat.com/integrations for installation instructions.
 from __future__ import absolute_import
 
 import zulip

api/integrations/twitter/twitter-bot

@@ -45,11 +45,12 @@ parser = optparse.OptionParser(r"""
 
     Slurp tweets on your timeline into a specific zulip stream.
 
-    Run this on your personal machine.  Your API key and twitter id are revealed to local
-    users through the command line or config file.
+    Run this on your personal machine.  Your API key and twitter id
+    are revealed to local users through the command line or config
+    file.
 
-    This bot uses OAuth to authenticate with twitter. Please create a ~/.zulip_twitterrc with
-    the following contents:
+    This bot uses OAuth to authenticate with twitter. Please create a
+    ~/.zulip_twitterrc with the following contents:
 
     [twitter]
     consumer_key =
@@ -57,14 +58,16 @@ parser = optparse.OptionParser(r"""
     access_token_key =
     access_token_secret =
 
-    In order to obtain a consumer key & secret, you must register a new application under your twitter account:
+    In order to obtain a consumer key & secret, you must register a
+    new application under your twitter account:
 
     1. Go to http://dev.twitter.com
     2. Log in
     3. In the menu under your username, click My Applications
     4. Create a new application
 
-    Make sure to go the application you created and click "create my access token" as well. Fill in the values displayed.
+    Make sure to go the application you created and click "create my
+    access token" as well.  Fill in the values displayed.
 
     Depends on: twitter-python
 """)

api/setup.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 
 from __future__ import print_function
-from typing import Any, Generator, List, Tuple
+if False: from typing import Any, Generator, List, Tuple
 
 import os
 import sys
@@ -10,6 +10,7 @@ import sys
 import itertools
 
 def version():
+    # type: () -> str
     version_py = os.path.join(os.path.dirname(__file__), "zulip", "__init__.py")
     with open(version_py) as in_handle:
         version_line = next(itertools.dropwhile(lambda x: not x.startswith("__version__"),
@@ -46,7 +47,7 @@ package_info = dict(
                                            "examples/print-messages", "examples/recent-messages"])] + \
         list(recur_expand('share/zulip', 'integrations/')),
     scripts=["bin/zulip-send"],
-)
+) # type: Dict[str, Any]
 
 setuptools_info = dict(
     install_requires=['requests>=0.12.1',

api/zulip/__init__.py

@@ -134,6 +134,17 @@ def generate_option_group(parser, prefix=''):
                           CA certificates. This will be used to
                           verify the server's identity. All
                           certificates should be PEM encoded.''')
+    group.add_option('--client-cert',
+                     action='store',
+                     dest='client_cert',
+                     help='''Specify a file containing a client
+                          certificate (not needed for most deployments).''')
+    group.add_option('--client-cert-key',
+                     action='store',
+                     dest='client_cert_key',
+                     help='''Specify a file containing the client
+                          certificate's key (if it is in a separate
+                          file).''')
     return group
 
 def init_from_options(options, client=None):
@@ -144,20 +155,24 @@ def init_from_options(options, client=None):
     return Client(email=options.zulip_email, api_key=options.zulip_api_key,
                   config_file=options.zulip_config_file, verbose=options.verbose,
                   site=options.zulip_site, client=client,
-                  cert_bundle=options.cert_bundle, insecure=options.insecure)
+                  cert_bundle=options.cert_bundle, insecure=options.insecure,
+                  client_cert=options.client_cert,
+                  client_cert_key=options.client_cert_key)
 
 def get_default_config_filename():
     config_file = os.path.join(os.environ["HOME"], ".zuliprc")
     if (not os.path.exists(config_file) and
         os.path.exists(os.path.join(os.environ["HOME"], ".humbugrc"))):
-        raise RuntimeError("The Zulip API configuration file is now ~/.zuliprc; please run:\n\n  mv ~/.humbugrc ~/.zuliprc\n")
+        raise RuntimeError("The Zulip API configuration file is now ~/.zuliprc; please run:\n\n"
+                           "  mv ~/.humbugrc ~/.zuliprc\n")
     return config_file
 
 class Client(object):
     def __init__(self, email=None, api_key=None, config_file=None,
                  verbose=False, retry_on_errors=True,
                  site=None, client=None,
-                 cert_bundle=None, insecure=None):
+                 cert_bundle=None, insecure=None,
+                 client_cert=None, client_cert_key=None):
         if client is None:
             client = _default_client()
 
@@ -173,6 +188,10 @@ class Client(object):
                 email = config.get("api", "email")
             if site is None and config.has_option("api", "site"):
                 site = config.get("api", "site")
+            if client_cert is None and config.has_option("api", "client_cert"):
+                client_cert = config.get("api", "client_cert")
+            if client_cert_key is None and config.has_option("api", "client_cert_key"):
+                client_cert_key = config.get("api", "client_cert_key")
             if cert_bundle is None and config.has_option("api", "cert_bundle"):
                 cert_bundle = config.get("api", "cert_bundle")
             if insecure is None and config.has_option("api", "insecure"):
@@ -219,6 +238,21 @@ class Client(object):
             # Default behavior: verify against system CA certificates
             self.tls_verification=True
 
+        if client_cert is None:
+            if client_cert_key is not None:
+                raise RuntimeError("client cert key '%s' specified, but no client cert public part provided"
+                                   %(client_cert_key,))
+        else: # we have a client cert
+            if not os.path.isfile(client_cert):
+                raise RuntimeError("client cert '%s' does not exist"
+                                   %(client_cert,))
+            if client_cert_key is not None:
+                if not os.path.isfile(client_cert_key):
+                    raise RuntimeError("client cert key '%s' does not exist"
+                                       %(client_cert_key,))
+        self.client_cert = client_cert
+        self.client_cert_key = client_cert_key
+
     def get_user_agent(self):
         vendor = ''
         vendor_version = ''
@@ -247,10 +281,10 @@ class Client(object):
         request = {}
 
         for (key, val) in six.iteritems(orig_request):
-            if not (isinstance(val, str) or isinstance(val, six.text_type)):
-                request[key] = simplejson.dumps(val)
-            else:
+            if isinstance(val, str) or isinstance(val, six.text_type):
                 request[key] = val
+            else:
+                request[key] = simplejson.dumps(val)
 
         query_state = {
             'had_error_retry': False,
@@ -288,12 +322,21 @@ class Client(object):
                 else:
                     kwarg = "data"
                 kwargs = {kwarg: query_state["request"]}
+
+                # Build a client cert object for requests
+                if self.client_cert_key is not None:
+                    client_cert = (self.client_cert, self.client_cert_key)
+                else:
+                    client_cert = self.client_cert
+
                 res = requests.request(
                         method,
                         urllib.parse.urljoin(self.base_url, url),
                         auth=requests.auth.HTTPBasicAuth(self.email,
                                                          self.api_key),
-                        verify=self.tls_verification, timeout=90,
+                        verify=self.tls_verification,
+                        cert=client_cert,
+                        timeout=90,
                         headers={"User-agent": self.get_user_agent()},
                         **kwargs)
 

bots/jabber_mirror.py

@@ -27,10 +27,13 @@ import subprocess
 import os
 import traceback
 import signal
+from types import FrameType
+from typing import Any
 from zulip import RandomExponentialBackoff
 
 def die(signal, frame):
-    # We actually want to exit, so run os._exit (so as not to be caught and restarted)
+    # type: (int, FrameType) -> None
+    """We actually want to exit, so run os._exit (so as not to be caught and restarted)"""
     os._exit(1)
 
 signal.signal(signal.SIGINT, die)

bots/jabber_mirror_backend.py

@@ -381,7 +381,8 @@ option does not affect login credentials.'''.replace("\n", " "))
             config.readfp(f, config_file)
     except IOError:
         pass
-    for option in ("jid", "jabber_password", "conference_domain", "mode", "zulip_email_suffix", "jabber_server_address", "jabber_server_port"):
+    for option in ("jid", "jabber_password", "conference_domain", "mode", "zulip_email_suffix",
+                   "jabber_server_address", "jabber_server_port"):
         if (getattr(options, option) is None
             and config.has_option("jabber_mirror", option)):
             setattr(options, option, config.get("jabber_mirror", option))

bots/log2zulip

@@ -1,16 +1,25 @@
 #!/usr/bin/env python
 from __future__ import print_function
-import subprocess
-import os
-import sys
-import shutil
+
 import errno
-import json
-import ujson
+import os
 import platform
 import re
+import sys
+import shutil
+import subprocess
 import traceback
 
+try:
+    # Use the Zulip virtualenv if available
+    sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
+    import scripts.lib.setup_path_on_import
+except ImportError:
+    pass
+
+import json
+import ujson
+
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../api"))
 import zulip
 

bots/zephyr_mirror.py

@@ -31,15 +31,21 @@ import signal
 
 from .zephyr_mirror_backend import parse_args
 
+(options, args) = parse_args()
+
+sys.path[:0] = [os.path.join(options.root_path, 'api')]
+
+from types import FrameType
+from typing import Any
+
 def die(signal, frame):
+    # type: (int, FrameType) -> None
+
     # We actually want to exit, so run os._exit (so as not to be caught and restarted)
     os._exit(1)
 
 signal.signal(signal.SIGINT, die)
 
-(options, args) = parse_args()
-
-sys.path[:0] = [os.path.join(options.root_path, 'api')]
 from zulip import RandomExponentialBackoff
 
 args = [os.path.join(options.root_path, "user_root", "zephyr_mirror_backend.py")]
@@ -57,6 +63,7 @@ if options.forward_class_messages and not options.noshard:
     print("Starting parallel zephyr class mirroring bot")
     jobs = list("0123456789abcdef")
     def run_job(shard):
+        # type: (str) -> int
         subprocess.call(args + ["--shard=%s" % (shard,)])
         return 0
     for (status, job) in run_parallel(run_job, jobs, threads=16):

bots/zephyr_mirror_backend.py

@@ -92,7 +92,7 @@ def different_paragraph(line, next_line):
             len(line) < len(words[0]))
 
 # Linewrapping algorithm based on:
-# http://gcbenison.wordpress.com/2011/07/03/a-program-to-intelligently-remove-carriage-returns-so-you-can-paste-text-without-having-it-look-awful/
+# http://gcbenison.wordpress.com/2011/07/03/a-program-to-intelligently-remove-carriage-returns-so-you-can-paste-text-without-having-it-look-awful/ #ignorelongline
 def unwrap_lines(body):
     lines = body.split("\n")
     result = ""
@@ -598,7 +598,7 @@ def zcrypt_encrypt_content(zephyr_class, instance, content):
 def forward_to_zephyr(message):
     support_heading = "Hi there! This is an automated message from Zulip."
     support_closing = """If you have any questions, please be in touch through the \
-Feedback button or at support@zulip.com."""
+Feedback button or at support@zulipchat.com."""
 
     wrapper = textwrap.TextWrapper(break_long_words=False, break_on_hyphens=False)
     wrapped_content = "\n".join("\n".join(wrapper.wrap(line))

confirmation/migrations/0002_realmcreationkey.py

@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+import django.utils.timezone
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('confirmation', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='RealmCreationKey',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('creation_key', models.CharField(max_length=40, verbose_name='activation key')),
+                ('date_created', models.DateTimeField(default=django.utils.timezone.now, verbose_name='created')),
+            ],
+        ),
+    ]

confirmation/models.py

@@ -30,15 +30,33 @@ except ImportError:
 
 B16_RE = re.compile('^[a-f0-9]{40}$')
 
+def check_key_is_valid(creation_key):
+    if not RealmCreationKey.objects.filter(creation_key=creation_key).exists():
+        return False
+    days_sofar = (now() - RealmCreationKey.objects.get(creation_key=creation_key).date_created).days
+    # Realm creation link expires after settings.REALM_CREATION_LINK_VALIDITY_DAYS
+    if days_sofar <= settings.REALM_CREATION_LINK_VALIDITY_DAYS:
+        return True
+    return False
+
 def generate_key():
     return generate_random_token(40)
 
-def generate_activation_url(key):
+def generate_activation_url(key, host=None):
+    if host is None:
+        host = settings.EXTERNAL_HOST
     return u'%s%s%s' % (settings.EXTERNAL_URI_SCHEME,
-                        settings.EXTERNAL_HOST,
+                        host,
                         reverse('confirmation.views.confirm',
                                 kwargs={'confirmation_key': key}))
 
+def generate_realm_creation_url():
+    key = generate_key()
+    RealmCreationKey.objects.create(creation_key=key, date_created=now())
+    return u'%s%s%s' % (settings.EXTERNAL_URI_SCHEME,
+                        settings.EXTERNAL_HOST,
+                        reverse('zerver.views.create_realm',
+                        kwargs={'creation_key': key}))
 
 class ConfirmationManager(models.Manager):
 
@@ -55,16 +73,17 @@ class ConfirmationManager(models.Manager):
             return obj
         return False
 
-    def get_link_for_object(self, obj):
+    def get_link_for_object(self, obj, host=None):
         key = generate_key()
         self.create(content_object=obj, date_sent=now(), confirmation_key=key)
-        return generate_activation_url(key)
+        return generate_activation_url(key, host=host)
 
     def send_confirmation(self, obj, email_address, additional_context=None,
-            subject_template_path=None, body_template_path=None):
+                          subject_template_path=None, body_template_path=None,
+                          host=None):
         confirmation_key = generate_key()
         current_site = Site.objects.get_current()
-        activate_url = generate_activation_url(confirmation_key)
+        activate_url = generate_activation_url(confirmation_key, host=host)
         context = Context({
             'activate_url': activate_url,
             'current_site': current_site,
@@ -74,8 +93,12 @@ class ConfirmationManager(models.Manager):
         })
         if additional_context is not None:
             context.update(additional_context)
+        if obj.realm is not None and obj.realm.is_zephyr_mirror_realm:
+            template_name = "mituser"
+        else:
+            template_name = obj._meta.model_name
         templates = [
-            'confirmation/%s_confirmation_email_subject.txt' % obj._meta.model_name,
+            'confirmation/%s_confirmation_email_subject.txt' % (template_name,),
             'confirmation/confirmation_email_subject.txt',
         ]
         if subject_template_path:
@@ -84,7 +107,7 @@ class ConfirmationManager(models.Manager):
             template = loader.select_template(templates)
         subject = template.render(context).strip().replace(u'\n', u' ') # no newlines, please
         templates = [
-            'confirmation/%s_confirmation_email_body.txt' % obj._meta.model_name,
+            'confirmation/%s_confirmation_email_body.txt' % (template_name,),
             'confirmation/confirmation_email_body.txt',
         ]
         if body_template_path:
@@ -111,3 +134,7 @@ class Confirmation(models.Model):
 
     def __unicode__(self):
         return _('confirmation email for %s') % (self.content_object,)
+
+class RealmCreationKey(models.Model):
+    creation_key = models.CharField(_('activation key'), max_length=40)
+    date_created = models.DateTimeField(_('created'), default=now)

confirmation/views.py

@@ -34,7 +34,7 @@ def confirm(request, confirmation_key):
         'key': confirmation_key,
         'full_name': request.GET.get("full_name", None),
         'support_email': settings.ZULIP_ADMINISTRATOR,
-        'voyager': settings.VOYAGER
+        'verbose_support_offers': settings.VERBOSE_SUPPORT_OFFERS,
     }
     templates = [
         'confirmation/confirm.html',

contrib_bots/lib/followup.py

@@ -0,0 +1,53 @@
+# See readme.md for instructions on running this code.
+
+class FollowupHandler(object):
+    '''
+    This plugin facilitates creating follow-up tasks when
+    you are using Zulip to conduct a virtual meeting.  It
+    looks for messages starting with '@followup'.
+
+    In this example, we write follow up items to a special
+    Zulip stream called "followup," but this code could
+    be adapted to write follow up items to some kind of
+    external issue tracker as well.
+    '''
+
+    def usage(self):
+        return '''
+            This plugin will allow users to flag messages
+            as being follow-up items.  Users should preface
+            messages with "@followup".
+
+            Before running this, make sure to create a stream
+            called "followup" that your API user can send to.
+            '''
+
+    def triage_message(self, message):
+        # return True iff we want to (possibly) response to this message
+
+        original_content = message['content']
+
+        # This next line of code is defensive, as we
+        # never want to get into an infinite loop of posting follow
+        # ups for own follow ups!
+        if message['display_recipient'] == 'followup':
+            return False
+        is_follow_up = (original_content.startswith('@followup') or
+                        original_content.startswith('@follow-up'))
+
+        return is_follow_up
+
+    def handle_message(self, message, client):
+        original_content = message['content']
+        original_sender = message['sender_email']
+        new_content = original_content.replace('@followup',
+                                               'from %s:' % (original_sender,))
+
+        client.send_message(dict(
+            type='stream',
+            to='followup',
+            subject=message['sender_email'],
+            content=new_content,
+        ))
+
+handler_class = FollowupHandler

contrib_bots/lib/help.py

@@ -0,0 +1,39 @@
+# See readme.md for instructions on running this code.
+
+class HelpHandler(object):
+    def usage(self):
+        return '''
+            This plugin will give info about Zulip to
+            any user that types a message saying "help".
+
+            This is example code; ideally, you would flesh
+            this out for more useful help pertaining to
+            your Zulip instance.
+            '''
+
+    def triage_message(self, message):
+        # return True if we think the message may be of interest
+        original_content = message['content']
+
+        if message['type'] != 'stream':
+            return True
+
+        if original_content.lower().strip() != 'help':
+            return False
+
+        return True
+
+    def handle_message(self, message, client):
+        help_content = '''
+            Info on Zulip can be found here:
+            https://github.com/zulip/zulip
+            '''.strip()
+
+        client.send_message(dict(
+            type='stream',
+            to=message['display_recipient'],
+            subject=message['subject'],
+            content=help_content,
+        ))
+
+handler_class = HelpHandler

contrib_bots/lib/readme.md

@@ -0,0 +1,109 @@
+# Overview
+
+This is the documentation for an experimental new system for writing
+bots that react to messages.
+
+This directory contains library code for running Zulip
+bots that react to messages sent by users.
+
+This document explains how to run the code, and it also
+talks about the architecture for creating bots.
+
+## Design goals
+
+The goal is to have a common framework for hosting a bot that reacts
+to messages in any of the following settings:
+
+* Run as a long-running process using `call_on_each_event`
+  (implemented today).
+
+* Run via a simple web service that can be deployed to PAAS providers
+  and handles outgoing webhook requests from Zulip.
+
+* Embedded into the Zulip server (so that no hosting is required),
+  which would be done for high quality, reusable bots; we would have a
+  nice "bot store" sort of UI for browsing and activating them.
+
+## Running bots
+
+Here is an example of running the "follow-up" bot from
+inside a Zulip repo:
+
+    cd ~/zulip/contrib_bots
+    python run.py lib/followup.py --config-file ~/.zuliprc-prod
+
+Once the bot code starts running, you will see a
+message explaining how to use the bot, as well as
+some log messages.  You can use the `--quiet` option
+to suppress these messages.
+
+The bot code will run continuously until you kill them with
+control-C (or otherwise).
+
+### Configuration
+
+For this document we assume you have some prior experience
+with using the Zulip API, but here is a quick review of
+what a `.zuliprc` files looks like.  You can connect to the
+API as your own human user, or you can go into the Zulip settings
+page to create a user-owned bot.
+
+    [api]
+    email=someuser@example.com
+    key=<your api key>
+    site=https://zulip.somewhere.com
+
+## Architecture
+
+In order to make bot development easy, we separate
+out boilerplate code (loading up the Client API, etc.)
+from bot-specific code (do what makes the bot unique).
+
+All of the boilerplate code lives in `../run.py`.  The
+runner code does things like find where it can import
+the Zulip API, instantiate a client with correct
+credentials, set up the logging level, find the
+library code for the specific bot, etc.
+
+Then, for bot-specific logic, you will find `.py` files
+in the `lib` directory (i.e. the same directory as the
+document you are reading now).
+
+Each bot library simply needs to do the following:
+
+- Define a class that supports the methods `usage`,
+`triage_message`, and `handle_message`.
+- Set `handler_class` to be the name of that class.
+
+(We make this a two-step process, so that you can give
+a descriptive name to your handler class.)
+
+## Portability
+
+Creating a handler class for each bot allows your bot
+code to be more portable.  For example, if you want to
+use your bot code in some other kind of bot platform, then
+if all of your bots conform to the `handler_class` protocol,
+you can write simple adapter code to use them elsewhere.
+
+Another future direction to consider is that Zulip will
+eventually support running certain types of bots on
+the server side, to essentially implement post-send
+hooks and things of those nature.
+
+Conforming to the `handler_class` protocol will make
+it easier for Zulip admins to integrate custom bots.
+
+In particular, `run.py` already passes in instances
+of a restricted variant of the Client class to your
+library code, which helps you ensure that your bot
+does only things that would be acceptable for running
+in a server-side environment.
+
+## Other approaches
+
+If you are not interested in running your bots on the
+server, then you can still use the full Zulip API.  The
+hope, though, is that this architecture will make
+writing simple bots a quick/easy process.
+

contrib_bots/run.py

@@ -0,0 +1,100 @@
+from __future__ import print_function
+
+import importlib
+import logging
+import optparse
+import os
+import sys
+
+our_dir = os.path.dirname(os.path.abspath(__file__))
+
+# For dev setups, we can find the API in the repo itself.
+if os.path.exists(os.path.join(our_dir, '../api/zulip')):
+    sys.path.append('../api')
+
+from zulip import Client
+
+class RestrictedClient(object):
+    def __init__(self, client):
+        # Only expose a subset of our Client's functionality
+        self.send_message = client.send_message
+
+def get_lib_module(lib_fn):
+    lib_fn = os.path.abspath(lib_fn)
+    if os.path.dirname(lib_fn) != os.path.join(our_dir, 'lib'):
+        print('Sorry, we will only import code from contrib_bots/lib.')
+        sys.exit(1)
+
+    if not lib_fn.endswith('.py'):
+        print('Please use a .py extension for library files.')
+        sys.exit(1)
+
+    sys.path.append('lib')
+    base_lib_fn = os.path.basename(os.path.splitext(lib_fn)[0])
+    module_name = 'lib.' + base_lib_fn
+    module = importlib.import_module(module_name)
+    return module
+
+def run_message_handler_for_bot(lib_module, quiet, config_file):
+    # Make sure you set up your ~/.zuliprc
+    client = Client(config_file=config_file)
+    restricted_client = RestrictedClient(client)
+
+    message_handler = lib_module.handler_class()
+
+    if not quiet:
+        print(message_handler.usage())
+
+    def handle_message(message):
+        logging.info('waiting for next message')
+        if message_handler.triage_message(message=message):
+            message_handler.handle_message(
+                message=message,
+                client=restricted_client)
+
+    logging.info('starting message handling...')
+    client.call_on_each_message(handle_message)
+
+def run():
+    usage = '''
+        python run.py <lib file>
+
+        Example: python run.py lib/followup.py
+
+        (This program loads bot-related code from the
+        library code and then runs a message loop,
+        feeding messages to the library code to handle.)
+
+        Please make sure you have a current ~/.zuliprc
+        file with the credentials you want to use for
+        this bot.
+
+        See lib/readme.md for more context.
+        '''
+
+    parser = optparse.OptionParser(usage=usage)
+    parser.add_option('--quiet', '-q',
+        action='store_true',
+        help='Turn off logging output.')
+    parser.add_option('--config-file',
+        action='store',
+        help='(alternate config file to ~/.zuliprc)')
+    (options, args) = parser.parse_args()
+
+    if len(args) == 0:
+        print('You must specify a library!')
+        sys.exit(1)
+
+    lib_module = get_lib_module(lib_fn=args[0])
+
+    if not options.quiet:
+        logging.basicConfig(stream=sys.stdout, level=logging.INFO)
+
+    run_message_handler_for_bot(
+        lib_module=lib_module,
+        config_file=options.config_file,
+        quiet=options.quiet
+    )
+
+if __name__ == '__main__':
+    run()

corporate/terms.md

@@ -0,0 +1,244 @@
+# Zulip Terms of Service
+
+### Welcome to Zulip!
+
+Thanks for using our products and services ("Services"). The Services are
+provided by Zulip, Inc. ("Zulip"), located at 552 Massachusetts Ave Suite 203,
+Cambridge, MA 02139, United States.
+
+By using our Services, you are agreeing to these terms. Please read them
+carefully.
+
+The Services are not intended for use by you if you are under 13 years of
+age. By agreeing to these terms, you are representing to us that you are over
+13.
+
+### Using our Services
+
+You must follow any policies made available to you within the Services.
+
+Don't misuse our Services. For example, don't interfere with our Services or
+try to access them using a method other than the interface and the instructions
+that we provide. You may use our Services only as permitted by law, including
+applicable export and re-export control laws and regulations. We may suspend or
+stop providing our Services to you if you do not comply with our terms or
+policies or if we are investigating suspected misconduct.
+
+Using our Services does not give you ownership of any intellectual property
+rights in our Services or the content you access. You may not use content from
+our Services unless you obtain permission from its owner or are otherwise
+permitted by law. These terms do not grant you the right to use any branding or
+logos used in our Services. Don't remove, obscure, or alter any legal notices
+displayed in or along with our Services.
+
+Our Services display some content that is not Zulip's. This content is the
+sole responsibility of the entity that makes it available. We may review
+content to determine whether it is illegal or violates our policies, and we may
+remove or refuse to display content that we reasonably believe violates our
+policies or the law. But that does not necessarily mean that we review content,
+so please don't assume that we do.
+
+In connection with your use of the Services, we may send you service
+announcements, administrative messages, and other information. You may opt out
+of some of those communications.
+
+### Your Zulip Account
+
+You may need a Zulip Account in order to use some of our Services. You may
+create your own Zulip Account, or your Zulip Account may be assigned to you
+by an administrator, such as your employer or educational institution. If you
+are using a Zulip Account assigned to you by an administrator, different or
+additional terms may apply and your administrator may be able to access or
+disable your account.
+
+If you learn of any unauthorized use of your password or account, contact
+[support@zulip.com](mailto:support@zulip.com).
+
+### Privacy and Copyright Protection
+
+Zulip's [privacy policy](/privacy) explains how we treat your
+personal data and protect your privacy when you use our Services. By using our
+Services, you agree that Zulip can use such data in accordance with our
+privacy policy.
+
+We respond to notices of alleged copyright infringement and terminate
+accounts of repeat infringers according to the process set out in the U.S.
+Digital Millennium Copyright Act.
+
+Our designated agent for notice of alleged copyright infringement on the
+Services is:
+
+> Copyright Agent  
+> Zulip, Inc.  
+> 552 Massachusetts Ave Suite 203  
+> Cambridge, MA 02139  
+> [copyright@zulip.com](mailto:copyright@zulip.com)
+
+### Your Content in our Services
+
+Some of our Services allow you to submit content. You retain ownership of
+any intellectual property rights that you hold in that content. In short, what
+belongs to you stays yours.
+
+When you upload or otherwise submit content to our Services, you give Zulip
+(and those we work with) a worldwide license to use, host, store, reproduce,
+modify, create derivative works (such as those resulting from translations,
+adaptations or other changes we make so that your content works better with our
+Services), communicate, publish, perform, display and distribute such content.
+The rights you grant in this license are for the limited purpose of operating
+and improving our Services, and to develop new ones. This license continues
+even if you stop using our Services (for example, so that we can deliver a
+message that you sent to another Zulip Account before you stopped using our
+Services). Some Services may offer you ways to access and remove content that
+has been provided to that Service. Also, in some of our Services, there may be
+terms or settings that narrow the scope of our use of the content submitted in
+those Services. Make sure you have the necessary rights to grant us this
+license for any content that you submit to our Services. If you use the
+Services to share content with others, anyone you've shared content with
+(including the general public, in certain circumstances) may have access to the
+content.
+
+In order to provide the Services, our servers save a record of the messages
+received by each Zulip Account (the "Received Messsage Information" for the
+account). If you are using our Services on behalf of a business and a
+representative of that business sends [data@zulip.com](mailto:data@zulip.com)
+a request to delete all of your business' accounts with us, then within a
+commercially reasonable period of time, we will close all of your business'
+accounts with us and delete the Received Message Information for each such
+account by removing pointers to the information on our active servers and
+overwriting it over time. Notwithstanding the foregoing, deleting the Received
+Message Information for your business' accounts will not require deleting any
+information about messages that were sent or received by any Zulip Accounts that
+are not one of your business' accounts with us (such as system-wide announcement
+messages or any messages corresponding with the Zulip support team).
+
+You can find more information about how Zulip uses and stores content in
+the privacy policy. If you submit feedback or suggestions about our Services,
+we may use your feedback or suggestions without obligation to you.
+
+
+### About Software in our Services
+
+When a Service requires or includes downloadable software, this software may
+update automatically on your device once a new version or feature is available.
+Some Services may let you adjust your automatic update settings.
+
+Zulip gives you a personal, worldwide, royalty-free, non-assignable and
+non-exclusive license to use the software provided to you by Zulip as part of
+the Services. This license is for the sole purpose of enabling you to use and
+enjoy the benefit of the Services as provided by Zulip, in the manner
+permitted by these terms. You may not copy, modify, distribute, sell, or lease
+any part of our Services or included software, nor may you reverse engineer or
+attempt to extract the source code of that software, unless laws prohibit those
+restrictions or you have our written permission.
+
+Some software used in our Services may be offered under an open source
+license that we will make available to you.  There may be provisions in the
+open source license that expressly override some of these terms.
+
+
+### Modifying and Terminating our Services
+
+We are constantly changing and improving our Services. We may add or remove
+functionalities or features, and we may suspend or stop a Service
+altogether.
+
+You can stop using our Services at any time, although we'll be sorry to see
+you go. Zulip may also stop providing Services to you, or add or create new
+limits to our Services at any time.
+
+We believe that you own your data and preserving your access to such data is
+important. If we discontinue a Service, we will, if it is practical in our sole
+discretion, give you reasonable advance notice and a chance to get information
+out of that Service.
+
+
+### Our Warranties and Disclaimers
+
+We hope that you will enjoy using our Services, but there are certain things
+that we don't promise about our Services.
+
+OTHER THAN AS EXPRESSLY SET OUT IN THESE TERMS, NEITHER ZULIP NOR ITS
+SUPPLIERS OR DISTRIBUTORS MAKE ANY SPECIFIC PROMISES ABOUT THE SERVICES.  FOR
+EXAMPLE, WE DON'T MAKE ANY COMMITMENTS ABOUT THE CONTENT WITHIN THE SERVICES,
+THE SPECIFIC FUNCTION OF THE SERVICES, OR THEIR RELIABILITY, AVAILABILITY, OR
+ABILITY TO MEET YOUR NEEDS. WE PROVIDE THE SERVICES "AS IS".
+
+SOME JURISDICTIONS PROVIDE FOR CERTAIN WARRANTIES, LIKE THE IMPLIED WARRANTY
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. TO
+THE EXTENT PERMITTED BY LAW, WE EXCLUDE ALL WARRANTIES.
+
+
+### Liability for our Services
+
+WHEN PERMITTED BY LAW, ZULIP, AND ZULIP'S SUPPLIERS AND DISTRIBUTORS, WILL
+NOT BE RESPONSIBLE FOR LOST PROFITS, REVENUES, OR DATA, FINANCIAL LOSSES OR
+INDIRECT, SPECIAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES.
+
+TO THE EXTENT PERMITTED BY LAW, THE TOTAL LIABILITY OF ZULIP, AND ITS
+SUPPLIERS AND DISTRIBUTORS, FOR ANY CLAIM UNDER THESE TERMS, INCLUDING FOR ANY
+IMPLIED WARRANTIES, IS LIMITED TO THE GREATER OF FIVE DOLLARS ($5) OR THE
+AMOUNT PAID BY YOU TO ZULIP FOR THE PAST THREE MONTHS OF THE SERVICES IN
+QUESTION.
+
+IN ALL CASES, ZULIP, AND ITS SUPPLIERS AND DISTRIBUTORS, WILL NOT BE LIABLE
+FOR ANY LOSS OR DAMAGE THAT IS NOT REASONABLY FORESEEABLE.
+
+
+### Business uses of our Services
+
+If you are using our Services on behalf of a business, that business accepts
+these terms. It will hold harmless and indemnify Zulip and its affiliates,
+officers, agents, and employees from any claim, suit or action arising from or
+related to the use of the Services or violation of these terms, including any
+liability or expense arising from claims, losses, damages, suits, judgments,
+litigation costs and attorneys' fees.
+
+You agree that we, in our sole discretion, may use your trade names,
+trademarks, service marks, logos, domain names and other distinctive brand
+features in presentations, marketing materials, customer lists, financial
+reports and Web site listings (including links to your website) for the purpose
+of advertising or publicizing your use of our products or services.
+
+### Export Compliance
+
+If you are located outside of the United States or are not a U.S. person,
+you certify that you do not reside in Cuba, Iran, North Korea, Sudan, or Syria,
+and you certify the following: "We certify that this beta test software will
+only be used for beta testing purposes, and will not be rented, leased, sold,
+sublicensed, assigned, or otherwise transferred. Further, we certify that we
+will not transfer or export any product, process, or service that is the direct
+product of the beta test software."
+
+
+### About these Terms
+
+If it turns out that a particular term is not enforceable, this will not
+affect any other terms.
+
+If you do not comply with these terms, and we don't take action right away,
+this doesn't mean that we are giving up any rights that we may have (such as
+taking action in the future).
+
+These terms control the relationship between Zulip and you. They do not
+create any third party beneficiary rights.
+
+The laws of Massachusetts, U.S.A., excluding Massachusetts's conflict of
+laws rules, will apply to any disputes arising out of or relating to these
+terms or the Services. All claims arising out of or relating to these terms or
+the Services will be litigated exclusively in the applicable federal or state
+courts of Massachusetts, and you and Zulip consent to personal jurisdiction in
+those courts.
+
+Zulip reserves the right to amend or modify these terms at any time and in
+any manner by providing reasonable notice to you. You agree that reasonable
+notice may be provided by posting on Zulip's web site, email, or other written
+notice.   By continuing to access or use the Services after revisions become
+effective, you agree to be bound by the revised terms. If you do not agree to
+the new terms, please stop using the Services.
+
+These terms constitute the whole legal agreement between you and Zulip, and
+completely replace any prior agreements between you and Zulip in relation to
+the Services.
+
+Last modified: October 4, 2013

corporate/urls.py

@@ -8,7 +8,6 @@ i18n_urlpatterns = [
     url(r'^zephyr-mirror/$', TemplateView.as_view(template_name='corporate/zephyr-mirror.html')),
 
     # Terms of service and privacy policy
-    url(r'^terms/$',   TemplateView.as_view(template_name='corporate/terms.html')),
     url(r'^terms-enterprise/$',  TemplateView.as_view(template_name='corporate/terms-enterprise.html')),
     url(r'^privacy/$', TemplateView.as_view(template_name='corporate/privacy.html')),
 ]

docs/README.md

@@ -1,3 +1,5 @@
+# Documentation
+
 These docs are written in [Commonmark
 Markdown](http://commonmark.org/) with a small bit of rST.  We've
 chosen Markdown because it is [easy to
@@ -14,6 +16,10 @@ cd docs/
 make html
 ```
 
+and then opening `file:///path/to/zulip/docs/_build/html/index.html` in
+your browser (you can also use e.g. `firefox
+docs/_build/html/index.html` from the root of your Zulip checkout).
+
 You can also usually test your changes by pushing a branch to GitHub
 and looking at the content on the GitHub web UI, since GitHub renders
 Markdown.

docs/architecture-overview.md

@@ -10,8 +10,7 @@ is a web application written in Python 2.7 (soon to also support
 Python 3) and using the Django framework. That codebase includes
 server-side code and the web client, as well as Python API bindings
 and most of our integrations with other services and applications (see
-[the directory structure
-guide](https://zulip.readthedocs.io/en/latest/directory-structure.html)).
+[the directory structure guide](directory-structure.html)).
 
 We maintain several separate repositories for integrations and other
 glue code: a [Hubot adapter](https://github.com/zulip/hubot-zulip);
@@ -22,14 +21,17 @@ integrations with
 [Redmine](https://github.com/zulip/zulip-redmine-plugin), and
 [Trello](https://github.com/zulip/trello-to-zulip); [node.js API
 bindings](https://github.com/zulip/zulip-node); and our [full-text
-search PostgreSQL extension](https://github.com/zulip/tsearch_extras) .
+search PostgreSQL extension](https://github.com/zulip/tsearch_extras).
 
 Our mobile clients are separate code repositories:
 [Android](https://github.com/zulip/zulip-android), [iOS
-(stable)](https://github.com/zulip/zulip-ios) , and [our experimental
+(stable)](https://github.com/zulip/zulip-ios), and [our experimental
 React Native iOS app](https://github.com/zulip/zulip-mobile). Our
-[desktop application](https://github.com/zulip/zulip-desktop) is also a
-separate repository.
+[legacy desktop application (implemented in
+QT/WebKit)](https://github.com/zulip/zulip-desktop) and our new, alpha
+[cross-platform desktop app (implemented in
+Electron)](https://github.com/zulip/zulip-electron) are also separate
+repositories.
 
 We use [Transifex](https://www.transifex.com/zulip/zulip/) to do
 translations.
@@ -45,31 +47,33 @@ similar groups ranging in size from a small team to more than a thousand
 users. It features real-time notifications, message persistence and
 search, public group conversations (*streams*), invite-only streams,
 private one-on-one and group conversations, inline image previews, team
-presence/a buddy list, a rich API, Markdown message support, and several
+presence/buddy list, a rich API, Markdown message support, and numerous
 integrations with other services. The maintainer team aims to support
 users who connect to Zulip using dedicated iOS, Android, Linux, Windows,
 and Mac OS X clients, as well as people using modern web browsers or
 dedicated Zulip API clients.
 
 A server can host multiple Zulip *realms* (organizations) at the same
-domain, each of which is a private chamber with its own users, streams,
-customizations, and so on. This means that one person might be a user of
-multiple Zulip realms. The administrators of a realm can choose whether
-to allow anyone to register an account and join, or only allow people
-who have been invited, or restrict registrations to members of
-particular groups (using email domain names or corporate single-sign-on
-login for verification). For more on scalability and security
-considerations, see [Zulip in
-production](https://github.com/zulip/zulip/blob/master/README.prod.md).
+domain, each of which is a private chamber with its own users,
+streams, customizations, and so on. This means that one person might
+be a user of multiple Zulip realms. The administrators of a realm can
+choose whether to allow anyone to register an account and join, or
+only allow people who have been invited, or restrict registrations to
+members of particular groups (using email domain names or corporate
+single-sign-on login for verification). For more on scalability and
+security considerations, see [the security section of the production
+maintenance
+instructions](prod-maintain-secure-upgrade.html#security-model).
 
 The default Zulip home screen is like a chronologically ordered inbox;
 it displays messages, starting at the oldest message that the user
-hasn't viewed yet. The home screen displays the most recent messages in
-all the streams a user has joined (except for the streams they've
-muted), as well as private messages from other users, in strict
-chronological order. A user can *narrow* to view only the messages in a
-single stream, and can further narrow to focus on a *topic* (thread)
-within that stream. Each narrow has its own URL.
+hasn't viewed yet (for more on that logic, see [the guide to the
+pointer and unread counts](pointer.html)). The home screen displays
+the most recent messages in all the streams a user has joined (except
+for the streams they've muted), as well as private messages from other
+users, in strict chronological order. A user can *narrow* to view only
+the messages in a single stream, and can further narrow to focus on a
+*topic* (thread) within that stream. Each narrow has its own URL.
 
 Zulip's philosophy is to provide sensible defaults but give the user
 fine-grained control over their incoming information flow; a user can
@@ -147,8 +151,8 @@ Tornado and Django are set up, as well as a number of background
 processes that process event queues. We use event queues for the kinds
 of tasks that are best run in the background because they are
 expensive (in terms of performance) and don't have to be synchronous
--- e.g., sending emails or updating analytics. Also see [the queuing
-guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+--- e.g., sending emails or updating analytics. Also see [the queuing
+guide](queuing.html).
 
 ### memcached
 
@@ -163,7 +167,7 @@ Redis is used for a few very short-term data stores, such as in the
 basis of `zerver/lib/rate_limiter.py`, a per-user rate limiting scheme
 [example](http://blog.domaintools.com/2013/04/rate-limiting-with-redis/)),
 and the [email-to-Zulip
-integration](https://zulip.com/integrations/#email).
+integration](https://zulipchat.com/integrations/#email).
 
 Redis is configured in `zulip/puppet/zulip/files/redis` and it's a
 pretty standard configuration except for the last line, which turns off
@@ -197,8 +201,7 @@ one queue or another. Most of the processes started by Supervisor are
 queue processors that continually pull things out of a RabbitMQ queue
 and handle them.
 
-Also see [the queuing
-guide](https://zulip.readthedocs.io/en/latest/queuing.html).
+Also see [the queuing guide](queuing.html).
 
 ### PostgreSQL
 
@@ -211,12 +214,13 @@ directory that would contain configuration files
 (`puppet/zulip/files/postgresql`) has only a utility script and a custom
 list of stopwords used by a Postgresql extension.
 
-In a development environment, configuration of that postgresql extension
-is handled by `tools/postgres-init-dev-db` (invoked by `provision.py`).
-That file also manages setting up the development postgresql user.
+In a development environment, configuration of that postgresql
+extension is handled by `tools/postgres-init-dev-db` (invoked by
+`tools/provision.py`).  That file also manages setting up the
+development postgresql user.
 
-`provision.py` also invokes `tools/do-destroy-rebuild-database` to
-create the actual database with its schema.
+`tools/provision.py` also invokes `tools/do-destroy-rebuild-database`
+to create the actual database with its schema.
 
 ### Nagios
 

docs/brief-install-vagrant-dev.md

@@ -0,0 +1,113 @@
+# Vagrant environment setup (in brief)
+
+Start by cloning this repository: `git clone https://github.com/zulip/zulip.git`
+
+This is the recommended approach for all platforms, and will install
+the Zulip development environment inside a VM or container and works
+on any platform that supports Vagrant.
+
+The best performing way to run the Zulip development environment is
+using an LXC container on a Linux host, but we support other platforms
+such as Mac via Virtualbox (but everything will be 2-3x slower).
+
+* If your host is Ubuntu 15.04 or newer, you can install and configure
+  the LXC Vagrant provider directly using apt:
+  ```
+  sudo apt-get install vagrant lxc lxc-templates cgroup-lite redir
+  vagrant plugin install vagrant-lxc
+  ```
+  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
+
+* If your host is Ubuntu 14.04, you will need to [download a newer
+  version of Vagrant][vagrant-dl], and then do the following:
+  ```
+  sudo apt-get install lxc lxc-templates cgroup-lite redir
+  sudo dpkg -i vagrant*.deb # in directory where you downloaded vagrant
+  vagrant plugin install vagrant-lxc
+  ```
+  You may want to [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo].
+
+* For other Linux hosts with a kernel above 3.12, [follow the Vagrant
+  LXC installation instructions][vagrant-lxc] to get Vagrant with LXC
+  for your platform.
+
+* If your host is OS X or older Linux, [download VirtualBox][vbox-dl],
+  [download Vagrant][vagrant-dl], and install them both.
+
+* If you're on OS X and have VMWare, it should be possible to patch
+  Vagrantfile to use the VMWare vagrant provider which should perform
+  much better than Virtualbox.  Patches to do this by default if
+  VMWare is available are welcome!
+
+* On Windows: You can use Vagrant and Virtualbox/VMWare on Windows
+  with Cygwin, similar to the Mac setup.  Be sure to create your git
+  clone using `git clone https://github.com/zulip/zulip.git -c
+  core.autocrlf=false` to avoid Windows line endings being added to
+  files (this causes weird errors).
+
+[vagrant-dl]: https://www.vagrantup.com/downloads.html
+[vagrant-lxc]: https://github.com/fgrehm/vagrant-lxc
+[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
+[avoiding-sudo]: https://github.com/fgrehm/vagrant-lxc#avoiding-sudo-passwords
+
+Once that's done, simply change to your zulip directory and run
+`vagrant up` in your terminal to install the development server.  This
+will take a long time on the first run because Vagrant needs to
+download the Ubuntu Trusty base image, but later you can run `vagrant
+destroy` and then `vagrant up` again to rebuild the environment and it
+will be much faster.
+
+Once that finishes, you can run the development server as follows:
+
+```
+vagrant ssh
+# Now inside the container
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+To get shell access to the virtual machine running the server to run
+lint, management commands, etc., use `vagrant ssh`.
+
+(A small note on tools/run-dev.py: the `--interface=''` option will
+make the development server listen on all network interfaces.  While
+this is correct for the Vagrant guest sitting behind a NAT, you
+probably don't want to use that option when using run-dev.py in other
+environments).
+
+At this point you should [read about using the development
+environment][using-dev].
+
+[using-dev]: using-dev-environment.html
+
+### Specifying a proxy
+
+If you need to use a proxy server to access the Internet, you will
+need to specify the proxy settings before running `Vagrant up`.
+First, install the Vagrant plugin `vagrant-proxyconf`:
+
+```
+vagrant plugin install vagrant-proxyconf.
+```
+
+Then create `~/.zulip-vagrant-config` and add the following lines to
+it (with the appropriate values in it for your proxy):
+
+```
+HTTP_PROXY http://proxy_host:port
+HTTPS_PROXY http://proxy_host:port
+NO_PROXY localhost,127.0.0.1,.example.com
+```
+
+Now run `vagrant up` in your terminal to install the development
+server. If you ran `vagrant up` before and failed, you'll need to run
+`vagrant destroy` first to clean up the failed installation.
+
+You can also change the port on the host machine that Vagrant uses by
+adding to your `~/.zulip-vagrant-config` file.  E.g. if you set:
+
+```
+HOST_PORT 9971
+```
+
+(and halt and restart the Vagrant guest), then you would visit
+http://localhost:9971/ to connect to your development server.

docs/changelog.md

@@ -4,6 +4,125 @@ All notable changes to the Zulip server are documented in this file.
 
 ### Unreleased
 
+### 1.4.3 - 2017-01-29
+- CVE-2017-0881: Users could subscribe to invite-only streams.
+
+### 1.4.2 - 2016-09-27
+- Upgraded Django to version 1.8.15 (with the Zulip patches applied),
+  fixing a CSRF vulnerability in Django (see
+  https://www.djangoproject.com/weblog/2016/sep/26/security-releases/),
+  and a number of other Django bugs from past Django stable releases
+  that largely affects parts of Django that are not used by Zulip.
+- Fixed buggy logrotate configuration.
+
+### 1.4.1 - 2016-09-03
+- Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0.
+- Fixed local file uploads integration being broken for new 1.4.0
+  installations.
+
+### 1.4 - 2016-08-25
+
+- Migrated Zulip's python dependencies to be installed via a virtualenv,
+  instead of the via apt.  This is a major change to how Zulip
+  is installed that we expect will simplify upgrades in the future.
+- Fixed unnecessary loading of zxcvbn password strength checker.  This
+  saves a huge fraction of the uncached network transfer for loading
+  Zulip.
+- Added support for using Ubuntu Xenial in production.
+- Added a powerful and complete realm import/export tool.
+- Added nice UI for selecting a default language to display settings.
+- Added UI for searching streams in left sidebar with hotkeys.
+- Added Semaphore, Bitbucket, and HelloWorld (example) integrations.
+- Added new webhook-based integration for Trello.
+- Added management command for creating realms through web UI.
+- Added management command to send password reset emails.
+- Added endpoint for mobile apps to query available auth backends.
+- Added LetsEncrypt documentation for getting SSL certificates.
+- Added nice rendering of unicode emoji.
+- Added support for pinning streams to the top of the left sidebar.
+- Added search box for filtering user list when creating a new stream.
+- Added realm setting to disable message editing.
+- Added realm setting to time-limit message editing.  Default is 10m.
+- Added realm setting for default language.
+- Added year to timestamps in message interstitials for old messages.
+- Added GitHub authentication (and integrated python-social-auth, so it's
+  easy to add additional social authentication methods).
+- Added TERMS_OF_SERVICE setting using markdown formatting to configure
+  the terms of service for a Zulip server.
+- Added numerous hooks to puppet modules to enable more configurations.
+- Moved several useful puppet components into the main puppet
+  manifests (setting a redis password, etc.).
+- Added automatic configuration of postgres/memcached settings based
+  on the server's available RAM.
+- Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo.
+- Added preliminary support for Python 3.  All of Zulip's test suites now
+  pass using Python 3.4.
+- Added support for `Name <email@example.com>` format when inviting users.
+- Added numerous special-purpose settings options.
+- Added a hex input field in color picker.
+- Documented new Electron beta app and mobile apps in the /apps/ page.
+- Enabled Android Google authentication support.
+- Enhanced logic for tracking origin of user uploads.
+- Improved error messages for various empty narrows.
+- Improved missed message emails to better support directly replying.
+- Increased backend test coverage of Python code to 85.5%.
+- Increased mypy static type coverage of Python code to 95%.  Also
+  fixed many string annotations to properly handle unicode.
+- Fixed major i18n-related frontend performance regression on
+  /#subscriptions page.  Saves several seconds of load time with 1k
+  streams.
+- Fixed Jinja2 migration bug when trying to register an email that
+  already has an account.
+- Fixed narrowing to a stream from other pages.
+- Fixed various frontend strings that weren't marked for translation.
+- Fixed several bugs around editing status (/me) messages.
+- Fixed queue workers not restarting after changes in development.
+- Fixed Casper tests hanging while development server is running.
+- Fixed browser autocomplete issue when adding new stream members.
+- Fixed broken create_stream and rename_stream management commands.
+- Fixed zulip-puppet-apply exit code when puppet throws errors.
+- Fixed EPMD restart being attempted on every puppet apply.
+- Fixed message cache filling; should improve perf after server restart.
+- Fixed caching race condition when changing user objects.
+- Fixed buggy puppet configuration for supervisord restarts.
+- Fixed some error handling race conditions when editing messages.
+- Fixed fastcgi_params to protect against the httpoxy attack.
+- Fixed bug preventing users with mit.edu emails from registering accounts.
+- Fixed incorrect settings docs for the email mirror.
+- Fixed APNS push notification support (had been broken by Apple changing
+  the APNS API).
+- Fixed some logic bugs in how attachments are tracked.
+- Fixed unnecessarily resource-intensive rabbitmq cron checks.
+- Fixed old deployment directories leaking indefinitely.
+- Fixed need to manually add localhost in ALLOWED_HOSTS.
+- Fixed display positioning for the color picker on subscriptions page.
+- Fixed escaping of Zulip extensions to markdown.
+- Fixed requiring a reload to see newly uploaded avatars.
+- Fixed @all warning firing even for `@all`.
+- Restyled password reset form to look nice.
+- Improved formatting in reset password links.
+- Improved alert words UI to match style of other settings.
+- Improved error experience when sending to nonexistent users.
+- Portions of integrations documentation are now automatically generated.
+- Restructured the URLs files to be more readable.
+- Upgraded almost all Python dependencies to current versions.
+- Substantially expanded and reorganized developer documentation.
+- Reorganized production documentation and moved to ReadTheDocs.
+- Reorganized .gitignore type files to be written under var/
+- Refactored substantial portions of templates to support subdomains.
+- Renamed local_settings.py symlink to prod_settings.py for clarity.
+- Renamed email-mirror management command to email_mirror.
+- Changed HTTP verb for create_user_backend to PUT.
+- Eliminated all remaining settings hardcoded for zulip.com.
+- Eliminated essentially all remaining hardcoding of mit.edu.
+- Optimized the performance of all the test suites.
+- Optimized Django memcached configuration.
+- Removed old prototype data export tool.
+- Disabled insecure RC4 cipher in nginx configuration.
+- Enabled shared SSL session cache in nginx configuration.
+- Updated header for Zulip static assets to reflect Zulip being
+  open source.
+
 ### 1.3.13 - 2016-06-21
 - Added nearly complete internationalization of the Zulip UI.
 - Added warning when using @all/@everyone.
@@ -39,7 +158,7 @@ All notable changes to the Zulip server are documented in this file.
 - Migrated development environment setup scripts to tools/setup/.
 - Expanded test coverage for several areas of the product.
 - Simplified the API for writing new webhook integrations.
-- Removed most of the remaining javascript global variables.
+- Removed most of the remaining JavaScript global variables.
 
 ### 1.3.12 - 2016-05-10
 - CVE-2016-4426: Bot API keys were accessible to other users in the same realm.

docs/code-contribution-checklist.rst

@@ -1,24 +0,0 @@
-=======================
-Code contribution guide
-=======================
-
-Thanks for contributing to Zulip! We hope this guide helps make the
-process smooth sailing for you.
-
-The "Ways to Contribute" and "How to get involved" sections of
-:doc:`readme-symlink` are a good place to start! After that, check out
-the tips below.
-
-
-.. toctree::
-   :maxdepth: 2
-
-   code-style
-   markdown
-   testing
-   logging
-   mypy
-   settings
-   front-end-build-process
-   queuing
-   schema-migrations

docs/code-style.md

@@ -1,8 +1,6 @@
-Code style and conventions
-==========================
+# Code style and conventions
 
-Be consistent!
---------------
+## Be consistent!
 
 Look at the surrounding code, or a similar part of the project, and try
 to do the same thing. If you think the other code has actively bad
@@ -11,8 +9,7 @@ style, fix it (in a separate commit).
 When in doubt, send an email to <zulip-devel@googlegroups.com> with your
 question.
 
-Lint tools
-----------
+## Lint tools
 
 You can run them all at once with
 
@@ -26,7 +23,7 @@ The Vagrant setup process runs this for you.
 
 `lint-all` runs many lint checks in parallel, including
 
--   Javascript ([JSLint](http://www.jslint.com/))
+-   JavaScript ([JSLint](http://www.jslint.com/))
 
     > `tools/jslint/check-all.js` contains a pretty fine-grained set of
     > JSLint options, rule exceptions, and allowed global variables. If
@@ -37,15 +34,13 @@ The Vagrant setup process runs this for you.
 -   Puppet configuration
 -   custom checks (e.g. trailing whitespace and spaces-not-tabs)
 
-Secrets
--------
+## Secrets
 
 Please don't put any passwords, secret access keys, etc. inline in the
 code. Instead, use the `get_secret` function in `zproject/settings.py`
 to read secrets from `/etc/zulip/secrets.conf`.
 
-Dangerous constructs
---------------------
+## Dangerous constructs
 
 ### Misuse of database queries
 
@@ -124,9 +119,9 @@ string, use the `id` function, as it will simplify future code changes.
 In most contexts in JavaScript where a string is needed, you can pass a
 number without any explicit conversion.
 
-### Javascript var
+### JavaScript var
 
-Always declare Javascript variables using `var`:
+Always declare JavaScript variables using `var`:
 
     var x = ...;
 
@@ -134,12 +129,12 @@ In a function, `var` is necessary or else `x` will be a global variable.
 For variables declared at global scope, this has no effect, but we do it
 for consistency.
 
-Javascript has function scope only, not block scope. This means that a
+JavaScript has function scope only, not block scope. This means that a
 `var` declaration inside a `for` or `if` acts the same as a `var`
 declaration at the beginning of the surrounding `function`. To avoid
 confusion, declare all variables at the top of a function.
 
-### Javascript `for (i in myArray)`
+### JavaScript `for (i in myArray)`
 
 Don't use it:
 [[1]](http://stackoverflow.com/questions/500504/javascript-for-in-with-arrays),
@@ -170,8 +165,7 @@ current working directory for the app changes every time we do a deploy.
 Instead, hardcode a path in settings.py -- see SERVER\_LOG\_PATH in
 settings.py for an example.
 
-JS array/object manipulation
-----------------------------
+## JS array/object manipulation
 
 For generic functions that operate on arrays or JavaScript objects, you
 should generally use [Underscore](http://underscorejs.org/). We used to
@@ -201,8 +195,7 @@ canonical name (given in large print in the Underscore documentation),
 with the exception of `_.any`, which we prefer over the less clear
 'some'.
 
-More arbitrary style things
----------------------------
+## More arbitrary style things
 
 ### General
 
@@ -236,7 +229,7 @@ Whitespace guidelines:
     used for inline dictionaries, put no space before it and at least
     one space after. Only use more than one space for alignment.
 
-### Javascript
+### JavaScript
 
 Don't use `==` and `!=` because these operators perform type coercions,
 which can mask bugs. Always use `===` and `!==`.
@@ -305,7 +298,7 @@ call a helper function instead.
 ### HTML / CSS
 
 Don't use the `style=` attribute. Instead, define logical classes and
-put your styles in `zulip.css`.
+put your styles in external files such as `zulip.css`.
 
 Don't use the tag name in a selector unless you have to. In other words,
 use `.foo` instead of `span.foo`. We shouldn't have to care if the tag
@@ -354,123 +347,6 @@ styles (separate lines for each selector):
 
     in case we ever change the primary keys.
 
-Version Control
----------------
-
-### Commit Discipline
-
-We follow the Git project's own commit discipline practice of "Each
-commit is a minimal coherent idea". This discipline takes a bit of work,
-but it makes it much easier for code reviewers to spot bugs, and
-makes the commit history a much more useful resource for developers
-trying to understand why the code works the way it does, which also
-helps a lot in preventing bugs.
-
-Coherency requirements for any commit:
-
--   It should pass tests (so test updates needed by a change should be
-    in the same commit as the original change, not a separate "fix the
-    tests that were broken by the last commit" commit).
--   It should be safe to deploy individually, or comment in detail in
-    the commit message as to why it isn't (maybe with a [manual] tag).
-    So implementing a new API endpoint in one commit and then adding the
-    security checks in a future commit should be avoided -- the security
-    checks should be there from the beginning.
--   Error handling should generally be included along with the code that
-    might trigger the error.
--   TODO comments should be in the commit that introduces the issue or
-    functionality with further work required.
-
-When you should be minimal:
-
--   Significant refactorings should be done in a separate commit from
-    functional changes.
--   Moving code from one file to another should be done in a separate
-    commits from functional changes or even refactoring within a file.
--   2 different refactorings should be done in different commits.
--   2 different features should be done in different commits.
--   If you find yourself writing a commit message that reads like a list
-    of somewhat dissimilar things that you did, you probably should have
-    just done 2 commits.
-
-When not to be overly minimal:
-
--   For completely new features, you don't necessarily need to split out
-    new commits for each little subfeature of the new feature. E.g. if
-    you're writing a new tool from scratch, it's fine to have the
-    initial tool have plenty of options/features without doing separate
-    commits for each one. That said, reviewing a 2000-line giant blob of
-    new code isn't fun, so please be thoughtful about submitting things
-    in reviewable units.
--   Don't bother to split back end commits from front end commits, even
-    though the backend can often be coherent on its own.
-
-Other considerations:
-
--   Overly fine commits are easily squashed, but not vice versa, so err
-    toward small commits, and the code reviewer can advise on squashing.
--   If a commit you write doesn't pass tests, you should usually fix
-    that by amending the commit to fix the bug, not writing a new "fix
-    tests" commit on top of it.
-
-Zulip expects you to structure the commits in your pull requests to form
-a clean history before we will merge them; it's best to write your
-commits following these guidelines in the first place, but if you don't,
-you can always fix your history using git rebase -i.
-
-It can take some practice to get used to writing your commits with a
-clean history so that you don't spend much time doing interactive
-rebases. For example, often you'll start adding a feature, and discover
-you need to a refactoring partway through writing the feature. When that
-happens, we recommend stashing your partial feature, do the refactoring,
-commit it, and then finish implementing your feature.
-
-### Commit Messages
-
--   The first line of commit messages should be written in the
-    imperative and be kept relatively short while concisely explaining
-    what the commit does. For example:
-
-Bad:
-
-    bugfix
-    gather_subscriptions was broken
-    fix bug #234.
-
-Good:
-
-    Fix gather_subscriptions throwing an exception when given bad input.
-
--   Use present-tense action verbs in your commit messages.
-
-Bad:
-
-    Fixing gather_subscriptions throwing an exception when given bad input.
-    Fixed gather_subscriptions throwing an exception when given bad input.
-
-Good:
-
-    Fix gather_subscriptions throwing an exception when given bad input.
-
--   Please use a complete sentence in the summary, ending with a period.
--   The rest of the commit message should be written in full prose and
-    explain why and how the change was made. If the commit makes
-    performance improvements, you should generally include some rough
-    benchmarks showing that it actually improves the performance.
--   When you fix a GitHub issue, [mark that you've fixed the issue in
-    your commit
-    message](https://help.github.com/articles/closing-issues-via-commit-messages/)
-    so that the issue is automatically closed when your code is merged.
-    Zulip's preferred style for this is to have the final paragraph of
-    the commit message read e.g. "Fixes: \#123."
--   Any paragraph content in the commit message should be line-wrapped
-    to less than 76 characters per line, so that your commit message
-    will be reasonably readable in git log in a normal terminal.
--   In your commit message, you should describe any manual testing you
-    did in addition to running the automated tests, and any aspects of
-    the commit that you think are questionable and you'd like special
-    attention applied to.
-
 ### Tests
 
 All significant new features should come with tests. See testing.

docs/conf.py

@@ -15,7 +15,7 @@
 import sys
 import os
 import shlex
-from typing import Dict, List, Optional
+if False: from typing import Any, Dict, List, Optional
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -43,7 +43,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Zulip'
-copyright = u'2015, The Zulip Team'
+copyright = u'2015-2016, The Zulip Team'
 author = u'The Zulip Team'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -51,9 +51,9 @@ author = u'The Zulip Team'
 # built documents.
 #
 # The short X.Y version.
-version = '0.1'
+version = '1.4'
 # The full version, including alpha/beta/rc tags.
-release = '0.1'
+release = '1.4.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -206,6 +206,8 @@ html_static_path = ['_static']
 htmlhelp_basename = 'zulip-contributor-docsdoc'
 
 def setup(app):
+    # type: (Any) -> None
+
     # overrides for wide tables in RTD theme
     app.add_stylesheet('theme_overrides.css')   # path relative to _static
 

docs/conversion.md

@@ -0,0 +1,242 @@
+# Exporting data
+
+## Overview
+
+Occasionally Zulip administrators will need to move data from one
+server to another.
+
+There are many major operational aspects to doing a conversion. I will
+list them here, noting that several are not within the scope of this
+document:
+
+- Get new servers running.
+- Export data from the old DB.
+- Export files from S3.
+- Import files into new storage.
+- Import data into new DB.
+- Restart new servers.
+- Decommission old server.
+
+This document focuses almost entirely on the **export** piece.  Issues
+with getting Zulip itself running are totally out of scope here.  For the
+import side of things, I only touch on it implicity.  (My reasoning is
+that we *have* to get the export piece right in a timely fashion, even
+if it means we have to sort out some straggling issues on the import side
+later.)
+
+## Export
+
+We have tools that essentially export Zulip data to the file system.
+
+A good overview of the process is here:
+[management/export.py](https://github.com/zulip/zulip/blob/master/zerver/management/commands/export.py)
+
+This document supplements that explanation, but here we focus more
+on the logistics of a big conversion.  For some historical perspective,
+this document was originally drafted as part of a big Zulip cut-over.
+
+The main exporting tools in place as of summer 2016 are below:
+
+- We can export single realms (but not yet limit users within the realm).
+- We can export single users (but then we get no realm-wide data in the process).
+- We can run exports simultaneously (but have to navigate a bunch of /tmp directories).
+
+Things that we still may need:
+- We may want to export multiple realms simultaneously.
+- We may want to export multiple single users simultaneously.
+- We may want to limit users within realm exports.
+- We may want more operational robustness/convenience while doing several exports simultaenously.
+- We may want to merge multiple export files to remove duplicates.
+
+We have a few major classes of data.  They are listed below in the order
+that we process them in `do_export_realm()`:
+
+#### Public Realm Data
+
+Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream.
+
+#### Cross Realm Data
+
+Client/zerver_userprofile_cross_realm
+
+This includes Client and three bots.
+
+Client is unique in being a fairly core table that is
+not tied to UserProfile or Realm (unless you somewhat painfully tie
+it back to users in a bottom-up fashion though other tables).
+
+#### Disjoint User Data
+
+UserProfile/UserActivity/UserActivityInterval/UserPresence.
+
+#### Recipient Data
+
+Recipient/Stream/Subscription/Huddle.
+
+These tables are tied back to users, but they introduce complications
+when you try to deal with multi-user subsets.
+
+#### File-related Data
+
+Attachment
+
+This includes Attachment, and it referencs the avatar_source field of
+UserProfile.  Most importantly, of course, it requires us to grab files
+from S3.  Finally, Attachment's m2m relationship ties to Message.
+
+#### Message Data
+
+Message/UserMessage
+
+### Summary
+
+Here are the same classes of data, listed in roughly
+decreasing order of riskiness:
+
+- Message Data (sheer volume/lack of time/security)
+- File-Related Data (S3/security/lots of moving parts)
+- Recipient Data (complexity/security/cross-realm considerations)
+- Cross Realm Data (duplicate ids)
+- Disjoint User Data
+- Public Realm Data
+
+(Note the above list is essentially in reverse order of how we
+process the data, which isn't surprising for a top-down approach.)
+
+The next section of the document talks about risk factors.
+
+# Risk Mitigation
+
+## Generic considerations
+
+We have two major mechanisms for getting data:
+
+##### Top Down
+
+Get realm data, then all users in realm, then all recipients, then all messages, etc.
+
+The problem with the top down approach will be **filtering**.  Also, if
+errors arise during top-down passes, it may be time consuming to re-run
+the processes.
+
+##### Bottom Up
+
+Start with users, get their recipient data, etc.
+
+The problems with the bottom up approach will be **merging**.  Also, if
+we run multiple bottom-up passes, there is the danger of duplicating some
+work, particularly on the message side of things.
+
+### Approved Transfers
+
+We have not yet integrated the approved-transfer model, which tells us
+which users can be moved.
+
+## Risk factors broken out by data categories
+
+### Message Data
+
+- models: Message/UserMessage.
+- assets: messages-*.json, subprocesses, partial files
+
+Rows in the Message model depend on Recipient/UserProfile.
+
+Rows in the UserMessage model depend on UserProfile/Message.
+
+The biggest concern here is the **sheer volume** of data, with
+security being a close second.  (They are interrelated, as without
+security concerns, we could just bulk-export everything one time.)
+
+We currently have these measures in place for top-down processing:
+- chunking
+- multi-processing
+- messages are filtered by both sender and recipient
+
+
+### File Related Data
+
+- models: Attachment
+- assets: S3, attachment.json, uploads-temp/, image files in avatars/, assorted files in uploads/, avatars/records.json, uploads/records.json, zerver_attachment_messages
+
+When it comes to exporting attachment data, we have some minor volume issues, but the
+main concern is just that there are **lots of moving parts**:
+
+- S3 needs to be up, and we get some metadata from it as well as files.
+- We have security concerns about copying over only files that belong to users who approved the transfer.
+- This piece is just different in how we store data from all the other DB-centric pieces.
+- At import time we have to populate the m2m table (but fortunately, this is pretty low
+  risk in terms of breaking anything.)
+
+### Recipient Data
+- models: Recipient/Stream/Subscription/Huddle
+- assets: realm.json, (user,stream,huddle)_(recipient,subscription)
+
+This data is fortunately low to medium in volume.  The risk here will come
+from **model complexity** and **cross-realm concerns**.
+
+From the top down, here are the dependencies:
+
+- Recipient depends on UserProfile
+- Subscription depends on Recipient
+- Stream currently depends on Realm (but maybe it should be tied to Subscription)
+- Huddle depends on Subscription and UserProfile
+
+The biggest risk factor here is probably just the possibility that we could introduce
+some bug in our code as we try to segment Recipient into user, stream, and huddle components,
+especially if we try to handle multiple users or realms.
+I think this can be largely mitigated by the new Config approach.
+
+And then we also have some complicated Huddle logic that will be customized
+regardless.  The fiddliest part
+of the Huddle logic is creating the set of unsafe_huddle_recipient_ids.
+
+Last but not least, if we go with some hybrid of bottom-up and top-down, these tables
+are neither close to the bottom nor close to the top, so they may have the most
+fiddly edge cases when it comes to filtering and merging.
+
+Recommendation: We probably want to get a backup of all this data that is very simply
+bulk-exported from the entire DB, and we should obviously put it in a secure place.
+
+### Cross Realm Data
+- models: Client
+- assets: realm.json, three bots (notification/email/welcome), id_maps
+
+The good news here is that Client is a small table, and there are
+only three special bots.
+
+The bad news is that cross-realm data **complicates everything else**,
+and we have to avoid **database id conflicts**.
+
+If we use bottom-up approaches to load small user populations at a time, we may
+have **merging** issues here.  We will need to consolidate ids either by merging
+exports in /tmp or handle it import time.
+
+For the three bots, they live in zerver_userprofile_crossrealm, and we re-map
+their ids on the new server.
+
+Recommendation: Do not sweat the exports too much.  Deal with all the messiness at
+import time, and rely on the tables being really small.  We already have logic
+to catch Client.DoesNotExist exceptions, for example.  As for possibly missing
+messages that the welcome bot and friends have sent in the past, I am not sure
+what our risk profile is there, but I imagine it is relatively low.
+
+### Disjoint User Data
+- models: UserProfile/UserActivity/UserActivityInterval/UserPresence
+- assets: realm.json, password, api_key, avatar salt, id_maps
+
+On the DB side this data should be fairly easy to deal with.  All of these
+tables are basically disjoint by user profile id.  Our biggest
+risk is **remapped user ids** at import time, but this is mostly covered
+in the section above.
+
+We have code in place to exclude password and api_key from UserProfile
+rows.  The import process calls set_unusable_password().
+
+### Public Realm Data
+
+- models: Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream
+- asserts: realm.json
+
+All of these tables are public (per-realm), and they are keyed by
+realm id.  There is not a ton to worry about here, except possibly
+**merging** if we run multiple bottom-up jobs for a single realm.

docs/dev-env-first-time-contributors.md

@@ -0,0 +1,697 @@
+## Vagrant environment setup tutorial
+
+This section guides first-time contributors through installing the Zulip dev
+environment on Windows 10, OS X El Capitan, Ubuntu 14.04, and Ubuntu 16.04.
+
+The recommended method for installing the Zulip dev environment is to use
+Vagrant with VirtualBox on Windows and OS X, and Vagrant with LXC on
+Ubuntu. This method creates a virtual machine (for Windows and OS X)
+or a Linux container (for Ubuntu) inside which the Zulip server and
+all related services will run.
+
+Contents:
+* [Requirements](#requirements)
+* [Step 1: Install Prerequisites](#step-1-install-prerequisites)
+* [Step 2: Get Zulip code](#step-2-get-zulip-code)
+* [Step 3: Start the dev environment](#step-3-start-the-dev-environment)
+* [Step 4: Developing](#step-4-developing)
+* [Troubleshooting & Common Errors](#troubleshooting-common-errors)
+
+If you encounter errors installing the Zulip development environment,
+check [Troubleshooting & Common
+Errors](#troubleshooting-common-errors). If that doesn't help, please
+visit [the `provision` stream in the Zulip developers'
+chat](https://zulip.tabbott.net/#narrow/stream/provision) for realtime
+help, or send a note to the [Zulip-devel Google
+group](https://groups.google.com/forum/#!forum/zulip-devel) or [file
+an issue](https://github.com/zulip/zulip/issues).
+
+### Requirements
+
+Installing the Zulip dev environment requires downloading several
+hundred megabytes of dependencies. You will need an active internet
+connection throughout the entire installation processes. (See
+[Specifying a
+proxy](brief-install-vagrant-dev.html#specifying-a-proxy) if you need
+a proxy to access the internet.)
+
+
+- **All**: 2GB available RAM, Active broadband internet connection.
+- **OS X**: OS X (El Capitan recommended, untested on previous versions), Git,
+  [VirtualBox][vbox-dl], [Vagrant][vagrant-dl].
+- **Ubuntu**: 14.04 64-bit or 16.04 64-bit, Git, [Vagrant][vagrant-dl], lxc.
+- **Windows**: Windows 64-bit (Win 10 recommended; Win 7 untested), hardware
+  virtualization enabled (VT-X or AMD-V), administrator access,
+  [Cygwin][cygwin-dl], [VirtualBox][vbox-dl], [Vagrant][vagrant-dl].
+
+Don't see your system listed above? Check out:
+* [Brief installation instructions for Vagrant development
+  environment](brief-install-vagrant-dev.html)
+* [Installing manually on UNIX-based platforms](install-generic-unix-dev.html)
+
+[cygwin-dl]: http://cygwin.com/
+
+### Step 1: Install Prerequisites
+
+Jump to:
+
+* [OS X](#os-x)
+* [Ubuntu](#ubuntu)
+* [Windows](#windows-10)
+
+#### OS X
+
+1. Install [VirtualBox][vbox-dl]
+2. Install [Vagrant][vagrant-dl]
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+#### Ubuntu
+
+The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same.
+
+If you're in a hurry, you can copy and paste the following into your terminal
+after which you can jump to [Step 2: Get Zulip Code](#step-2-get-zulip-code):
+
+```
+sudo apt-get purge vagrant
+wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb
+sudo dpkg -i vagrant*.deb
+sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
+vagrant plugin install vagrant-lxc
+vagrant lxc sudoers
+```
+
+For a step-by-step explanation, read on.
+
+##### 1. Install Vagrant
+
+For both 14.04 Trusty and 16.04 Xenial, you'll need a more recent version of
+Vagrant than what's available in the official Ubuntu repositories.
+
+First uninstall any vagrant package you may have installed from the Ubuntu
+repository:
+
+```
+christie@ubuntu-desktop:~
+$ sudo apt-get purge vagrant
+```
+
+Now download and install the most recent .deb package from [Vagrant][vagrant-dl]:
+
+```
+christie@ubuntu-desktop:~
+$ wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.deb
+
+christie@ubuntu-desktop:~
+$ sudo dpkg -i vagrant*.deb
+```
+
+##### 2. Install remaining dependencies
+
+Now install git and lxc-related packages:
+
+```
+christie@ubuntu-desktop:~
+$ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
+```
+
+##### 3. Install the vagrant lxc plugin:
+
+```
+christie@ubuntu-desktop:~
+$ vagrant plugin install vagrant-lxc
+Installing the 'vagrant-lxc' plugin. This can take a few minutes...
+Installed the plugin 'vagrant-lxc (1.2.1)'!
+```
+
+If you encounter an error when trying to install the vagrant-lxc plugin, [see
+this](#nomethoderror-when-installing-vagrant-lxc-plugin-ubuntu-1604).
+
+##### 4. Configure sudo to be passwordless
+
+Finally, [configure sudo to be passwordless when using Vagrant LXC][avoiding-sudo]:
+
+```
+christie@ubuntu-desktop:~
+$ vagrant lxc sudoers
+[sudo] password for christie:
+```
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+[vagrant-dl]: https://www.vagrantup.com/downloads.html
+[vagrant-lxc]: https://github.com/fgrehm/vagrant-lxc
+[vbox-dl]: https://www.virtualbox.org/wiki/Downloads
+[avoiding-sudo]: https://github.com/fgrehm/vagrant-lxc#avoiding-sudo-passwords
+
+#### Windows 10
+
+1. Install [Cygwin][cygwin-dl]. Make sure to install default required
+   packages along with **git**, **curl**, **openssh**, and **rsync**
+   binaries.
+2. Install [VirtualBox][vbox-dl]
+3. Install [Vagrant][vagrant-dl]
+
+##### Configure Cygwin
+
+In order for symlinks to work within the Ubuntu virtual machine, you must tell
+Cygwin to create them as [native Windows
+symlinks](https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks). The
+easiest way to do this is to add a line to `~/.bash_profile` setting the CYGWIN
+environment variable.
+
+Open a Cygwin window and do this:
+
+```
+christie@win10 ~
+$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
+```
+
+Next, close that Cygwin window and open another. If you `echo` $CYGWIN you
+should see:
+
+```
+christie@win10 ~
+$ echo $CYGWIN
+winsymlinks:native
+```
+
+Now you are ready for [Step 2: Get Zulip Code.](#step-2-get-zulip-code)
+
+### Step 2: Get Zulip Code
+
+If you haven't already created an ssh key and added it to your Github account,
+you should do that now by following [these
+instructions](https://help.github.com/articles/generating-an-ssh-key/).
+
+1. In your browser, visit https://github.com/zulip/zulip and click the
+   `fork` button. You will need to be logged in to Github to do this.
+2. Open Terminal (OS X/Ubuntu) or Cygwin (Windows; must run as an Administrator)
+3. In Terminal/Cygwin, clone your fork:
+```
+git clone git@github.com:YOURUSERNAME/zulip.git
+```
+
+This will create a 'zulip' directory and download the Zulip code into it.
+
+Don't forget to replace YOURUSERNAME with your git username. You will see
+something like:
+
+```
+christie@win10 ~
+$ git clone git@github.com:YOURUSERNAME/zulip.git
+Cloning into 'zulip'...
+remote: Counting objects: 73571, done.
+remote: Compressing objects: 100% (2/2), done.
+remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569
+Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done.
+Resolving deltas: 100% (51448/51448), done.
+Checking connectivity... done.
+Checking out files: 100% (1912/1912), done.`
+```
+
+Now you are ready for [Step 3: Start the dev
+environment.](#step-3-start-the-dev-environment)
+
+### Step 3: Start the dev environment
+
+Change into the zulip directory and tell vagrant to start the Zulip
+dev environment with `vagrant up`.
+
+```
+christie@win10 ~
+$ cd zulip
+
+christie@win10 ~/zulip
+$ vagrant up
+```
+
+The first time you run this command it will take some time because vagrant
+does the following:
+
+- downloads the base Ubuntu 14.04 virtual machine image (for OS X and Windows)
+  or container (for Ubuntu)
+- configures this virtual machine/container for use with Zulip,
+- creates a shared directory mapping your clone of the Zulip code inside the
+  virtual machine/container at `/srv/zulip`
+- runs the `tools/provision.py` script inside the virtual machine/container, which
+  downloads all required dependencies, sets up the python environment for
+  the Zulip dev environment, and initializes a default test database.
+
+You will need an active internet connection during the entire
+processes. (See [Specifying a
+proxy](brief-install-vagrant-dev.html#specifying-a-proxy) if you need
+a proxy to access the internet.) And if you're running into any
+problems, please come chat with us [in the `provision` stream of our
+developers' chat](https://zulip.tabbott.net/#narrow/stream/provision).
+
+Once `vagrant up` has completed, connect to the dev environment with `vagrant
+ssh`:
+
+```
+christie@win10 ~/zulip
+$ vagrant ssh
+```
+
+You should see something like this on Windows and OS X:
+
+```
+Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com/
+
+  System information as of Wed May  4 21:45:43 UTC 2016
+
+  System load:  0.61              Processes:           88
+  Usage of /:   3.5% of 39.34GB   Users logged in:     0
+  Memory usage: 7%                IP address for eth0: 10.0.2.15
+  Swap usage:   0%
+
+  Graph this data and manage this system at:
+    https://landscape.canonical.com/
+
+  Get cloud support with Ubuntu Advantage Cloud Guest:
+    http://www.ubuntu.com/business/services/cloud
+
+0 packages can be updated.
+0 updates are security updates.
+```
+
+Or something as brief as this in the case of Ubuntu:
+
+```
+Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com/
+```
+
+Congrats, you're now inside the Zulip dev environment!
+
+You can confirm this by looking at the command prompt, which starts with
+`(zulip-venv)`.
+
+Next, start the Zulip server:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+As you can see above the application's root directory, where you can
+execute Django's command line utilities is:
+
+```
+/srv/zulip/
+```
+
+You will see several lines of output starting with something like:
+
+```
+2016-05-04 22:20:33,895 INFO: process_fts_updates starting
+Recompiling templates
+2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates
+done
+Validating Django models.py...
+System check identified no issues (0 silenced).
+
+Django version 1.8
+Tornado server is running at http://localhost:9993/
+Quit the server with CTRL-C.
+2016-05-04 18:20:40,716 INFO     Tornado loaded 0 event queues in 0.001s
+2016-05-04 18:20:40,722 INFO     Tornado  95.5% busy over the past  0.0 seconds
+Performing system checks...
+```
+And ending with something similar to:
+
+```
+http://localhost:9994/webpack-dev-server/
+webpack result is served from http://localhost:9991/webpack/
+content is served from /srv/zulip
+
+webpack: bundle is now VALID.
+2016-05-06 21:43:29,553 INFO     Tornado  31.6% busy over the past 10.6 seconds
+2016-05-06 21:43:35,007 INFO     Tornado  23.9% busy over the past 16.0 seconds
+```
+
+Now the Zulip server should be running and accessible. Verify this by
+navigating to [http://localhost:9991/](http://localhost:9991/) in your browser
+on your main machine.
+
+You should see something like [(this screenshot of the Zulip dev
+environment)](https://raw.githubusercontent.com/zulip/zulip/master/docs/images/zulip-dev.png).
+
+![Image of Zulip dev environment](https://raw.githubusercontent.com/zulip/zulip/master/docs/images/zulip-dev.png)
+
+The Zulip server will continue to run and send output to the terminal window.
+When you navigate to Zulip in your browser, check your terminal and you
+should see something like:
+
+```
+2016-05-04 18:21:57,547 INFO     127.0.0.1       GET     302 582ms (+start: 417ms) / (unauth via ?)
+[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0
+2016-05-04 18:21:57,568 INFO     127.0.0.1       GET     301   4ms /login (unauth via ?)
+[04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0
+2016-05-04 18:21:57,819 INFO     127.0.0.1       GET     200 209ms (db: 7ms/2q) /login/ (unauth via ?)
+```
+
+Now you're ready for [Step 4: Developing.](#step-4-developing)
+
+### Step 4: Developing
+
+#### Where to edit files
+
+You'll work by editing files on your host machine, in the directory where you
+cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++,
+etc.).
+
+When you save changes they will be synced automatically to the Zulip dev environment
+on the virtual machine/container.
+
+Each component of the Zulip development server will automatically
+restart itself or reload data appropriately when you make changes. So,
+to see your changes, all you usually have to do is reload your
+browser.  More details on how this works are available below.
+
+Don't forget to read through the [code style
+guidelines](https://zulip.readthedocs.io/en/latest/code-style.html#general) for
+details about how to configure your editor for Zulip. For example, indentation
+should be set to 4 spaces rather than tabs.
+
+#### Understanding run-dev.py debugging output
+
+It's good to have the terminal running `run-dev.py` up as you work since error
+messages including tracebacks along with every backend request will be printed
+there.
+
+See [Logging](http://zulip.readthedocs.io/en/latest/logging.html) for
+further details on the run-dev.py console output.
+
+#### Committing and pushing changes with git
+
+When you're ready to commit or push changes via git, you will do this by
+running git commands in Terminal (OS X/Ubuntu) or Cygwin (Windows) in the directory
+where you cloned Zulip on your main machine.
+
+If you're new to working with Git/Github, check out [this
+guide](https://help.github.com/articles/create-a-repo/#commit-your-first-change).
+
+#### Maintaining the dev environment
+
+If after rebasing onto a new version of the Zulip server, you receive
+new errors while starting the Zulip server or running tests, this is
+probably not because Zulip's master branch is broken.  Instead, this
+is likely because we've recently merged changes to the development
+environment provisioning process that you need to apply to your
+development environmnet.  To update your environment, you'll need to
+re-provision your vagrant machine using `vagrant provision`
+(or just `python tools/provision.py` from `/srv/zulip` inside the Vagrant
+guest); this should be pretty fast and we're working to make it faster.
+
+See also the documentation on the [testing
+page](http://zulip.readthedocs.io/en/latest/testing.html#manual-testing-local-app-web-browser)
+for how to destroy and rebuild your database if you want to clear out test data.
+
+#### Rebuilding the dev environment
+
+If you ever want to recreate your development environment again from
+scratch (e.g. to test as change you've made to the provisioning
+process, or because you think something is broken), you can do so
+using `vagrant destroy` and then `vagrant up`.  This will usually be
+much faster than the original `vagrant up` since the base image is
+already cached on your machine (it takes about 5 minutes to run with a
+fast Internet connection).
+
+#### Shutting down the dev environment for use later
+
+To shut down but preserve the dev environment so you can use it again
+later use `vagrant halt` or `vagrant suspend`.
+
+You can do this from the same Terminal/Cygwin window that is running
+run-dev.py by pressing ^C to halt the server and then typing `exit`. Or you
+can halt vagrant from another Terminal/Cygwin window.
+
+From the window where run-dev.py is running:
+
+```
+2016-05-04 18:33:13,330 INFO     127.0.0.1       GET     200  92ms /register/ (unauth via ?)
+^C
+KeyboardInterrupt
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ exit
+logout
+Connection to 127.0.0.1 closed.
+christie@win10 ~/zulip
+```
+Now you can suspend the dev environment:
+
+```
+christie@win10 ~/zulip
+$ vagrant suspend
+==> default: Saving VM state and suspending execution...
+```
+
+If `vagrant suspend` doesn't work, try `vagrant halt`:
+
+```
+christie@win10 ~/zulip
+$ vagrant halt
+==> default: Attempting graceful shutdown of VM...
+```
+
+Check out the Vagrant documentation to learn more about
+[suspend](https://www.vagrantup.com/docs/cli/suspend.html) and
+[halt](https://www.vagrantup.com/docs/cli/halt.html).
+
+#### Resuming the dev environment
+
+When you're ready to work on Zulip again, run `vagrant up`. You will also need
+to connect to the virtual machine with `vagrant ssh` and re-start the Zulip
+server:
+
+```
+christie@win10 ~/zulip
+$ vagrant up
+$ vagrant ssh
+/srv/zulip/tools/run-dev.py --interface=''
+```
+
+#### Next Steps
+
+At this point you should [read about using the development
+environment][using-dev-environment.html].
+
+### Troubleshooting & Common Errors
+
+Zulip's `vagrant` provisioning process logs useful debugging output to
+`/var/log/zulip_provision.log`; if you encounter a new issue, please
+attach a copy of that file to your bug report.
+
+#### The box 'ubuntu/trusty64' could not be found (Windows/Cygwin)
+
+If you see the following error when you run `vagrant up` on Windows:
+
+```
+The box 'ubuntu/trusty64' could not be found or
+could not be accessed in the remote catalog. If this is a private
+box on HashiCorp's Atlas, please verify you're logged in via
+`vagrant login`. Also, please double-check the name. The expanded
+URL and error message are shown below:
+URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
+```
+
+Then the version of curl that ships with Vagrant is not working on your
+machine. The fix is simple: replace it with the version from Cygwin.
+
+First, determine the location of Cygwin's curl with `which curl`:
+
+```
+christie@win10 ~/zulip
+$ which curl
+/usr/bin/curl
+```
+Now determine the location of Vagrant with `which vagrant`:
+```
+christie@win10 ~/zulip
+$ which vagrant
+/cygdrive/c/HashiCorp/Vagrant/bin/vagrant
+```
+The path **up until `/bin/vagrant`** is what you need to know. In the example above it's `/cygdrive/c/HashiCorp/Vagrant`.
+
+Finally, copy Cygwin's curl to Vagrant `embedded/bin` directory:
+```
+christie@win10 ~/zulip
+$ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
+```
+
+Now re-run `vagrant up` and vagrant should be able to fetch the required
+box file.
+
+#### os.symlink error
+
+If you receive the following error while running `vagrant up`:
+
+```
+==> default: Traceback (most recent call last):
+==> default: File "./emoji_dump.py", line 75, in <module>
+==> default:
+==> default: os.symlink('unicode/{}.png'.format(code_point), 'out/{}.png'.format(name))
+==> default: OSError
+==> default: :
+==> default: [Errno 71] Protocol error
+```
+
+Then Vagrant was not able to create a symbolic link.
+
+First, if you are using Windows, **make sure you have run Cygwin as an
+administrator**. By default, only administrators can create symbolic links on
+Windows.
+
+Second, VirtualBox does not enable symbolic links by default. Vagrant
+starting with version 1.6.0 enables symbolic links for VirtualBox shared
+folder.
+
+You can check to see that this is enabled for your virtual machine with
+`vboxmanage` command.
+
+Get the name of your virtual machine by running `vboxmanage list vms` and
+then print out the custom settings for this virtual machine with
+`vboxmanage getextradata YOURVMNAME enumerate`:
+
+```
+christie@win10 ~/zulip
+$ vboxmanage list vms
+"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
+
+christie@win10 ~/zulip
+$ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate
+Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1
+Key: supported, Value: false
+```
+
+If you see "command not found" when you try to run VBoxManage, you need to
+add the VirtualBox directory to your path. On Windows this is mostly likely
+`C:\Program Files\Oracle\VirtualBox\`.
+
+If `vboxmanage enumerate` prints nothing, or shows a value of 0 for
+VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable
+symbolic links by running this command in Terminal/Cygwin:
+
+```
+vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip 1
+```
+
+The virtual machine needs to be shut down when you run this command.
+
+#### Connection timeout on `vagrant up`
+
+If you see the following error after running `vagrant up`:
+
+```
+default: SSH address: 127.0.0.1:2222
+default: SSH username: vagrant
+default: SSH auth method: private key
+default: Error: Connection timeout. Retrying...
+default: Error: Connection timeout. Retrying...
+default: Error: Connection timeout. Retrying...
+
+```
+A likely cause is that hardware virtualization is not enabled for your
+computer. This must be done via your computer's BIOS settings. Look for a
+setting called VT-x (Intel) or (AMD-V).
+
+If this is already enabled in your BIOS, double-check that you are running a
+64-bit operating system.
+
+For further information about troubleshooting vagrant timeout errors [see
+this post](http://stackoverflow.com/questions/22575261/vagrant-stuck-connection-timeout-retrying#22575302).
+
+#### npm install error
+
+The `tools/provision.py` script may encounter an error related to `npm install`
+that looks something like:
+
+```
+==> default: + npm install
+==> default: Traceback (most recent call last):
+==> default:   File "/srv/zulip/tools/provision.py", line 195, in <module>
+==> default:
+==> default: sys.exit(main())
+==> default:   File "/srv/zulip/tools/provision.py", line 191, in main
+==> default:
+==> default: run(["npm", "install"])
+==> default:   File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run
+==> default:
+==> default: raise subprocess.CalledProcessError(rc, args)
+==> default: subprocess
+==> default: .
+==> default: CalledProcessError
+==> default: :
+==> default: Command '['npm', 'install']' returned non-zero exit status 34
+The SSH command responded with a non-zero exit status. Vagrant
+assumes that this means the command failed. The output for this command
+should be in the log above. Please read the output to determine what
+went wrong.
+```
+
+Usually this error is not fatal. Try connecting to the dev environment and
+re-trying the command from withing the virtual machine:
+
+```
+christie@win10 ~/zulip
+$ vagrant ssh
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
+$ cd /srv/zulip
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip
+$ npm install
+npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
+npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.12
+```
+
+These are just warnings so it is okay to proceed and start the Zulip server.
+
+#### NoMethodError when installing vagrant-lxc plugin (Ubuntu 16.04)
+
+If you see the following error when you try to install the vagrant-lxc plugin:
+
+```
+/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_by' for nil:NilClass (NoMethodError)
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem'
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install'
+  from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install'
+  from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_plugin'
+  from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin'
+  from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run'
+  from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy'
+  from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run'
+  from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in execute'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each'
+  from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute'
+  from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute'
+  from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute'
+  from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli'
+  from /usr/bin/vagrant:173:in `<main>'
+```
+
+And you have vagrant version 1.8.1, then you need to patch vagrant manually.
+See [this post](https://github.com/mitchellh/vagrant/issues/7073) for an
+explanation of the issue, which should be fixed when Vagrant 1.8.2 is released.
+
+In the meantime, read [this
+post](http://stackoverflow.com/questions/36811863/cant-install-vagrant-plugins-in-ubuntu-16-04/36991648#36991648)
+for how to create and apply the patch.
+
+It will look something like this:
+
+```
+christie@xenial:~
+$ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch
+patching file bundler.rb
+```
+
+#### Permissions errors when running the test suite in LXC
+
+See ["Possible testing issues"](testing.html#possible-testing-issues).

docs/dev-overview.md

@@ -0,0 +1,17 @@
+# Development environment options
+
+Zulip offers a wide range of options for how to install the
+development environment:
+
+* [Detailed tutorial for Vagrant development environment](dev-env-first-time-contributors.html).  Recommended for first-time contributors.
+* [Brief installation instructions for Vagrant development environment](brief-install-vagrant-dev.html)
+* [Installing on Ubuntu 14.04 Trusty or 16.04 Xenial directly](install-ubuntu-without-vagrant-dev.html) (convenient but more work to maintain/uninstall).
+* [Installing manually on other UNIX platforms](install-generic-unix-dev.html)
+* [Using Docker (experimental)](install-docker-dev.html)
+* [Using the Development Environment](using-dev-environment.html)
+* [Testing](testing.html)
+
+If you have a slow network connection, you should probably avoid
+installing Vagrant (which is large) and either install
+[directly](install-ubuntu-without-vagrant-dev.html) or use [the manual
+install process](install-generic-unix-dev.html) instead.

docs/directory-structure.md

@@ -1,112 +1,177 @@
-Directory structure
-===================
+# Directory structure
 
-This page documents the Zulip directory structure and how to decide
-where to put a file.
+This page documents the Zulip directory structure, where to find
+things, and how to decide where to put a file.
 
-Scripts
--------
+You may also find the [new application feature
+tutorial](new-feature-tutorial.html) helpful for understanding the
+flow through these files.
+
+### Core Python files
+
+Zulip uses the [Django web
+framework](https://docs.djangoproject.com/en/1.8/), so a lot of these
+paths will be familiar to Django developers.
+
+* `zproject/urls.py` Main [Django routes file](https://docs.djangoproject.com/en/1.8/topics/http/urls/).  Defines which URLs are handled by which view functions or templates.
+
+* `zerver/models.py` Main [Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/) file.  Defines Zulip's database tables.
+
+* `zerver/lib/actions.py` Most code doing writes to user-facing database tables.
+
+* `zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/).
+
+* `zerver/views/webhooks/` Webhook views for [Zulip integrations](integration-guide.html).
+
+* `zerver/tornadoviews.py` Tornado views.
+
+* `zerver/worker/queue_processors.py` [Queue workers](queuing.html).
+
+* `zerver/lib/*.py` Most library code.
+
+* `zerver/lib/bugdown/` [Backend Markdown processor](markdown.html).
+
+* `zproject/backends.py` [Authentication backends](https://docs.djangoproject.com/en/1.8/topics/auth/customizing/).
+
+-------------------------------------------------------------------
+
+### HTML Templates
+
+See [our translating docs](translating.html) for details on Zulip's
+templating systems.
+
+* `templates/zerver/` For [Jinja2](http://jinja.pocoo.org/) templates for the backend (for zerver app).
+
+* `static/templates/` [Handlebars](http://handlebarsjs.com/) templates for the frontend.
+
+----------------------------------------
+
+### JavaScript and other static assets
+
+* `static/js/` Zulip's own JavaScript.
+
+* `static/styles/` Zulip's own CSS.
+
+* `static/images/` Zulip's images.
+
+* `static/third/` Third-party JavaScript and CSS that has been vendored.
+
+* `node_modules/` Third-party JavaScript installed via `npm`.
+
+* `assets/` For assets not to be served to the web (e.g. the system to
+            generate our favicons).
+
+-----------------------------------------------------------------------
+
+### Tests
+
+* `zerver/tests/` Backend tests.
+
+* `frontend_tests/node_tests/` Node Frontend unit tests.
+
+* `frontend_tests/casper_tests/` Casper frontend tests.
+
+* `tools/test-*` Developer-facing test runner scripts.
+
+-----------------------------------------------------
+
+### Management commands
+
+These are distinguished from scripts, below, by needing to run a
+Django context (i.e. with database access).
+
+* `zerver/management/commands/` Management commands one might run at a
+  production deployment site (e.g. scripts to change a value or
+  deactivate a user properly).
+
+---------------------------------------------------------------
+
+### Scripts
 
 * `scripts/` Scripts that production deployments might run manually
   (e.g., `restart-server`).
 
 * `scripts/lib/` Scripts that are needed on production deployments but
-  humans should never run.
+  humans should never run directly.
 
-* `scripts/setup/` Tools that production deployments will only run
+* `scripts/setup/` Scripts that production deployments will only run
   once, during installation.
 
-* `tools/` Development tools.
+* `tools/` Scripts used only in a Zulip development environment.
+  These are not included in production release tarballs for Zulip, so
+  that we can include scripts here one wouldn't want someone to run in
+  production accidentally (e.g. things that delete the Zulip database
+  without prompting).
+
+* `tools/setup/` Subdirectory of `tools/` for things only used during
+  the development environment setup process.
+
+* `tools/travis/` Subdirectory of `tools/` for things only used to
+  setup and run our tests in Travis CI.  Actually test suites should
+  go in `tools/`.
 
 ---------------------------------------------------------
 
-Bots
-----
+### API and Bots
+
+* `api/` Zulip's Python API bindings (released separately).
+
+* `api/examples/` API examples.
 
 * `api/integrations/` Bots distributed as part of the Zulip API bundle.
 
 * `bots/` Previously Zulip internal bots. These usually need a bit of
    work.
 
------------------------------------------------------
-
-Management commands
--------------------
-
-* `zerver/management/commands/` Management commands one might run at a
-  production deployment site (e.g. scripts to change a value or
-  deactivate a user properly)
-
 -------------------------------------------------------------------------
 
-Views
------
+### Production puppet configuration
 
-* `zerver/tornadoviews.py` Tornado views
+This is used to deploy essentially all configuration in production.
 
-* `zerver/views/webhooks.py` Webhook views
+* `puppet/zulip/` For configuration for production deployments.
 
-* `zerver/views/messages.py` message-related views
+* `puppet/zulip/manifests/voyager.pp` Main manifest for Zulip standalone deployments.
 
-* `zerver/views/__init__.py` other Django views
+-----------------------------------------------------------------------
 
-----------------------------------------
+### Additional Django apps
 
-Jinja2 Compatibility Files
---------------------------
+* `confirmation` Email confirmation system.
 
-* `zproject/jinja2/__init__.py` Jinja2 environment
+* `analytics` Analytics for the Zulip server administrator (needs work to
+  be useful to normal Zulip sites).
 
-* `zproject/jinja2/backends.py` Jinja2 backend
+* `corporate` The old Zulip.com website.  Not included in production
+  distribution.
+
+* `zilencer` Primarily used to hold management commands that aren't
+  used in production.  Not included in production distribution.
+
+-----------------------------------------------------------------------
+
+### Jinja2 Compatibility Files
+
+* `zproject/jinja2/__init__.py` Jinja2 environment.
+
+* `zproject/jinja2/backends.py` Jinja2 backend.
 
 * `zproject/jinja2/compressors.py` Jinja2 compatible functions of
-   Django-Pipeline
+   Django-Pipeline.
 
 -----------------------------------------------------------------------
 
-Static assets
--------------
+### Translation files
 
-* `assets/` For assets not to be served to the web (e.g. the system to
-            generate our favicons)
+* `locale/` Backend (Django) translations data files.
 
-* `static/` For things we do want to both serve to the web and
-            distribute to production deployments (e.g. the webpages)
-
----------------------------------------------------------------
-
-Puppet
-------
-
-* `puppet/zulip/` For configuration for production deployments
-
--------------------------------------------------------------------
-
-Templates
----------
-
-* `templates/zerver/` For Jinja2 templates for the backend (for zerver app)
-
-* `static/templates/` Handlebars templates for the frontend
+* `static/locale/` Frontend translations data files.
 
 -----------------------------------------------------------------------
 
-Tests
------
+### Documentation
 
-* `zerver/tests/` Backend tests
-
-* `frontend_tests/node_tests/` Node Frontend unit tests
-
-* `frontend_tests/casper_tests/` Casper frontend tests
-
------------------------------------------------------------------------
-
-
-Documentation
--------------
-
-*  `docs/`        Source for this documentation
+*  `docs/`        Source for this documentation.
 
 --------------------------------------------------------------
 

docs/front-end-build-process.md

@@ -1,5 +1,4 @@
-Front End Build Process
-=======================
+# Static asset pipeline
 
 This page documents additional information that may be useful when
 developing new features for Zulip that require front-end changes. For a
@@ -7,8 +6,7 @@ more general overview, see the new-feature-tutorial. The code-style
 documentation also has relevant information about how Zulip's code is
 structured.
 
-Primary build process
----------------------
+## Primary build process
 
 Most of the existing JS in Zulip is written in IIFE-wrapped modules, one
 per file in the static/js directory. When running Zulip in development
@@ -20,8 +18,7 @@ If you add a new JavaScript file, it needs to be specified in the
 JS\_SPECS dictionary defined in zproject/settings.py to be included in
 the concatenated file.
 
-Webpack/CommonJS modules
-------------------------
+## Webpack/CommonJS modules
 
 New JS written for Zulip can be written as CommonJS modules (bundled
 using [webpack](https://webpack.github.io/), though this will taken care
@@ -49,8 +46,7 @@ The entry point file for the bundle generated by webpack is
 from this file (or one of its dependencies) in order to be included in
 the script bundle.
 
-Adding static files
--------------------
+## Adding static files
 
 To add a static file to the app (JavaScript, CSS, images, etc), first
 add it to the appropriate place under `static/`.
@@ -59,14 +55,15 @@ add it to the appropriate place under `static/`.
     with "[third]" when adding or modifying a third-party package.
 -   Our own JS lives under `static/js`; CSS lives under `static/styles`.
 -   JavaScript and CSS files are combined and minified in production. In
-    this case all you need to do is add the filename to PIPELINE\_CSS or
-    JS\_SPECS in `zproject/settings.py`. (If you plan to only use the
-    JS/CSS within the app proper, and not on the login page or other
-    standalone pages, put it in the 'app' category.)
+    this case all you need to do is add the filename to
+    PIPELINE['STYLESHEET'] or JS\_SPECS in `zproject/settings.py`. (If
+    you plan to only use the JS/CSS within the app proper, and not on
+    the login page or other standalone pages, put it in the 'app'
+    category.)
 
 If you want to test minified files in development, look for the
-`PIPELINE =` line in `zproject/settings.py` and set it to `True` -- or
-just set `DEBUG = False`.
+`PIPELINE_ENABLED =` line in `zproject/settings.py` and set it to `True`
+-- or just set `DEBUG = False`.
 
 Note that `static/html/{400,5xx}.html` will only render properly if
 minification is enabled, since they hardcode the path

docs/full-text-search.md

@@ -0,0 +1,39 @@
+# Full-text search
+
+Zulip supports full-text search, which can be combined arbitrarily
+with Zulip's full suite of narrowing operators.  By default, it only
+supports English text, but there is an experimental
+[PGroonga](http://pgroonga.github.io/) integration that provides
+full-text search for all languages.
+
+The user interface and feature set for Zulip's full-text search is
+documented in the "Search help" documentation section in the Zulip
+app's gear menu.
+
+## The default full-text search implementation
+
+Zulip's uses [PostgreSQL's built-in full-text search
+feature](http://www.postgresql.org/docs/current/static/textsearch.html),
+with a custom set of English stop words to improve the quality of the
+search results.
+
+We use a small extension,
+[tsearch_extras](https://github.com/zulip/tsearch_extras), for
+highlighting of the matching words.  There is [some discussion of
+removing this extension, at least as an
+option](https://github.com/zulip/zulip/issues/467), so that Zulip can
+be used with database-as-a-service platforms.
+
+In order to optimize the performance of delivering messages, the
+full-text search index is updated for newly sent messages in the
+background, after the message has been delivered.  This background
+updating is done by
+`puppet/zulip/files/postgresql/process_fts_updates`, which is usually
+deployed on the database server, but could be deployed on an
+application server instead.
+
+## An optional full-text search implementation
+
+See [the option PGroonga pull
+request](https://github.com/zulip/zulip/pull/700) for details on the
+status of the PGroonga integration.

docs/html_css.md

@@ -0,0 +1,72 @@
+# HTML and CSS
+
+## Zulip CSS organization
+
+The Zulip application's CSS can be found in the `static/styles/`
+directory.  Zulip uses [Bootstrap](http://getbootstrap.com/) as its
+main third-party CSS library.
+
+Zulip currently does not use any CSS preprocessors, and is organized
+into several files.  For most pages, the CSS is combined into a single
+CSS file by the [static asset pipeline](front-end-build-process.html),
+controlled by the `PIPELINE_CSS` code in `zproject/settings.py`.
+
+The CSS files are:
+
+* `portico.css` - Main CSS for logged-out pages
+* `pygments.css` - CSS for Python syntax highlighting
+* `activity.css` - CSS for the `activity` app
+* `fonts.css` - Fonts for text in the Zulip app
+* `static/styles/thirdparty-fonts.css` - Font Awesome (used for icons)
+
+The CSS for the Zulip web application UI is primarily here:
+
+* `settings.css` - CSS for the Zulip settings and administration pages
+* `zulip.css` - CSS for the rest of the Zulip logged-in app
+* `media.css` - CSS for media queries (particularly related to screen width)
+
+We are in the process of [splitting zulip.css into several more
+files](https://github.com/zulip/zulip/issues/731); help with that
+project is very welcome!
+
+## Editing Zulip CSS
+
+If you aren't experienced with doing web development and want to make
+CSS changes, we recommend reading the excellent [Chrome web inspector
+guide on editing HTML/CSS](https://developer.chrome.com/devtools/docs/dom-and-styles),
+especially the [section on
+CSS](https://developer.chrome.com/devtools/docs/dom-and-styles#styles)
+to learn about all the great tools that you can use to modify and test
+changes to CSS interactively in-browser (without even having the
+reload the page!).
+
+## CSS Style guidelines
+
+### Avoid duplicated code
+
+Without care, it's easy for a web application to end up with thousands
+of lines of duplicated CSS code, which can make it very difficult to
+understand the current styling or modify it.  We would very much like
+to avoid such a fate.  So please make an effort to reuse existing
+styling, clean up now-unused CSS, etc., to keep things maintainable.
+
+### Be consistent with existing similar UI
+
+Ideally, do this by reusing existing CSS declarations, so that any
+improvements we make to the styling can improve all similar UI
+elements.
+
+### Use clear, unique names for classes and object IDs
+
+This makes it much easier to read the code and use `git grep` to find
+where a particular class is used.
+
+## Validating CSS
+
+When changing any part of the Zulip CSS, it's important to check that
+the new CSS looks good at a wide range of screen widths, from very
+wide screen (e.g. 1920px) all the way down to narrow phone screens
+(e.g. 480px).
+
+For complex changes, it's definitely worth testing in a few different
+browsers to make sure things look the same.

docs/index.rst

@@ -23,18 +23,92 @@ This set of documents covers installation and contribution instructions.
 
 Contents:
 
+* :ref:`user-docs`
+* :ref:`prod-install-docs`
+* :ref:`dev-install-docs`
+* :ref:`tutorial-docs`
+* :ref:`code-docs`
+* :ref:`system-docs`
+
+.. _user-docs:
+
 .. toctree::
-   :maxdepth: 3
+   :maxdepth: 2
+   :caption: Overview
 
    readme-symlink
    architecture-overview
+   directory-structure
+   roadmap
+   changelog
+
+.. _prod-install-docs:
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Zulip in production
+
+   prod-requirements
+   prod-install
+   prod-troubleshooting
+   prod-customize
+   prod-maintain-secure-upgrade
+   prod-authentication-methods
+   prod-postgres
+
+.. _dev-install-docs:
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Installation for developers
+
+   dev-overview
+   dev-env-first-time-contributors
+   brief-install-vagrant-dev
+   install-ubuntu-without-vagrant-dev
+   install-generic-unix-dev
+   install-docker-dev
+   using-dev-environment
+
+.. _tutorial-docs:
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Developer tutorials
+
    integration-guide
    new-feature-tutorial
-   code-contribution-checklist
-   directory-structure
+   writing-views
+   life-of-a-request
+
+.. _code-docs:
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Code contribution guide
+
+   version-control
+   code-style
+   testing
+   mypy
+
+.. _system-docs:
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Subsystem documentation
+
+   settings
+   queuing
+   pointer
+   markdown
+   front-end-build-process
+   schema-migrations
+   html_css
+   full-text-search
    translating
-   changelog
-   roadmap
+   logging
+   README
 
 Indices and tables
 ==================

docs/install-docker-dev.md

@@ -0,0 +1,71 @@
+Using Docker (experimental)
+---------------------------
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+The docker instructions for development are experimental, so they may
+have bugs.  If you try them and run into any issues, please report
+them!
+
+You can also use Docker to run a Zulip development environment.
+First, you need to install Docker in your development machine
+following the [instructions][docker-install].  Some other interesting
+links for somebody new in Docker are:
+
+* [Get Started](https://docs.docker.com/engine/installation/linux/)
+* [Understand the architecture](https://docs.docker.com/engine/understanding-docker/)
+* [Docker run reference](https://docs.docker.com/engine/reference/run/)
+* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)
+
+[docker-install]: https://docs.docker.com/engine/installation/
+
+Then you should create the Docker image based on Ubuntu Linux, first
+go to the directory with the Zulip source code:
+
+```
+docker build -t user/zulipdev .
+```
+
+Now you're going to install Zulip dependencies in the image:
+
+```
+docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash
+$ /usr/bin/python /srv/zulip/tools/provision.py --docker
+docker ps -af ancestor=user/zulipdev
+docker commit -m "Zulip installed" <container id> user/zulipdev:v2
+```
+
+Finally you can run the docker server with:
+
+```
+docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \
+    /srv/zulip/tools/start-dockers
+```
+
+If you want to connect to the Docker instance to build a release
+tarball you can use:
+
+```
+docker ps
+docker exec -it <container id> /bin/bash
+$ source /home/zulip/.bash_profile
+$ <Your commands>
+$ exit
+```
+
+To stop the server use:
+```
+docker ps
+docker kill <container id>
+```
+
+If you want to run all the tests you need to start the servers first,
+you can do it with:
+
+```
+docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash
+$ tools/test-all-docker
+```
+
+You can modify the source code in your development machine and review
+the results in your browser.

docs/install-generic-unix-dev.md

@@ -0,0 +1,319 @@
+# Installing manually on UNIX
+
+* [Debian or Ubuntu systems](#on-debian-or-ubuntu-systems)
+* [Fedora 22 (experimental)](#on-fedora-22-experimental)
+* [CentOS 7 Core (experimental)](#on-centos-7-core-experimental)
+* [OpenBSD 5.8 (experimental)](#on-openbsd-5-8-experimental)
+* [Fedora/CentOS common steps](#common-to-fedora-centos-instructions)
+* [Steps for all systems](#all-systems)
+
+If you really want to install everything manually, the below instructions
+should work.
+
+Install the following non-Python dependencies:
+ * libffi-dev — needed for some Python extensions
+ * postgresql 9.1 or later — our database (client, server, headers)
+ * nodejs 0.10 (and npm)
+ * memcached (and headers)
+ * rabbitmq-server
+ * libldap2-dev
+ * python-dev
+ * redis-server — rate limiting
+ * tsearch-extras — better text search
+ * libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs
+
+### On Debian or Ubuntu systems:
+
+#### Using the official Ubuntu repositories and `tsearch-extras` deb package:
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
+    memcached rabbitmq-server libldap2-dev redis-server \
+    postgresql-server-dev-all libmemcached-dev python-dev \
+    hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \
+    puppet gettext postgresql
+
+# Next, install Zulip's tsearch-extras postgresql extension
+# If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras:
+cd zulip
+sudo apt-add-repository -yus ppa:tabbott/zulip
+# On 14.04
+sudo apt-get install postgresql-9.3-tsearch-extras
+# On 16.04
+sudo apt-get install postgresql-9.5-tsearch-extras
+
+
+# Otherwise, you can download a .deb directly
+# If on 12.04 or wheezy:
+wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
+sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
+
+# If on 14.04:
+https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-tsearch-extras_0.1.3_amd64.deb
+sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb
+
+# If on 15.04 or jessie:
+wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-extras_0.1_amd64.deb
+sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
+
+# If on 16.04 or stretch
+wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-tsearch-extras_0.2_amd64.deb
+sudo dpkg -i postgresql-9.5-tsearch-extras_0.2_amd64.deb
+```
+
+Alternatively, you can always build the package from [tsearch-extras
+git](https://github.com/zulip/tsearch_extras).
+
+Now continue with the [All Systems](#all-systems) instructions below.
+
+#### Using the [official Zulip PPA](https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+packages) (for 14.04 Trusty or 16.04 Xenial):
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+sudo add-apt-repository ppa:tabbott/zulip
+sudo apt-get update
+sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
+    memcached rabbitmq-server libldap2-dev redis-server \
+    postgresql-server-dev-all libmemcached-dev python-dev \
+    hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \
+    puppet gettext tsearch-extras
+```
+
+Now continue with the [All Systems](#all-systems) instructions below.
+
+### On Fedora 22 (experimental):
+
+These instructions are experimental and may have bugs; patches
+welcome!
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+sudo dnf install libffi-devel memcached rabbitmq-server \
+    openldap-devel python-devel redis postgresql-server \
+    postgresql-devel postgresql libmemcached-devel freetype-devel \
+    nodejs npm yuicompressor closure-compiler gettext
+```
+
+Now continue with the [Common to Fedora/CentOS](#common-to-fedora-centos-instructions) instructions below.
+
+### On CentOS 7 Core (experimental):
+
+These instructions are experimental and may have bugs; patches
+welcome!
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+# Add user zulip to the system (not necessary if you configured zulip
+# as the administrator user during the install process of CentOS 7).
+useradd zulip
+
+# Create a password for zulip user
+passwd zulip
+
+# Allow zulip to sudo
+visudo
+# Add this line after line `root    ALL=(ALL)       ALL`
+zulip   ALL=(ALL)       ALL
+
+# Switch to zulip user
+su zulip
+
+# Enable EPEL 7 repo so we can install rabbitmq-server, redis and
+# other dependencies
+sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+
+# Install dependencies
+sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \
+    python-devel redis postgresql-server postgresql-devel postgresql \
+    libmemcached-devel wget python-pip openssl-devel freetype-devel \
+    libjpeg-turbo-devel zlib-devel nodejs yuicompressor \
+    closure-compiler gettext
+
+# We need these packages to compile tsearch-extras
+sudo yum groupinstall "Development Tools"
+
+# clone Zulip's git repo and cd into it
+cd && git clone https://github.com/zulip/zulip && cd zulip/
+
+## NEEDS TESTING: The next few DB setup items may not be required at all.
+# Initialize the postgres db
+sudo postgresql-setup initdb
+
+# Edit the postgres settings:
+sudo vi /var/lib/pgsql/data/pg_hba.conf
+
+# Change these lines:
+host    all             all             127.0.0.1/32            ident
+host    all             all             ::1/128                 ident
+# to this:
+host    all             all             127.0.0.1/32            md5
+host    all             all             ::1/128                 md5
+```
+
+Now continue with the [Common to Fedora/CentOS](#common-to-fedora-centos-instructions) instructions below.
+
+### On OpenBSD 5.8 (experimental):
+
+These instructions are experimental and may have bugs; patches
+welcome!
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \
+    memcached node libmemcached py-Pillow py-cryptography py-cffi
+
+# Get tsearch_extras and build it (using a modified version which
+# aliases int4 on OpenBSD):
+git clone https://github.com/blablacio/tsearch_extras
+cd tsearch_extras
+gmake && sudo gmake install
+
+# Point environment to custom include locations and use newer GCC
+# (needed for Node modules):
+export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl"
+export CXX=eg++
+
+# Create tsearch_data directory:
+sudo mkdir /usr/local/share/postgresql/tsearch_data
+
+
+# Hack around missing dictionary files -- need to fix this to get the
+# proper dictionaries from what in debian is the hunspell-en-us
+# package.
+sudo touch /usr/local/share/postgresql/tsearch_data/english.stop
+sudo touch /usr/local/share/postgresql/tsearch_data/en_us.dict
+sudo touch /usr/local/share/postgresql/tsearch_data/en_us.affix
+```
+
+Finally continue with the [All Systems](#all-systems) instructions below.
+
+### Common to Fedora/CentOS instructions
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+```
+# Build and install postgres tsearch-extras module
+wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.tar.gz
+tar xvzf tsearch-extras_0.1.3.tar.gz
+cd ts2
+make
+sudo make install
+
+# Hack around missing dictionary files -- need to fix this to get the
+# proper dictionaries from what in debian is the hunspell-en-us
+# package.
+sudo touch /usr/share/pgsql/tsearch_data/english.stop
+sudo touch /usr/share/pgsql/tsearch_data/en_us.dict
+sudo touch /usr/share/pgsql/tsearch_data/en_us.affix
+
+# Edit the postgres settings:
+sudo vi /var/lib/pgsql/data/pg_hba.conf
+
+# Add this line before the first uncommented line to enable password
+# auth:
+host    all             all             127.0.0.1/32            md5
+
+# Start the services
+sudo systemctl start redis memcached rabbitmq-server postgresql
+
+# Enable automatic service startup after the system startup
+sudo systemctl enable redis rabbitmq-server memcached postgresql
+```
+
+Finally continue with the [All Systems](#all-systems) instructions below.
+
+### All Systems:
+
+Make sure you have followed the steps specific for your platform:
+
+* [Debian or Ubuntu systems](#on-debian-or-ubuntu-systems)
+* [Fedora 22 (experimental)](#on-fedora-22-experimental)
+* [CentOS 7 Core (experimental)](#on-centos-7-core-experimental)
+* [OpenBSD 5.8 (experimental)](#on-openbsd-5-8-experimental)
+* [Fedora/CentOS](#common-to-fedora-centos-instructions)
+
+For managing Zulip's python dependencies, we recommend using
+[virtualenvs](https://virtualenv.pypa.io/en/stable/).
+
+You must create two virtualenvs. One for Python 2 and one for Python 3.
+You must also install appropriate python packages in them.
+
+You should either install the virtualenvs in `/srv`, or put symlinks to
+them in `/srv`.  If you don't do that, some scripts might not work correctly.
+
+You can run `tools/setup/setup_venvs.py` to do this.  This script will create two
+virtualenvs - /srv/zulip-venv and /srv/zulip-py3-venv.
+
+If you want to do it manually, here are the steps:
+
+```
+virtualenv /srv/zulip-venv -p python2 # Create a python2 virtualenv
+source /srv/zulip-venv/bin/activate # Activate python2 virtualenv
+pip install --upgrade pip # upgrade pip itself because older versions have known issues
+pip install --no-deps -r requirements/py2_dev.txt # install python packages required for development
+
+virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv
+source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv
+pip install --upgrade pip # upgrade pip itself because older versions have known issues
+pip install --no-deps -r requirements/py3_dev.txt # install python packages required for development
+```
+
+Now run these commands:
+
+```
+./tools/setup/install-phantomjs
+./tools/install-mypy
+./tools/setup/download-zxcvbn
+./tools/setup/emoji_dump/build_emoji
+./scripts/setup/generate_secrets.py -d
+if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/tsearch_data/; else sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/9.*/tsearch_data/; fi
+./scripts/setup/configure-rabbitmq
+./tools/setup/postgres-init-dev-db
+./tools/do-destroy-rebuild-database
+./tools/setup/postgres-init-test-db
+./tools/do-destroy-rebuild-test-database
+./manage.py compilemessages
+npm install
+```
+
+If `npm install` fails, the issue may be that you need a newer version
+of `npm`.  You can use `npm install -g npm` to update your version of
+`npm` and try again.
+
+To start the development server:
+
+```
+./tools/run-dev.py
+```
+
+… and visit [http://localhost:9991/](http://localhost:9991/).
+
+#### Proxy setup for by-hand installation
+
+If you are building the development environment on a network where a
+proxy is required to access the Internet, you will need to set the
+proxy in the environment as follows:
+
+- On Ubuntu, set the proxy environment variables using:
+ ```
+ export https_proxy=http://proxy_host:port
+ export http_proxy=http://proxy_host:port
+ ```
+
+- And set the npm proxy and https-proxy using:
+ ```
+ npm config set proxy http://proxy_host:port
+ npm config set https-proxy http://proxy_host:port
+ ```

docs/install-ubuntu-without-vagrant-dev.md

@@ -0,0 +1,24 @@
+# Installing directly on Ubuntu
+
+Start by cloning this repository: `git clone
+https://github.com/zulip/zulip.git`
+
+If you'd like to install a Zulip development environment on a computer
+that's already running Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial, you
+can do that by just running:
+
+```
+# From a clone of zulip.git
+./tools/provision.py
+source /srv/zulip-venv/bin/activate
+./tools/run-dev.py  # starts the development server
+```
+
+Note that there is no supported uninstallation process without Vagrant
+(with Vagrant, you can just do `vagrant destroy` to clean up the
+development environment).
+
+Once you've done the above setup, you can pick up the [documentation
+on using the Zulip development
+environment](dev-env-first-time-contributors.html#step-4-developing),
+ignoring the parts about `vagrant` (since you're not using it).

docs/integration-guide.md

@@ -1,4 +1,4 @@
-# How to write a new integration
+# Writing a new integration
 
 Integrations are one of the most important parts of a group chat tool
 like Zulip, and we are committed to making integrating with Zulip and
@@ -6,6 +6,18 @@ getting you integration merged upstream so everyone else can benefit
 from it as easy as possible while maintaining the high quality of the
 Zulip integrations library.
 
+On this page you'll find:
+
+* An overvew of the different [types of integrations](#types-of-integrations)
+  possible with Zulip.
+* [General advice](#general-advice) for writing integrations.
+* Details about writing [webhook integrations](#webhook-integrations).
+* Details about writing [Python script and plugin
+  integrations](#python-script-and-plugin-integrations).
+* A guide to [documenting your integration](#documenting-your-integration).
+* A [detailed walkthrough](#hello-world-webhook-walkthrough) of a simple "Hello
+  World" integration.
+
 Contributions to this guide are very welcome, so if you run into any
 issues following these instructions or come up with any tips or tools
 that help writing integration, please email
@@ -17,25 +29,25 @@ to share your ideas!
 We have several different ways that we integrate with 3rd part
 products, ordered here by which types we prefer to write:
 
-1. Webhook integrations (examples: Freshdesk, GitHub), where the
-third-party service supports posting content to a particular URI on
-our site with data about the event.  For these, you usually just need
-to add a new handler in `zerver/views/webhooks.py` (plus
-test/document/etc.).  An example commit implementing a new webhook is:
-https://github.com/zulip/zulip/pull/324.
+1. **[Webhook integrations](#webhook-integrations)** (examples: Freshdesk,
+   GitHub), where the third-party service supports posting content to a
+   particular URI on our site with data about the event.  For these, you
+   usually just need to add a new handler in `zerver/views/webhooks.py` (plus
+   test/document/etc.).  An example commit implementing a new webhook is:
+   https://github.com/zulip/zulip/pull/324.
 
-2. Python script integrations (examples: SVN, Git), where we can get
-the service to call our integration (by shelling out or otherwise),
-passing in the required data.  Our preferred model for these is to
-ship these integrations in our API release tarballs (by writing the
-integration in `api/integrations`).
+2. **[Python script integrations](#python-script-and-plugin-integrations)**
+   (examples: SVN, Git), where we can get the service to call our integration
+   (by shelling out or otherwise), passing in the required data.  Our preferred
+   model for these is to ship these integrations in our API release tarballs
+   (by writing the integration in `api/integrations`).
 
-3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user
-needs to install a plugin into their existing software.  These are
-often more work, but for some products are the only way to integrate
-with the product at all.
+3. **[Plugin integrations](#python-script-and-plugin-integrations)** (examples:
+   Jenkins, Hubot, Trac) where the user needs to install a plugin into their
+   existing software.  These are often more work, but for some products are the
+   only way to integrate with the product at all.
 
-## General advice for writing integrations
+## General advice
 
 * Consider using our Zulip markup to make the output from your
   integration especially attractive or useful (e.g.  emoji, markdown
@@ -62,7 +74,14 @@ with the product at all.
   don't have an API or webhook we can use -- sometimes the right API
   is just not properly documented.
 
-## Writing Webhook integrations
+* A helpful tool for testing your integration is
+  [UltraHook](http://www.ultrahook.com/), which allows you to receive webhook
+  calls via your local Zulip dev environment. This enables you to do end-to-end
+  testing with live data from the service you're integrating and can help you
+  spot why something isn't working or if the service is using custom HTTP
+  headers.
+
+## Webhook integrations
 
 New Zulip webhook integrations can take just a few hours to write,
 including tests and documentation, if you use the right process.
@@ -78,8 +97,9 @@ Here's how we recommend doing it:
   templating off a short one (like `stash.py` or `zendesk.py`), since
   the longer ones usually just have more complex parsing which can
   obscure what's common to all webhook integrations.  In addition to
-  writing the integration itself, you'll need to add an entry in
-  `zproject/urls.py` for your webhook; search for `webhook` in that
+  writing the integration itself, you'll need to create `Integration`
+  object and add it to `WEBHOOK_INTEGRATIONS` in
+  `zerver/lib/integrations.py'; search for `webhook` in that
   file to find the existing ones (and please add yours in the
   alphabetically correct place).
 
@@ -93,9 +113,8 @@ Here's how we recommend doing it:
   test-backend zerver.tests.test_hooks.PagerDutyHookTests
   ```
 
-  See
-  https://github.com/zulip/zulip/blob/master/README.dev.md#running-the-test-suite
-  for more details on the Zulip test runner.
+  See [this guide](testing.html) for more details on the Zulip test
+  runner.
 
 * Once you've gotten your webhook working and passing a test, capture
   payloads for the other common types of posts the service's webhook
@@ -105,9 +124,44 @@ Here's how we recommend doing it:
   can't run without Internet access and some sort of credentials for
   the service.
 
-* Finally, write documentation for the integration (see below)!
+* Finally, write documentation for the integration; there's a
+  [detailed guide](#documenting-your-integration) below.
 
-## Writing Python script and plugin integrations integrations
+See the [Hello World webhook Walkthrough](#hello-world-webhook-walkthrough)
+below for a detailed look at how to write a simple webhook.
+
+### Files that need to be created
+
+Select a name for your webhook and use it consistently. The examples below are
+for a webhook named 'MyWebHook'.
+
+* `static/images/integrations/logos/mywebhook.png`: An image to represent
+  your integration in the user interface. Generally this Should be the logo of the
+  platform/server/product you are integrating. See [Documenting your
+  integration](#documenting-your-integration) for details.
+* `static/images/integrations/mywebbook/001.png`: A screen capture of your
+  integration for use in the user interface. You can add as many images as needed
+  to effectively document your webhook integration. See [Documenting your
+  integration](#documenting-your-integration) for details.
+* `zerver/fixtures/mywebhook/mywebhook_messagetype.json`: Sample json payload data
+  used by tests. Add one fixture file per type of message supported by your
+  integration. See [Testing and writing tests](testing.html) for details.
+* `zerver/views/webhooks/mywebhook.py`: Includes the main webhook integration
+  function including any needed helper functions.
+
+### Files that need to be updated
+
+* `templates/zerver/integrations.html`: Edit to add end-user documentation. See
+  [Documenting your integration](#documenting-your-integration) for details.
+* `zerver/test_hooks.py`: Edit to include tests for your webbook. See [Testing
+  and writing tests](testing.html) for details.
+* `zerver/lib/integrations.py`: Add your integration to
+`WEBHOOK_INTEGRATIONS` to register it.  This will automatically
+register a url for the webhook of the form `api/v1/external/mywebhook`
+and associate with the function called `api_mywebhook_webhook` in
+`zerver/views/webhooks/mywebhook.py`.
+
+## Python script and plugin integrations
 
 For plugin integrations, usually you will need to consult the
 documentation for the third party software in order to learn how to
@@ -144,9 +198,12 @@ Every Zulip integration must be documented in
 `templates/zerver/integrations.html`.  Usually, this involves a few
 steps:
 
-* Add an `integration-lozenge` class block in the alphabetically
-  correct place in the main integration list, using the logo for the
-  integrated software.
+* Make sure you've added your integration to
+  `zerver/lib/integrations.py`; this results in your integration
+  appearing on the `/integrations` page.  You'll need to add a logo
+  image for your integration under the
+  `static/images/integrations/logos/<name>.png`, where `<name>` is the
+  name of the integration, all in lower case.
 
 * Add an `integration-instructions` class block also in the
   alphabetically correct place, explaining all the steps required to
@@ -180,3 +237,350 @@ documentation will provide the correct URL for whatever server it is
 deployed on.  If special configuration is required to set the SITE
 variable, you should document that too, inside an `{% if
 api_site_required %}` check.
+
+## `Hello World` webhook Walkthrough
+
+Below explains each part of a simple webhook integration, called **Hello
+World**. This webhook sends a "hello" message to the `test` stream and includes
+a link to the Wikipedia article of the day, which it formats from json data it
+receives in the http request.
+
+Use this walkthrough to learn how to write your first webhook
+integration.
+
+### Step 0: Create fixtures
+
+The first step in creating a webhook is to examine the data that the
+service you want to integrate will be sending to Zulip.
+
+You can use [requestb.in](http://requestb.in/) or a similar tool to capture
+webook payload(s) from the service you are integrating. Examining this
+data allows you to do two things:
+
+1. Determine how you will need to structure your webook code, including what
+   message types your integration should support and how; and,
+2. Create fixtures for your webook tests.
+
+Fixtures enable the testing of webhook integration code without the need to
+actually contact the service being integrated.
+
+Because `Hello World` is a very simple webhook that does one thing, it requires
+only one fixture, `zerver/fixtures/helloworld/helloworld_hello.json`:
+
+```
+{
+  "featured_title":"Marilyn Monroe",
+  "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe",
+}
+```
+
+When writing your own webhook integration, you'll want to write a test function
+for each distinct message condition your webhook supports. You'll also need a
+corresponding fixture for each of these tests. See [Step 3: Create
+tests](#step-3-create-tests) or [Testing](testing.html) for further details.
+
+### Step 1: Create main webhook code
+
+The majority of the code for your webhook integration will be in a single
+python file in `zerver/views/webhooks/`. The name of this file should be the
+name of your webhook, all lower-case, with file extension `.py`:
+`mywebhook.py`.
+
+The Hello World integration is in `zerver/views/webhooks/helloworld.py`:
+
+```
+from __future__ import absolute_import
+from django.utils.translation import ugettext as _
+from zerver.lib.actions import check_send_message
+from zerver.lib.response import json_success, json_error
+from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view
+from zerver.lib.validator import check_dict, check_string
+
+from zerver.models import Client, UserProfile
+
+from django.http import HttpRequest, HttpResponse
+from six import text_type
+from typing import Dict, Any, Iterable, Optional
+
+@api_key_only_webhook_view('HelloWorld')
+@has_request_variables
+def api_helloworld_webhook(request, user_profile, client,
+                           payload=REQ(argument_type='body'),
+                           stream=REQ(default='test'),
+                           topic=REQ(default='Hello World')):
+    # type: (HttpRequest, UserProfile, Client, Dict[str, Iterable[Dict[str, Any]]], text_type, Optional[text_type]) -> HttpResponse
+
+  # construct the body of the message
+  body = 'Hello! I am happy to be here! :smile:'
+
+  # try to add the Wikipedia article of the day
+  # return appropriate error if not successful
+  try:
+      body_template = '\nThe Wikipedia featured article for today is **[{featured_title}]({featured_url})**'
+      body += body_template.format(**payload)
+  except KeyError as e:
+      return json_error(_("Missing key {} in JSON").format(str(e)))
+
+  # send the message
+  check_send_message(user_profile, client, 'stream', [stream], topic, body)
+
+  # return json result
+  return json_success()
+
+```
+
+The above code imports the required functions and defines the main webhook
+function `api_helloworld_webook`, decorating it with `api_key_only_webhook_view` and
+`has_request_variables`.
+
+You must pass the name of your webhook to the `api_key_only_webhook_view`
+decorator. Here we have used `HelloWorld`. To be consistent with Zulip code
+style, use the name of the product you are integrating in camel case, spelled
+as the product spells its own name (except always first letter upper-case).
+
+You should name your webhook function as such `api_webhookname_webhook` where
+`webhookname` is the name of your webhook and is always lower-case.
+
+At minimum, the webhook function must accept `request` (Django
+[HttpRequest](https://docs.djangoproject.com/en/1.8/ref/request-response/#django.http.HttpRequest)
+object), `user_profile` (Zulip's user object), and `client` (Zulip's analogue
+of UserAgent). You may also want to define additional parameters using the
+`REQ` object.
+
+In the example above, we have defined `payload` which is populated from the
+body of the http request, `stream` with a default of `test` (available by
+default in Zulip dev environment), and `topic` with a default of `Hello World`.
+
+The line that begins `# type` is a mypy type annotation. See [this
+page](mypy.html) for details about how to properly annotate your webhook
+functions.
+
+In the body of the function we define the body of the message as `Hello! I am
+happy to be here! :smile:`. The `:smile:` indicates an emoji. Then we append a
+link to the Wikipedia article of the day as provided by the json payload. If
+the json payload does not include data for `featured_title` and `featured_url`
+we catch a `KeyError` and use `json_error` to return the appropriate
+information: a 400 http status code with relevant details.
+
+Then we send a public (stream) message with `check_send_message` which will
+validate the message and then send it.
+
+Finally, we return a 200 http status with a JSON format success message via
+`json_success()`.
+
+### Step 2: Create an api endpoint for the webhook
+
+In order for a webhook to be externally available, it must be mapped to a url.
+This is done in `zerver/lib/integrations.py`.
+
+Look for the lines beginning with:
+
+```
+WEBHOOK_INTEGRATIONS = [
+```
+
+And you'll find the entry for Hello World:
+
+```
+  WebhookIntegration('helloworld', display_name='Hello World'),
+```
+
+This tells the Zulip api to call the `api_helloworld_webhook` function in
+`zerver/views/webhooks/helloworld.py` when it receives a request at
+`/api/v1/external/helloworld`.
+
+This line also tells Zulip to generate an entry for Hello World on the Zulip
+integrations page using `static/images/integrations/logos/helloworld.png` as its
+icon.
+
+At this point, if you're following along and/or writing your own Hello World
+webhook, you have written enough code to test your integration.
+
+You can do so by using Zulip itself or curl on the command line.
+
+Using `manage.py` from within Zulip Dev environment:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./manage.py send_webhook_fixture_message \
+> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \
+> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'
+```
+After which you should see something similar to:
+
+```
+2016-07-07 15:06:59,187 INFO     127.0.0.1       POST    200 143ms (mem: 6ms/13) (md: 43ms/1) (db: 20ms/9q) (+start: 147ms) /api/v1/external/helloworld (helloworld-bot@zulip.com via ZulipHelloWorldWebhook)
+```
+
+Using curl:
+
+```
+curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>
+```
+
+After which you should see:
+```
+{"msg":"","result":"success"}
+```
+
+Using either method will create a message in Zulip:
+
+![Image of Hello World webhook message](images/helloworld-webhook.png)
+
+### Step 3: Create tests
+
+Every webhook integraton should have a corresponding test class in
+`zerver/tests/test_hooks.py`.
+
+You should name the class `<WebhookName>HookTests` and this class should accept
+`WebhookTestCase`. For our HelloWorld webhook, we name the test class
+`HelloWorldHookTests`:
+
+```
+class HelloWorldHookTests(WebhookTestCase):
+    STREAM_NAME = 'test'
+    URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"
+    FIXTURE_DIR_NAME = 'helloworld'
+
+    # Note: Include a test function per each distinct message condition your integration supports
+    def test_hello_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile: \nThe Wikipedia featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/Marilyn_Monroe)**";
+
+        # use fixture named helloworld_hello
+        self.send_and_test_stream_message('hello', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+
+    def get_body(self, fixture_name):
+        # type: (text_type) -> text_type
+        return self.fixture_data("helloworld", fixture_name, file_type="json")
+
+```
+
+When writing tests for your webook, you'll want to include one test function
+(and corresponding fixture) per each distinct message condition that your
+integration supports.
+
+If, for example, we added support for sending a goodbye message to our `Hello
+World` webook, we would add another test function to `HelloWorldHookTests`
+class called something like `test_goodbye_message`:
+
+```
+    def test_goodbye_message(self):
+        # type: () -> None
+        expected_subject = u"Hello World";
+        expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
+
+        # use fixture named helloworld_goodbye
+        self.send_and_test_stream_message('goodbye', expected_subject, expected_message,
+                                          content_type="application/x-www-form-urlencoded")
+```
+
+As well as a new fixture `helloworld_goodbye.json` in
+`zerver/fixtures/helloworld/`:
+
+```
+{
+  "featured_title":"Goodbye",
+  "featured_url":"https://en.wikipedia.org/wiki/Goodbye",
+}
+```
+
+Once you have written some tests, you can run just these new tests from within
+the Zulip dev environment with this command:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./tools/test-backend zerver.tests.test_hooks.HelloWorldHookTests
+```
+
+(Note: You must run the tests from `/srv/zulip` directory.)
+
+You will see some script output and if all the tests have passed, you will see:
+
+```
+Running zerver.tests.test_hooks.HelloWorldHookTests.test_hello_message
+DONE!
+```
+
+### Step 4: Create documentation
+
+Next, we add end-user documentation for our webhook integration to
+`templates/zerver/integrations.html`.
+
+There are two parts to the end-user documentation on this page.
+
+The first is a `div` with class `integration-lozenge` for each integration.
+This div shows the logo of your webhook, its name, and a link to its
+installation and usage instructions.
+
+Because there is an entry for the Hello World webhook in WEBHOOK_INTEGRATIONS
+in `zerver/lib/integratins.py`, this div will be generated automatically.
+
+The second part is a `div` with the webhook's usage instructions:
+
+```
+<div id="helloworld" class="integration-instructions">
+
+    <p>Learn how Zulip integrations work with this simple Hello World example!</p>
+
+    <p>The Hello World webhook will use the <code>test<code> stream, which is
+    created by default in the Zulip dev environment. If you are running
+    Zulip in production, you should make sure this stream exists.</p>
+
+    <p>Next, on your <a href="/#settings" target="_blank">Zulip
+    settings page</a>, create a Hello World bot.  Construct the URL for
+    the Hello World bot using the API key and stream name:
+      <code>{{ external_api_uri }}/v1/external/helloworld?api_key=abcdefgh&amp;stream=test</code>
+    </p>
+
+    <p>To trigger a notication using this webhook, use `send_webhook_fixture_message` from the Zulip command line:</p>
+    <div class="codehilite">
+      <pre>(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
+./manage.py send_webhook_fixture_message \
+> --fixture=zerver/fixtures/helloworld/helloworld_hello.json \
+> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'</pre>
+    </div>
+
+    <p>Or, use curl:</p>
+    <div class="codehilite">
+      <pre>curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key></pre>
+    </div>
+
+    <p><b>Congratulations! You're done!</b><br /> Your messages may look like:</p>
+
+    <img class="screenshot" src="/static/images/integrations/helloworld/001.png" />
+</div>
+```
+
+These documentation blocks should fall alphabetically. For the
+`integration-lozenge` div this happens automatically when the html is
+generated. For the `integration-instructions` div, we have added the div
+between the blocks for Github and Hubot, respectively.
+
+See [Documenting your integration](#documenting-your-integration) for further
+details, including how to easily create the message screenshot.
+
+### Step 5: Preparing a pull request to zulip/zulip
+
+When you have finished your webhook integration and are ready for it to be
+available in the Zulip product, follow these steps to prepare your pull
+request:
+
+1. Run tests including linters and ensure you have addressed any issues they
+   report. See [Testing](testing.html) for details.
+2. Read through [Code styles and conventions](code-style.html) and take a look
+   through your code to double-check that you've followed Zulip's guidelines.
+3. Take a look at your git history to ensure your commits have been clear and
+   logical (see [Version Control](version-control.html) for tips). If not,
+   consider revising them with `git rebase --interactive`. For most webhooks,
+   you'll want to squash your changes into a single commit and include a good,
+   clear commit message.
+4. Push code to your fork.
+5. Submit a pull request to zulip/zulip.
+
+If you would like feedback on your integration as you go, feel free to submit
+pull requests as you go, prefixing them with `[WIP]`.
+
+

docs/life-of-a-request.md

@@ -0,0 +1,190 @@
+# Life of a Request
+
+It can sometimes be confusing to figure out how to write a new feature,
+or debug an existing one. Let us try to follow a request through the
+Zulip codebase, and dive deep into how each part works.
+
+We will use as our example the creation of users through the API, but we
+will also highlight how alternative requests are handled.
+
+## A request is sent to the server, and handled by [Nginx](http://nginx.org/en/docs/)
+
+When Zulip is deployed in production, all requests go through nginx.
+For the most part we don't need to know how this works, except for when
+it isn't working. Nginx does the first level of routing--deciding which
+application will serve the request (or deciding to serve the request
+itself for static content).
+
+In development, `tools/run-dev.py` fills the role of nginx. Static files
+are in your git checkout under `static`, and are served unminified.
+
+## Nginx secures traffic with [SSL](https://zulip.readthedocs.io/en/latest/prod-install.html)
+
+If you visit your Zulip server in your browser and discover that your
+traffic isn't being properly encrypted, an [nginx misconfiguration](https://github.com/zulip/zulip/blob/master/puppet/zulip/files/nginx/sites-available/zulip-enterprise) is the
+likely culprit.
+
+## Static files are [served directly](https://github.com/zulip/zulip/blob/master/puppet/zulip/files/nginx/zulip-include-frontend/app) by Nginx
+
+Static files include JavaScript, css, static assets (like emoji, avatars),
+and user uploads (if stored locally and not on S3).
+
+```
+location /static/ {
+    alias /home/zulip/prod-static/;
+    error_page 404 /static/html/404.html;
+}
+```
+
+## Nginx routes other requests [between tornado and django](http://zulip.readthedocs.io/en/latest/architecture-overview.html?highlight=tornado#tornado-and-django)
+
+All our connected clients hold open long-polling connections so that
+they can recieve events (messages, presence notifications, and so on) in
+real-time. Events are served by Zulip's `tornado` application.
+
+Nearly every other kind of request is served by the `zerver` Django
+application.
+
+[Here is the relevant nginx routing configuration.](https://github.com/zulip/zulip/blob/master/puppet/zulip/files/nginx/zulip-include-frontend/app)
+
+## Django routes the request to a view in urls.py files
+
+There are various [urls.py](https://docs.djangoproject.com/en/1.8/topics/http/urls/) files throughout the server codebase, which are
+covered in more detail in [the directory structure doc](http://zulip.readthedocs.io/en/latest/directory-structure.html).
+
+The main Zulip Django app is `zerver`. The routes are found in
+```
+zproject/urls.py
+zproject/legacy_urls.py
+```
+
+There are HTML-serving, REST API, legacy, and webhook url patterns. We
+will look at how each of these types of requests are handled, and focus
+on how the REST API handles our user creation example.
+
+## Views serving HTML are internationalized by server path
+
+If we look in [zproject/urls.py](https://github.com/zulip/zulip/blob/master/zproject/urls.py), we can see something called
+`i18n_urls`. These urls show up in the address bar of the browser, and
+serve HTML.
+
+For example, the `/hello` page (preview [here](https://zulip.com/hello/))
+gets translated in Chinese at `zh-cn/hello/` (preview [here](https://zulip.com/zh-cn/hello/)).
+
+Note the `zh-cn` prefix--that url pattern gets added by `i18n_patterns`.
+
+## API endpoints use [REST](http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm)
+
+Our example is a REST API endpoint. It's a PUT to `/users`.
+
+With the exception of Webhooks (which we do not usually control the
+format of), legacy endpoints, and logged-out endpoints, Zulip uses REST
+for its API. This means that we use:
+
+* POST for creating something new where we don't have a unique ID. Also used as a catch-all if no other verb is appropriate.
+* PUT for creating something for which we have a unique ID.
+* DELETE for deleting something
+* PATCH for updating or editing attributes of something.
+* GET to get something (read-only)
+* HEAD to check the existence of something to GET, without getting it;
+  useful to check a link without downloading a potentially large link
+* OPTIONS (handled automatically, see more below)
+
+Of these, PUT, DELETE, HEAD, OPTIONS, and GET are *idempotent*, which
+means that we can send the request multiple times and get the same
+state on the server. You might get a different response after the first
+request, as we like to give our clients an error so they know that no
+new change was made by the extra requests.
+
+POST is not idempotent--if I send a message multiple times, Zulip will
+show my message multiple times. PATCH is special--it can be
+idempotent, and we like to write API endpoints in an idempotent fashion,
+as much as possible.
+
+This [cookbook](http://restcookbook.com/) and [tutorial](http://www.restapitutorial.com/) can be helpful if you are new to REST web applications.
+
+### PUT is only for creating new things
+
+If you're used to using PUT to update or modify resources, you might
+find our convention a little strange.
+
+We use PUT to create resources with unique identifiers, POST to create
+resources without unique identifiers (like sending a message with the
+same content multiple times), and PATCH to modify resources.
+
+In our example, `create_user_backend` uses PUT, because there's a unique
+identifier, the user's email.
+
+### OPTIONS
+
+The OPTIONS method will yield the allowed methods.
+
+This request:
+`OPTIONS https://zulip.tabbott.net/api/v1/users`
+yields a response with this HTTP header:
+`Allow: PUT, GET`
+
+We can see this reflected in [zproject/urls.py](https://github.com/zulip/zulip/blob/master/zproject/urls.py):
+
+    url(r'^users$', 'zerver.lib.rest.rest_dispatch',
+        {'GET': 'zerver.views.users.get_members_backend',
+         'PUT': 'zerver.views.users.create_user_backend'}),
+
+In this way, the API is partially self-documenting.
+
+### Legacy endpoints are used by the web client
+
+The endpoints from the legacy JSON API are written without REST in
+mind. They are used extensively by the web client, and use POST.
+
+You can see them in [zproject/legacy_urls.py](https://github.com/zulip/zulip/blob/master/zproject/legacy_urls.py).
+
+### Webhook integrations may not be RESTful
+
+Zulip endpoints that are called by other services for integrations have
+to conform to the service's request format. They are likely to use
+only POST.
+
+## Django calls rest_dispatch for REST endpoints, and authenticates
+
+For requests that correspond to a REST url pattern, Zulip configures its
+url patterns (see [zerver/lib/rest.py](https://github.com/zulip/zulip/blob/master/zerver/lib/rest.py)) so that the action called is
+`rest_dispatch`. This method will authenticate the user, either through
+a session token from a cookie, or from an `email:api-key` string given
+via HTTP Basic Auth for API clients.
+
+It will then look up what HTTP verb was used (GET, POST, etc) to make
+the request, and then figure out which view to show from that.
+
+In our example,
+```
+{'GET': 'zerver.views.users.get_members_backend',
+ 'PUT': 'zerver.views.users.create_user_backend'}
+```
+is supplied as an argument to `rest_dispatch`, along with the [HTTPRequest](https://docs.djangoproject.com/en/1.8/ref/request-response/).
+The request has the HTTP verb `PUT`, which `rest_dispatch` can use to
+find the correct view to show: `zerver.views.users.create_user_backend`.
+
+## The view will authorize the user, extract request variables, and validate them
+
+This is covered in good detail in the [writing views doc](https://zulip.readthedocs.io/en/latest/writing-views.html)
+
+## Results are given as JSON
+
+Our API works on JSON requests and responses. Every API endpoint should
+return `json_error` in the case of an error, which gives a JSON string:
+
+`{'result': 'error', 'msg': <some error message>}`
+
+in a [HTTP Response](https://docs.djangoproject.com/en/1.8/ref/request-response/) with a content type of 'application/json'.
+
+To pass back data from the server to the calling client, in the event of
+a successfully handled request, we use `json_success(data=<some python object which can be converted to a JSON string>`.
+
+This will result in a JSON string:
+
+`{'result': 'success', 'msg': '', 'data'='{'var_name1': 'var_value1', 'var_name2': 'var_value2'...}`
+
+with a HTTP 200 status and a content type of 'application/json'.
+
+That's it!

docs/linters.md

@@ -0,0 +1,239 @@
+# Linters
+
+## Overview
+
+Zulip does extensive linting of much of its source code, including
+Python/JavaScript files, HTML templates (Django/handlebars), CSS files,
+JSON fixtures, Markdown documents, puppet manifests, and shell scripts.
+
+For some files we simply check for small things like trailing whitespace,
+but for other files, we are quite thorough about checking semantic
+correctness.
+
+Obviously, a large reason for linting code is to enforce the [Zulip
+coding standards](code-style.html).  But we also use the linters to
+prevent common coding errors.
+
+We borrow some open source tools for much of our linting, and the links
+below will direct you to the official documentation for these projects.
+
+- [jslint](https://github.com/douglascrockford/JSLint)
+- [mypy](http://mypy-lang.org/)
+- [puppet](https://puppet.com/) (puppet provides its own mechanism for validating manifests)
+- [pyflakes](https://pypi.python.org/pypi/pyflakes)
+
+Zulip also uses some home-grown code to perform tasks like validating
+indentation in template files, enforcing coding standards that are unique
+to Zulip, allowing certain errors from third party linters to pass through,
+and exempting legacy files from lint checks.
+
+## Running the linters
+
+If you run `./tools/test-all`, it will automatically run the linters (with
+one small exception: it does not run mypy against scripts).
+
+You can also run them individually:
+
+    ./tools/lint-all
+    ./tools/run-mypy
+    ./tools/run-mypy --scripts-only
+
+Finally, you can rely on our Travis CI setup to run linters for you, but
+it is good practice to run lint checks locally.
+
+Our linting tools generally support the ability to lint files
+individually--with some caveats--and those options will be described
+later in this document.
+
+We may eventually bundle `run-mypy` into `lint-all`, but mypy is pretty
+resource intensive compared to the rest of the linters, because it does
+static code analysis.  So we keep mypy separate to allow folks to quickly run
+the other lint checks.
+
+## General considerations
+
+Once you have read the [Zulip coding guidelines](code-style.html), you can
+be pretty confident that 99% of the code that you write will pass through
+the linters fine, as long as you are thorough about keeping your code clean.
+And, of course, for minor oversights, `lint-all` is your friend, not your foe.
+
+Occasionally, our linters will complain about things that are more of
+an artifact of the linter limitations than any actual problem with your
+code.  There is usually a mechanism where you can bypass the linter in
+extreme cases, but often it can be a simple matter of writing your code
+in a slightly different style to appease the linter.  If you have
+problems getting something to lint, you can submit an unfinished PR
+and ask the reviewer to help you work through the lint problem, or you
+can find other people in the [Zulip Community](readme-symlink.html#community)
+to help you.
+
+Also, bear in mind that 100% of the lint code is open source, so if you
+find limitations in either the Zulip home-grown stuff or our third party
+tools, feedback will be highly appreciated.
+
+Finally, one way to clean up your code is to thoroughly exercise it
+with tests.  The [Zulip test documentation](testing.html)
+describes our test system in detail.
+
+## Lint checks
+
+Most of our lint checks get performed by `./tools/lint-all`.  These include the
+following checks:
+
+- Check Python code with pyflakes.
+- Check JavaScript code with jslint.
+- Check Python code for custom Zulip rules.
+- Check non-Python code for custom Zulip rules.
+- Check puppet manifests with the puppet validator.
+- Check HTML templates for matching tags and indentations.
+- Check CSS for parsability.
+- Check JavaScript code for addClass calls.
+
+The remaining lint checks occur in `./tools/run-mypy`.  It is probably somewhat
+of an understatement to call "mypy" a "linter," as it performs static
+code analysis of Python type annotations throughout our Python codebase.
+
+Our [documentation on using mypy](mypy.html) covers mypy in more detail.
+
+The rest of this document pertains to the checks that occur in `./tools/lint-all`.
+
+## lint-all
+
+Zulip has a script called `lint-all` that lives in our "tools" directory.
+It is the workhorse of our linting system, although in some cases it
+dispatches the heavy lifting to other components such as pyflakes,
+jslint, and other home grown tools.
+
+You can find the source code [here](https://github.com/zulip/zulip/blob/master/tools/lint-all).
+
+In order for our entire lint suite to run in a timely fashion, the `lint-all`
+script performs several lint checks in parallel by forking out subprocesses.  This mechanism
+is still evolving, but you can look at the method `run_parallel` to get the
+gist of how it works.
+
+### Special options
+
+You can use the `-h` option for `lint-all` to see its usage.  One particular
+flag to take note of is the `--modified` flag, which enables you to only run
+lint checks against files that are modified in your git repo.  Most of the
+"sub-linters" respect this flag, but some will continue to process all the files.
+Generally, a good workflow is to run with `--modified` when you are iterating on
+the code, and then run without that option right before commiting new code.
+
+If you need to troubleshoot the linters, there is a `--verbose` option that
+can give you clues about which linters may be running slow, for example.
+
+### Lint checks
+
+The next part of this document describes the lint checks that we apply to
+various file types.
+
+#### Generic source code checks
+
+We check almost our entire codebase for trailing whitespace.  Also, we
+disallow tab (\t) characters in all but two files.
+
+We also have custom regex-based checks that apply to specific file types.
+For relatively minor files like Markdown files and JSON fixtures, this
+is the extent of our checking.
+
+Finally, we're checking line length in Python code (and hope to extend
+this to other parts of the codebase soon).  You can use
+`#ignorelinelength` for special cases where a very long line makes
+sense (e.g. a link in a comment to an extremely long URL).
+
+#### Python code
+
+The bulk of our Python linting gets outsourced to the "pyflakes" tool.  We
+call "pyflakes" in a fairly vanilla fashion, and then we post-process its
+output to exclude certain types of errors that Zulip is comfortable
+ignoring.  (One notable class of error that Zulip currently tolerates is
+unused imports--because of the way mypy type annotations work in Python 2,
+it would be inconvenient to enforce this too strictly.)
+
+Zulip also has custom regex-based rules that it applies to Python code.
+Look for `python_rules` in the source code for `lint-all`.  Note that we
+provide a mechanism to excude certain lines of codes from these checks.
+Often, it is simply the case that our regex approach is too crude to
+correctly exonerate certain valid constructs.  In other cases, the code
+that we exempt may be deemed not worthwhile to fix.
+
+#### JavaScript code
+
+We check our JavaScript code in a few different ways:
+- We run jslint.
+- We perform custom Zulip regex checks on the code.
+- We verify that all addClass calls, with a few exceptions, explicitly
+  contain a CSS class.
+
+The last check happens via a call to `./tools/find-add-class`.  This
+particular check is a work in progress, as we are trying to evolve a
+more rigorous system for weeding out legacy CSS styles, and the ability
+to quickly introspect our JS code for `addClass` calls is part of our
+vision.
+
+#### Puppet manifests
+
+We use Puppet as our tool to manage configuration files, using
+puppet "manifests."  To lint puppet manifests, we use the "parser validate"
+option of puppet.
+
+#### HTML Templates
+
+Zulip uses two HTML templating systems:
+
+- [Django templates](https://docs.djangoproject.com/en/1.10/topics/templates/)
+- [handlebars](http://handlebarsjs.com/)
+
+Zulip has a home grown tool that validates both types of templates for
+correct indentation and matching tags.  You can find the code here:
+
+- driver: [check-templates](https://github.com/zulip/zulip/blob/master/tools/check-templates)
+- engine: [lib/template_parser.py](https://github.com/zulip/zulip/blob/master/tools/lib/template_parser.py)
+
+We exempt some legacy files from indentation checks, but we are hoping to
+clean those files up eventually.
+
+#### CSS
+
+Zulip does not currently lint its CSS for any kind of semantic correctness,
+but that is definitely a goal moving forward.
+
+We do ensure that our home-grown CSS parser can at least parse the CSS code.
+This is a slightly more strict check than checking that the CSS is
+compliant to the official spec, as our parser will choke on unusual
+constructs that we probably want to avoid in our code, anyway.  (When
+the parser chokes, the lint check will fail.)
+
+You can find the code here:
+
+- driver: [check-css](https://github.com/zulip/zulip/blob/master/tools/check-css)
+- engine: [lib/css_parser.py](https://github.com/zulip/zulip/blob/master/tools/lib/css_parser.py)
+
+#### Markdown, shell scripts, JSON fixtures
+
+We mostly validate miscellaneous source files like `.sh`, `.json`, and `.md` files for
+whitespace issues.
+
+## Philosophy
+
+If you want to help improve Zulip's system for linting, here are some
+considerations.
+
+#### Speed
+
+We want our linters to be fast enough that most developers
+will feel comfortable running them in a pre-commit hook, so we run
+our linters in parallel and support incremental checks.
+
+#### Accuracy
+
+We try to catch as many common mistakes as possible, either via a
+linter or an automated test.
+
+#### Completeness
+
+Our goal is to have most common style issues by caught by the linters, so new
+contributors to the codebase can efficiently fix produce code with correct
+style without needing to go back-and-forth with a reviewer.
+

docs/markdown.md

@@ -1,4 +1,4 @@
-# Zulip's Markdown implementation
+# Markdown implementation
 
 Zulip has a special flavor of Markdown, currently called 'bugdown'
 after Zulip's original name of "humbug". End users are using Bugdown

docs/migration-renumbering.md

@@ -0,0 +1,302 @@
+# Renumbering Migrations
+When you rebase your development branch off of
+a newer copy of master, and your branch contains
+new database migrations, you can occasionally get
+thrown off by conflicting migrations that are
+new in master.
+
+To help you understand how to deal with these
+conflicts, I am about narrate an exercise
+where I bring my development branch called
+showell-topic up to date with master.
+
+In this example,
+there is a migration on master called
+`0024_realm_allow_message_editing.py`, and
+that was the most recent migration at the
+time I started working on my branch.  In
+my branch I created migrations 0025 and 0026,
+but then meanwhile on master somebody else
+created their own migration 0025.
+
+Anyway, on with the details...
+
+First, I go to showell-topic and run tests to
+make sure that I'm starting with a clean, albeit
+out-of-date, dev branch:
+
+```
+showell@Steves-MBP ~/zulip (showell-approximate) $ git checkout showell-topic
+Switched to branch 'showell-topic'
+
+showell@Steves-MBP ~/zulip (showell-topic) $ git status
+# On branch showell-topic
+nothing to commit, working directory clean
+
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend
+<output skipped>
+DONE!
+```
+
+Next, I fetch changes from upstream:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ git checkout master
+Switched to branch 'master'
+
+showell@Steves-MBP ~/zulip (master) $ git fetch upstream
+
+showell@Steves-MBP ~/zulip (master) $ git merge upstream/master
+Updating 2967341..09754c9
+Fast-forward
+<etc.>
+```
+
+Then I go back to showell-topic:
+
+```
+showell@Steves-MBP ~/zulip (master) $ git checkout showell-topic
+Switched to branch 'showell-topic'
+```
+
+You may want to make note of your HEAD commit on your branch
+before you start rebasing, in case you need to start over, or
+do like I do and rely on being able to find it via github.  I'm
+not showing the details of that, since people have different
+styles for managing botched rebases.
+
+Anyway, I rebase to master as follows:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ git rebase -i master
+Successfully rebased and updated refs/heads/showell-topic.
+```
+
+Note that my rebase was conflict-free from git's point of view,
+but I still need to run the tests to make sure there weren't any
+semantic conflicts with the new changes from master:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend
+
+<output skipped>
+
+  File "/srv/zulip-venv-cache/ad3a375e95a56d911510d7edba7e17280d227bc7/zulip-venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 105, in handle
+    "'python manage.py makemigrations --merge'" % name_str
+django.core.management.base.CommandError: Conflicting migrations detected (0026_topics_backfill, 0025_realm_message_content_edit_limit in zerver).
+To fix them run 'python manage.py makemigrations --merge'
+
+<output skipped>
+
+  File "/srv/zulip/zerver/lib/db.py", line 33, in execute
+    return wrapper_execute(self, super(TimeTrackingCursor, self).execute, query, vars)
+  File "/srv/zulip/zerver/lib/db.py", line 20, in wrapper_execute
+    return action(sql, params)
+django.db.utils.ProgrammingError: relation "zerver_realmfilter" does not exist
+LINE 1: ...n", "zerver_realmfilter"."url_format_string" FROM "zerver_re...
+```
+
+The above traceback is fairly noisy, but it's pretty apparent that
+I have migrations that are out of order.  More precisely, my 0025 migration
+points to 0024 as its dependency, where it really should point to the
+other 0025 as its dependency, and I need to renumber my migrations to
+0026 and 0027.
+
+Let's take a peek at the migrations directory:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ ls -r zerver/migrations/*.py | head -6
+zerver/migrations/__init__.py
+zerver/migrations/0026_topics_backfill.py
+zerver/migrations/0025_realm_message_content_edit_limit.py
+zerver/migrations/0025_add_topic_table.py
+zerver/migrations/0024_realm_allow_message_editing.py
+zerver/migrations/0023_userprofile_default_language.py
+```
+
+We have two different 0025 migrations that both depend on 0024:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ grep -B 1 0024 zerver/migrations/*.py
+zerver/migrations/0025_add_topic_table.py-    dependencies = [
+zerver/migrations/0025_add_topic_table.py:        ('zerver', '0024_realm_allow_message_editing'),
+--
+zerver/migrations/0025_realm_message_content_edit_limit.py-    dependencies = [
+zerver/migrations/0025_realm_message_content_edit_limit.py:        ('zerver', '0024_realm_allow_message_editing'),
+```
+
+I will now start the process of renaming `0025_add_topic_table.py` to
+be `0026_add_topic_table.py` and having it depend on
+`0025_realm_message_content_edit_limit`.  Before I start, I want to
+know which of my commits created my 0025 migration:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ git log --pretty=oneline zerver/migrations/0025_add_topic_table.py
+d859e6ffc165e822cec39152a5814ca7ce94d172 Add Topic and Message.topic to models
+```
+
+Here is the transcript, and hopefully what I did inside of vim is apparent
+from the diff:
+
+```
+showell@Steves-MBP ~/zulip (showell-topic) $ cd zerver/migrations/
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git mv 0025_add_topic_table.py 0026_add_topic_table.py
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic +) $ vim 0026_add_topic_table.py
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git diff
+diff --git a/zerver/migrations/0026_add_topic_table.py b/zerver/migrations/0026_add_topic_table.py
+index 2c8c07a..43351eb 100644
+--- a/zerver/migrations/0026_add_topic_table.py
++++ b/zerver/migrations/0026_add_topic_table.py
+@@ -8,7 +8,7 @@ import zerver.lib.str_utils
+ class Migration(migrations.Migration):
+
+     dependencies = [
+-        ('zerver', '0024_realm_allow_message_editing'),
++        ('zerver', '0025_realm_message_content_edit_limit'),
+     ]
+
+     operations = [
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git commit -am 'temp rename migration'
+[showell-topic 45cf5e9] temp rename migration
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+ rename zerver/migrations/{0025_add_topic_table.py => 0026_add_topic_table.py} (93%)
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git status
+# On branch showell-topic
+nothing to commit, working directory clean
+```
+
+Next, I want to rewrite the history of my branch.  When I'm in the
+interactive rebase (not shown), I need to make the temp commit be a
+"fix" for the original commit that I noted above:
+
+```
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git rebase -i master
+[detached HEAD c1f2e69] Add Topic and Message.topic to models
+ 2 files changed, 38 insertions(+)
+ create mode 100644 zerver/migrations/0026_add_topic_table.py
+Successfully rebased and updated refs/heads/showell-topic.
+```
+
+I did this to verify that I rebased correctly:
+
+```
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log 0026_add_topic_table.py
+commit c1f2e69e716beb4031c628fe2189b49f04770d03
+Author: Steve Howell <showell30@yahoo.com>
+Date:   Thu Jul 14 13:26:15 2016 -0700
+
+    Add Topic and Message.topic to models
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git show c1f2e69e716beb4031c628fe2189b49f04770d03
+<not shown here>
+```
+
+Next, I follow a very similar process for my second migration-related
+commit on my branch:
+
+
+```
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log --pretty=oneline 0026_topics_backfill.py
+ba43e1ffb072f4e6a66ffb5c4030ff3a17d53792 (unfinished) stub commit for topic backfill
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git mv 0026_topics_backfill.py 0027_topics_backfill.py
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic +) $ vim 0027_topics_backfill.py
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git diff
+diff --git a/zerver/migrations/0027_topics_backfill.py b/zerver/migrations/0027_topics_backfill.py
+index 766b075..05ea2bb 100644
+--- a/zerver/migrations/0027_topics_backfill.py
++++ b/zerver/migrations/0027_topics_backfill.py
+@@ -8,7 +8,7 @@ import zerver.lib.str_utils
+ class Migration(migrations.Migration):
+
+     dependencies = [
+-        ('zerver', '0025_add_topic_table'),
++        ('zerver', '0026_add_topic_table'),
+     ]
+
+     operations = [
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic *+) $ git commit -am 'temp rename migration'
+[showell-topic 02ef15b] temp rename migration
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+ rename zerver/migrations/{0026_topics_backfill.py => 0027_topics_backfill.py} (90%)
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git rebase -i master
+[detached HEAD 8022839] (unfinished) stub commit for topic backfill
+ 1 file changed, 20 insertions(+)
+ create mode 100644 zerver/migrations/0027_topics_backfill.py
+Successfully rebased and updated refs/heads/showell-topic.
+```
+
+My rebase looked something like this after I edited the commits:
+
+```
+  1 pick eee291d Add subject_topic_awareness() test helper.
+  2 pick c1f2e69 Add Topic and Message.topic to models
+  3 pick 46c04b5 Call new update_topic() in pre_save_message().
+  4 pick df20dc8 Write to Topic table for topic edits.
+  5 pick ba43e1f (unfinished) stub commit for topic backfill
+  6 f 02ef15b temp rename migration
+  7 pick b8e93d2 Add CATCH_TOPIC_MIGRATION_BUGS.
+  8 pick 644ccae Have get_context_for_message use topic_id.
+  9 pick 6303f5b Have update_message_flags user topic_id.
+ 10 pick 9f9da5a Have narrowing searches use topic id.
+ 11 pick 0b37ef7 Assert that new_topic=True obliterates message.subject
+ 12 pick dff8eee Use new_topics=True in test_bulk_message_fetching().
+ 13 pick bc377a0 Use topic_id when propagating message edits.
+ 14 pick 11fde9c Have message cache use Topic table (and more...).
+ 15 pick 518649b Use topic_name() in to_log_dict().
+ 16
+ 17 # Rebase 09754c9..02ef15b onto 09754c9
+ 18 #
+ 19 # Commands:
+ 20 #  p, pick = use commit
+ 21 #  r, reword = use commit, but edit the commit message
+ 22 #  e, edit = use commit, but stop for amending
+ 23 #  s, squash = use commit, but meld into previous commit
+ 24 #  f, fixup = like "squash", but discard this commit's log message
+ 25 #  x, exec = run command (the rest of the line) using shell
+```
+
+I double check that everything went fine:
+
+```
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git log 0027_topics_backfill.py
+commit 8022839f9168e643ae08365bfb40f1de2e64d426
+Author: Steve Howell <showell30@yahoo.com>
+Date:   Thu Jul 14 15:51:06 2016 -0700
+
+    (unfinished) stub commit for topic backfill
+showell@Steves-MBP ~/zulip/zerver/migrations (showell-topic) $ git show 8022839f9168e643ae08365bfb40f1de2e64d426
+<not shown here>
+```
+
+And then I run the tests and cross my fingers!!!:
+
+```
+(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend
+<output skipped>
+  Applying zerver.0023_userprofile_default_language... OK
+  Applying zerver.0024_realm_allow_message_editing... OK
+  Applying zerver.0025_realm_message_content_edit_limit... OK
+  Applying zerver.0026_add_topic_table... OK
+  Applying zerver.0027_topics_backfill... OK
+  Applying zilencer.0001_initial... OK
+Successfully populated test database.
+DROP DATABASE
+CREATE DATABASE
+Running zerver.tests.test_auth_backends.AuthBackendTest.test_devauth_backend
+Running zerver.tests.test_auth_backends.AuthBackendTest.test_dummy_backend
+Running zerver.tests.test_auth_backends.AuthBackendTest.test_email_auth_backend
+<output skipped>
+FAILED!
+```
+
+Ugh, my tests still fail due to some non-migration-related changes on master.
+The good news, however, is that my migrations are cleaned up.
+
+So I've shown you the excruciating details of fixing up migrations
+in a complicated branch, but let's step back and look at the big
+picture.  You need to get these things right:
+
+- Rename the migrations on your branch.
+- Fix their dependencies.
+- Rewrite your git history so that it appears like you never branched off an old copy of master.
+
+The hardest part of the process will probably be cleaning up your git history.

docs/mypy.md

@@ -1,4 +1,4 @@
-# Testing with the mypy Python static type checker
+# Python static type checker (mypy)
 
 [mypy](http://mypy-lang.org/) is a compile-time static type checker
 for Python, allowing optional, gradual typing of Python code.  Zulip
@@ -28,7 +28,45 @@ You can learn more about it at:
 * [Using mypy with Python 2 code](http://mypy.readthedocs.io/en/latest/python2.html)
 
 The mypy type checker is run automatically as part of Zulip's Travis
-CI testing process.
+CI testing process in the 'static-analysis' build.
+
+## `type_debug.py`
+
+`zerver/lib/type_debug.py` has a useful decorator `print_types`.  It
+prints the types of the parameters of the decorated function and the
+return type whenever that function is called.  This can help find out
+what parameter types a function is supposed to accept, or if
+parameters with the wrong types are being passed to a function.
+
+Here is an example using the interactive console:
+
+```
+>>> from zerver.lib.type_debug import print_types
+>>>
+>>> @print_types
+... def func(x, y):
+...     return x + y
+...
+>>> func(1.0, 2)
+func(float, int) -> float
+3.0
+>>> func('a', 'b')
+func(str, str) -> str
+'ab'
+>>> func((1, 2), (3,))
+func((int, int), (int,)) -> (int, int, int)
+(1, 2, 3)
+>>> func([1, 2, 3], [4, 5, 6, 7])
+func([int, ...], [int, ...]) -> [int, ...]
+[1, 2, 3, 4, 5, 6, 7]
+```
+
+`print_all` prints the type of the first item of lists.  So `[int, ...]` represents
+a list whose first element's type is `int`.  Types of all items are not printed
+because a list can have many elements, which would make the output too large.
+
+Similarly in dicts, one key's type and the corresponding value's type are printed.
+So `{1: 'a', 2: 'b', 3: 'c'}` will be printed as `{int: str, ...}`.
 
 ## Zulip goals
 
@@ -51,8 +89,8 @@ To run mypy on Zulip's python code, run the command:
 
     tools/run-mypy
 
-It will output errors in the same style of a compiler.  For example,
-if your code has a type error like this:
+It will output errors in the same style as a compiler would.  For
+example, if your code has a type error like this:
 
 ```
 foo = 1
@@ -75,16 +113,16 @@ contribute improvements to error messages upstream.
 Since mypy is a new tool under rapid development and occasionally
 makes breaking changes, Zulip is using a pinned version of mypy from
 its [git repository](https://github.com/python/mypy) rather than
-tracking the (older) latest mypy release on pypi.
+tracking the (older) latest mypy release on PyPI.
 
 ## Excluded files
 
-Since several python files in Zulip's code don't pass mypy's checks
+Since several Python files in Zulip's code don't pass mypy's checks
 (even for unannotated code) right now, a list of files to be excluded
-from the check for CI is present in tools/run-mypy.
+from the check for CI is present in `tools/run-mypy`.
 
-To run mypy on all python files, ignoring the exclude list, you can
-pass the `--all` option to tools/run-mypy.
+To run mypy on all Python files, ignoring the exclude list, you can
+pass the `--all` option to `tools/run-mypy`.
 
     tools/run-mypy --all
 
@@ -95,7 +133,7 @@ errors, please remove them from the exclude list.
 
 For the purposes of Zulip development, you can treat `mypy` like a
 much more powerful linter that can catch a wide range of bugs.  If,
-after running tools/run-mypy on your Zulip branch, you get mypy
+after running `tools/run-mypy` on your Zulip branch, you get mypy
 errors, it's important to get to the bottom of the issue, not just do
 something quick to silence the warnings.  Possible explanations include:
 
@@ -110,35 +148,35 @@ something quick to silence the warnings.  Possible explanations include:
 Each explanation has its own solution, but in every case the result
 should be solving the mypy warning in a way that makes the Zulip
 codebase better.  If you need help understanding an issue, please feel
-free to, mention @sharmaeklavya2 or @timabbott on the relevant pull
+free to mention @sharmaeklavya2 or @timabbott on the relevant pull
 request or issue on GitHub.
 
 If you think you have found a bug in Zulip or mypy, inform the zulip
-developers by opening an issue on [Zulip's github
+developers by opening an issue on [Zulip's GitHub
 repository](https://github.com/zulip/zulip/issues) or posting on
 [zulip-devel](https://groups.google.com/d/forum/zulip-devel).  If it's
 indeed a mypy bug, we can help with reporting it upstream.
 
 ## Annotating strings
 
-In python 3, strings can have non-ASCII characters without any problems.
+In Python 3, strings can have non-ASCII characters without any problems.
 Such characters are required to support languages which use non-latin
 scripts like Japanese and Hindi.  They are also needed to support special
 characters like mathematical symbols, musical symbols, etc.
-In python 2, however, `str` generally doesn't work well with non-ASCII
-characters.  That's why `unicode` was introduced in python 2.
+In Python 2, however, `str` generally doesn't work well with non-ASCII
+characters.  That's why `unicode` was introduced in Python 2.
 
 But there are problems with the `unicode` and `str` system.  Implicit
 conversions between `str` and `unicode` use the `ascii` codec, which
 fails on strings containing non-ASCII characters.  Such errors are hard
 to detect by people who always write in English.  To minimize such
 implicit conversions, we should have a strict separation between `str`
-and `unicode` in python 2.  It might seem that using `unicode` everywhere
+and `unicode` in Python 2.  It might seem that using `unicode` everywhere
 will solve all problems, but unfortunately it doesn't.  This is because
-some parts of the standard library and the python language (like keyword
+some parts of the standard library and the Python language (like keyword
 argument unpacking) insist that parameters passed to them are `str`.
 
-To make our code work correctly on python 2, we have to identify strings
+To make our code work correctly in Python 2, we have to identify strings
 which contain data which could come from non-ASCII sources like stream
 names, people's names, domain names, content of messages, emails, etc.
 These strings should be `unicode`.  We also have to identify strings
@@ -148,11 +186,11 @@ names, etc.
 Mypy can help with this.  We just have to annotate each string as either
 `str` or `unicode` and mypy's static type checking will tell us if we
 are incorrectly mixing the two.  However, `unicode` is not defined in
-python 3.  We want our code to be python 3 compatible in the future.
+Python 3.  We want our code to be Python 3 compatible in the future.
 This can be achieved using 'six', a Python 2 and 3 compatibility library.
 
-`six.text_type` is defined as `str` on python 3 and as `unicode` on
-python 2.  We'll be using `text_type` (instead of `unicode`) and `str`
+`six.text_type` is defined as `str` in Python 3 and as `unicode` in
+Python 2.  We'll be using `text_type` (instead of `unicode`) and `str`
 to annotate strings in Zulip's code.  We follow the style of doing
 `from six import text_type` and using `text_type` for annotation instead
 of doing `import six` and using `six.text_type` for annotation, because

docs/new-feature-tutorial.md

@@ -1,5 +1,4 @@
-How to write a new application feature
-======================================
+# Writing a new application feature
 
 The changes needed to add a new feature will vary, of course, but this
 document provides a general outline of what you may need to do, as well
@@ -7,8 +6,7 @@ as an example of the specific steps needed to add a new feature: adding
 a new option to the application that is dynamically synced through the
 data system in real-time to all browsers the user may have open.
 
-General Process
----------------
+## General Process in brief
 
 ### Adding a field to the database
 
@@ -16,9 +14,11 @@ General Process
 `zerver/ models.py`. Add a new field in the appropriate class.
 
 **Create and run the migration:** To create and apply a migration, run:
-:
 
-./manage.py makemigrations ./manage.py migrate
+```
+./manage.py makemigrations
+./manage.py migrate
+```
 
 **Test your changes:** Once you've run the migration, restart memcached
 on your development server (`/etc/init.d/memcached restart`) and then
@@ -37,6 +37,10 @@ based on the event you just created.
 **Backend implementation:** Make any other modifications to the backend
 required for your change.
 
+**New views:** Add any new application views to `zerver/urls.py`. This
+includes both views that serve HTML (new pages on Zulip) as well as new
+API endpoints that serve JSON-formatted data.
+
 **Testing:** At the very least, add a test of your event data flowing
 through the system in `test_events.py`.
 
@@ -60,11 +64,11 @@ precompiled as part of the build/deploy process.
 tests and blackbox end-to-end tests. The blackbox tests are run in a
 headless browser using Casper.js and are located in
 `frontend_tests/casper_tests/`. The unit tests use Node's `assert`
-module are located in `frontend_tests/node_tests/`. For more information
-on writing and running tests see the testing documentation \<testing\>.
+module are located in `frontend_tests/node_tests/`. For more
+information on writing and running tests see the [testing
+documentation](testing.html).
 
-Example Feature
----------------
+## Example Feature
 
 This example describes the process of adding a new setting to Zulip: a
 flag that restricts inviting new users to admins only (the default
@@ -75,12 +79,68 @@ repo](https://github.com/zulip/zulip/commit/5b7f3466baee565b8e5099bcbd3e1ccdbdb0
 (Note that Zulip has since been upgraded from Django 1.6 to 1.8, so the
 migration format has changed.)
 
+### Update the model
+
 First, update the database and model to store the new setting. Add a new
-boolean field, `realm_invite_by_admins_only`, to the Realm model in
+boolean field, `invite_by_admins_only`, to the Realm model in
 `zerver/models.py`.
 
-Then create a Django migration that adds a new field,
-`invite_by_admins_only`, to the `zerver_realm` table.
+``` diff
+--- a/zerver/models.py
++++ b/zerver/models.py
+@@ -139,6 +139,7 @@ class Realm(ModelReprMixin, models.Model):
+     restricted_to_domain = models.BooleanField(default=True) # type: bool
+     invite_required = models.BooleanField(default=False) # type: bool
++    invite_by_admins_only = models.BooleanField(default=False) # type: bool
+     create_stream_by_admins_only = models.BooleanField(default=False) # type: bool
+     mandatory_topics = models.BooleanField(default=False) # type: bool
+```
+
+### Create the migration
+
+Create the migration file: `./manage.py makemigrations`. Make sure to
+commit the generated file to git: `git add zerver/migrations/NNNN_realm_invite_by_admins_only.py`
+(NNNN is a number that is equal to the number of migrations.)
+
+If you run into problems, the [Django migration documentation](https://docs.djangoproject.com/en/1.8/topics/migrations/) is helpful.
+
+### Test your migration changes
+
+Apply the migration: `./manage.py migrate`
+
+Output:
+```
+shell $ ./manage.py migrate
+Operations to perform:
+  Synchronize unmigrated apps: staticfiles, analytics, pipeline
+  Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites, auth, contenttypes
+Synchronizing apps without migrations:
+  Creating tables...
+    Running deferred SQL...
+  Installing custom SQL...
+Running migrations:
+  Rendering model states... DONE
+  Applying zerver.0026_realm_invite_by_admins_only... OK
+```
+
+### Handle database interactions
+
+Next, we will move on to implementing the backend part of this feature.
+Like typical apps, we will need our backend to update the database and
+send some response to the client that made the request.
+
+Beyond that, we need to orchestrate notifications to *other*
+clients (or other users, if you will) that our setting has changed.
+Clients find out about settings through two closely related code
+paths.  When a client first contacts the server, the server sends
+the client its initial state.  Subsequently, clients subscribe to
+"events," which can (among other things) indicate that settings have
+changed.  For the backend piece, we will need our action to make a call to
+`send_event` to send the event to clients that are active.  We will
+also need to modify `fetch_initial_state_data` so that future clients
+see the new changes.
+
+Anyway, getting back to implementation details...
 
 In `zerver/lib/actions.py`, create a new function named
 `do_set_realm_invite_by_admins_only`. This function will update the
@@ -111,6 +171,8 @@ realm. :
       send_event(event, active_user_ids(realm))
       return {}
 
+### Update application state
+
 You then need to add code that will handle the event and update the
 application state. In `zerver/lib/actions.py` update the
 `fetch_initial_state` and `apply_events` functions. :
@@ -129,6 +191,8 @@ already code that will correctly handle the realm update event type: :
            field = 'realm_' + event['property']
            state[field] = event['value']
 
+### Add a new view
+
 You then need to add a view for clients to access that will call the
 newly-added `actions.py` code to update the database. This example
 feature adds a new parameter that should be sent to clients when the
@@ -199,20 +263,18 @@ the server. :
 
     # static/js/server_events.js
 
-    function get_events_success(events) {
-      # ...
-      var dispatch_event = function dispatch_event(event) {
-          switch (event.type) {
-          # ...
-          case 'realm':
+    function dispatch_normal_event(event) {
+        switch (event.type) {
+        # ...
+        case 'realm':
             if (event.op === 'update' && event.property === 'invite_by_admins_only') {
-              page_params.realm_invite_by_admins_only = event.value;
-          }
-      }
+                page_params.realm_invite_by_admins_only = event.value;
+            }
+        }
     }
 
-Any code needed to update the UI should be placed in `dispatch_event`
-callback (rather than the `channel.patch`) function. This ensures the
-appropriate code will run even if the changes are made in another
-browser window. In this example most of the changes are on the backend,
-so no UI updates are required.
+Any code needed to update the UI should be placed in
+`dispatch_normal_event` callback (rather than the `channel.patch`)
+function. This ensures the appropriate code will run even if the
+changes are made in another browser window. In this example most of
+the changes are on the backend, so no UI updates are required.

docs/pointer.md

@@ -0,0 +1,110 @@
+# Unread counts and the pointer
+
+When you're using Zulip and you reload, or narrow to a stream, how
+does Zulip decide where to place you?
+
+Conceptually, Zulip takes you to the place where you left off
+(e.g. the first unread message), not the most recent messages, to
+facilitate reviewing all the discussions that happened while you were
+away from your computer. The scroll position is then set to keep that
+message in view and away from both the top and bottom of the visible
+section of messages.
+
+But there a lot of details around doing this right, and around
+counting unread messages. Here's how Zulip currently decides which
+message to select, along with some notes on improvements we'd like to
+make to the model.
+
+First a bit of terminology:
+
+* "Narrowing" is the process of filtering to a particular subset of
+  the messages the user has access to.
+
+* The blue cursor box (the "pointer") is around is called the
+  "selected" message.  Zulip ensures that the currently selected
+  message is always in-view.
+
+## Pointer logic
+
+### Recipient bar: message you clicked
+
+If you enter a narrow by clicking on a message group's *recipient bar*
+(stream/topic or private message recipient list at the top of a group
+of messages), Zulip will select the the message you clicked on. This
+provides a nice user experience where you get to see the stuff near
+what you clicked on, and in fact the message you clicked on stays at
+exactly the same scroll position in the window after the narrowing as
+it was at before.
+
+### Search or sidebar click: unread/recent matching narrow
+
+If you instead narrow by clicking on something in the left sidebar or
+typing some terms into the search box, Zulip will instead selected on
+the first unread message matching that narrow, or if there are none,
+the most recent messages matching that narrow. This provides the nice
+user experience of taking you to the start of the new stuff (with
+enough messages you'ev seen before still in view at the top to provide
+you with context), which is usually what you want. (When finding the
+"first unread message", Zulip ignores unread messages in muted streams
+or in muted topics within non-muted streams.)
+
+### Unnarrow: previous sequence
+
+When you unnarrow using e.g. the escape key, you will automatically be
+taken to the same message that was selected in the home view before
+you narrowed, unless in the narrow you read new messages, in which
+case you will be jumped forward to the first unread and non-muted
+message in the home view (or the bottom of the feed if there is
+none). This makes for a nice experience reading threads via the home
+view in sequence.
+
+### New home view: "high watermark"
+
+When you open a new browser window or tab to the home view (a.k.a. the
+interleaved view you get if you visit `/`), Zulip will select the
+furthest down that your cursor has ever reached in the home
+view. Because of the logic around unnarrowing in the last bullet, this
+is usually just before the first unread message in the home view, but
+if you never go to the home view, or you leave messages unread on some
+streams in your home view, this can lag.
+
+We plan to change this to automatically advance the pointer in a way
+similar to the unnarrow logic.
+
+### Narrow in a new tab: closest to pointer
+
+When you load a new browser tab or window to a narrowed view, Zulip
+will select the message closest to your pointer, which is what you
+would have got had you loaded the browser window to your home view and
+then clicked on the nearest message matching your narrow (which might
+have been offscreen).
+
+We plan to change this to match the Search/sidebar behavior.
+
+### Forced reload: state preservation
+
+When the server forces a reload of a browser that's otherwise caught
+up (which happens within 30 minutes when a new version of the server
+is deployed, usually at a type when the user isn't looking at the
+browser), Zulip will preserve the state -- what (if any) narrow the
+user was in, the selected message, and even exact scroll position!
+
+For more on the user experience philosophy guiding these decisions,
+see [the architectural overview](architecture-overview.html).
+
+## Unread count logic
+
+How does Zulip decide whether a message has been read by the user?
+The algorithm needs to correctly handle a range of ways people might
+use the product.  The algorithm is as follows:
+
+* Any message which is selected or above a message which is selected
+  is marked as read.  So messages are marked as read as you scroll
+  down the keyboard when the pointer passes over them.
+
+* If the whitspace at the very bottom of the feed is in view, all
+  messages in view are marked as read.
+
+These two simple rules, combined with the pointer logic above, end up
+matching user expectations well for whether the product should treat
+them as having read a set of messages (or not).

docs/prod-authentication-methods.md

@@ -0,0 +1,138 @@
+# Authentication methods
+
+Zulip supports several different authentications methods:
+
+* `EmailAuthBackend` - Email/password authentication.
+* `ZulipLDAPAuthBackend` - LDAP username/password authentication.
+* `GoogleMobileOauth2Backend` - Google authentication.
+* `GitHubAuthBackend` - GitHub authentication.
+* `ZulipRemoteUserBackend` - Authentication using an existing
+  Single-Sign-On (SSO) system that can set REMOTE_USER in Apache.
+* `DevAuthBackend` - Only for development, passwordless login as any user.
+
+It's easy to add more, see the docs on python-social-auth below.
+
+The setup documentation for most of these is simple enough that we've
+included it inline in `/etc/zulip/settings.py`, right above to the
+settings used to configure them.  The remote user authentication
+backend is more complex since it requires interfacing with a generic
+third-party authentication system, and so we've documented it in
+detail below.
+
+## Adding additional methods using python-social-auth
+
+The implementation for GitHubAuthBackend is a small wrapper around the
+popular [python-social-auth] library.  So if you'd like to integrate
+Zulip with another authentication provider (e.g. Facebook, Twitter,
+etc.), you can do this by writing a class similar to
+`GitHubAuthBackend` in `zproject/backends.py` and adding a few
+settings.  Pull requests to add new backends are welcome; they should
+be tested using the framework in `test_auth_backends.py`.
+
+[python-social-auth]: http://psa.matiasaguirre.net/
+
+## Remote User SSO Authentication
+
+Zulip supports integrating with a Single-Sign-On solution.  There are
+a few ways to do it, but this section documents how to configure Zulip
+to use an SSO solution that best supports Apache and will set the
+`REMOTE_USER` variable:
+
+(0) Check that `/etc/zulip/settings.py` has
+`zproject.backends.ZulipRemoteUserBackend` as the only enabled value
+in the `AUTHENTICATION_BACKENDS` list, and that `SSO_APPEND_DOMAIN` is
+correct set depending on whether your SSO system uses email addresses
+or just usernames in `REMOTE_USER`.
+
+Make sure that you've restarted the Zulip server since making this
+configuration change.
+
+(1) Edit `/etc/zulip/zulip.conf` and change the `puppet_classes` line to read:
+
+```
+puppet_classes = zulip::voyager, zulip::apache_sso
+```
+
+(2) As root, run `/home/zulip/deployments/current/scripts/zulip-puppet-apply`
+to install our SSO integration.
+
+(3) To configure our SSO integration, edit
+`/etc/apache2/sites-available/zulip-sso.example` and fill in the
+configuration required for your SSO service to set `REMOTE_USER` and
+place your completed configuration file at `/etc/apache2/sites-available/zulip-sso.conf`
+
+`zulip-sso.example` is correct configuration for using an `htpasswd`
+file for `REMOTE_USER` authentication, which is useful for testing
+quickly.  You can set it up by doing the following:
+
+```
+/home/zulip/deployments/current/scripts/restart-server
+cd /etc/apache2/sites-available/
+cp zulip-sso.example zulip-sso.conf
+htpasswd -c /home/zulip/zpasswd username@example.com # prompts for a password
+```
+
+and then continuing with the steps below.
+
+(4) Run `a2ensite zulip-sso` to enable the Apache integration site.
+
+(5) Run `service apache2 reload` to use your new configuration.  If
+Apache isn't already running, you may need to run `service apache2
+start` instead.
+
+Now you should be able to visit `https://zulip.example.com/` and
+login via the SSO solution.
+
+
+### Troubleshooting Remote User SSO
+
+This system is a little finicky to networking setup (e.g. common
+issues have to do with /etc/hosts not mapping settings.EXTERNAL_HOST
+to the Apache listening on 127.0.0.1/localhost, for example).  It can
+often help while debugging to temporarily change the Apache config in
+/etc/apache2/sites-available/zulip-sso to listen on all interfaces
+rather than just 127.0.0.1 as you debug this.  It can also be helpful
+to change /etc/nginx/zulip-include/app.d/external-sso.conf to
+proxy_pass to a more explicit URL possibly not over HTTPS when
+debugging.  The following log files can be helpful when debugging this
+setup:
+
+* /var/log/zulip/{errors.log,server.log} (the usual places)
+* /var/log/nginx/access.log (nginx access logs)
+* /var/log/apache2/zulip_auth_access.log (you may want to change
+  LogLevel to "debug" in the apache config file to make this more
+  verbose)
+
+Here's a summary of how the remote user SSO system works assuming
+you're using HTTP basic auth; this summary should help with
+understanding what's going on as you try to debug:
+
+* Since you've configured /etc/zulip/settings.py to only define the
+  zproject.backends.ZulipRemoteUserBackend, zproject/settings.py
+  configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes
+  `https://zulip.example.com/` aka the homepage for the main Zulip
+  Django app running behind nginx redirect to /accounts/login/sso if
+  you're not logged in.
+
+* nginx proxies requests to /accounts/login/sso/ to an Apache instance
+  listening on localhost:8888 apache via the config in
+  /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream
+  localhost:8888 defined in /etc/nginx/zulip-include/upstreams).
+
+* The Apache zulip-sso site which you've enabled listens on
+  localhost:8888 and presents the htpasswd dialogue; you provide
+  correct login information and the request reaches a second Zulip
+  Django app instance that is running behind Apache with with
+  REMOTE_USER set.  That request is served by
+  `zerver.views.remote_user_sso`, which just checks the REMOTE_USER
+  variable and either logs in (sets a cookie) or registers the new
+  user (depending whether they have an account).
+
+* After succeeding, that redirects the user back to / on port 443
+  (hosted by nginx); the main Zulip Django app sees the cookie and
+  proceeds to load the site homepage with them logged in (just as if
+  they'd logged in normally via username/password).
+
+Again, most issues with this setup tend to be subtle issues with the
+hostname/DNS side of the configuration.  Suggestions for how to
+improve this SSO setup documentation are very welcome!

docs/prod-customize.md

@@ -0,0 +1,115 @@
+# Customize Zulip
+
+Once you've got Zulip setup, you'll likely want to configure it the
+way you like.  There are four big things to focus on:
+
+1. [Integrations](#integrations)
+2. [Streams and Topics](#streams-and-topics)
+3. [Notification settings](#notification-settings)
+4. [Mobile and desktop apps](#mobile-and-desktop-apps)
+
+Lastly, read about Zulip's other [great features](#all-other-features), and
+then [enjoy your Zulip installation](#enjoy-your-zulip-installation)!
+
+## Integrations
+
+We recommend setting up integrations for the major
+tools that your team works with.  For example, if you're a software
+development team, you may want to start with integrations for your
+version control, issue tracker, CI system, and monitoring tools.
+
+Spend time configuring these integrations to be how you like them --
+if an integration is spammy, you may want to change it to not send
+messages that nobody cares about (E.g. for the zulip.com trac
+integration, some teams find they only want notifications when new
+tickets are opened, commented on, or closed, and not every time
+someone edits the metadata).
+
+If Zulip doesn't have an integration you want, you can add your own!
+Most integrations are very easy to write, and even more complex
+integrations usually take less than a day's work to build.  We very
+much appreciate contributions of new integrations; see the brief
+[integration writing guide](integration-guide.html).
+
+
+It can often be valuable to integrate your own internal processes to
+send notifications into Zulip; e.g. notifications of new customer
+signups, new error reports, or daily reports on the team's key
+metrics; this can often spawn discussions in response to the data.
+
+## Streams and Topics
+
+If it feels like a stream has too much
+traffic about a topic only of interest to some of the subscribers,
+consider adding or renaming streams until you feel like your team is
+working productively.
+
+Second, most users are not used to topics.  It can require a bit of
+time for everyone to get used to topics and start benefitting from
+them, but usually once a team is using them well, everyone ends up
+enthusiastic about how much topics make life easier.  Some tips on
+using topics:
+
+* When replying to an existing conversation thread, just click on the
+  message, or navigate to it with the arrow keys and hit "r" or
+  "enter" to reply on the same topic
+* When you start a new conversation topic, even if it's related to the
+  previous conversation, type a new topic in the compose box
+* You can edit topics to fix a thread that's already been started,
+  which can be helpful when onboarding new batches of users to the platform.
+
+Third, setting default streams for new users is a great way to get
+new users involved in conversations before they've accustomed
+themselves with joining streams on their own. You can use the
+[`set_default_streams`](https://github.com/zulip/zulip/blob/master/zerver/management/commands/set_default_streams.py)
+command to set default streams for users within a realm:
+
+```
+python manage.py set_default_streams --domain=example.com --streams=foo,bar,...
+```
+
+## Notification settings
+
+Zulip gives you a great deal of control
+over which messages trigger desktop notifications; you can configure
+these extensively in the `/#settings` page (get there from the gear
+menu).  If you find the desktop notifications annoying, consider
+changing the settings to only trigger desktop notifications when you
+receive a PM or are @-mentioned.
+
+## Mobile and desktop apps
+
+Currently, the Zulip Desktop app
+only supports talking to servers with a properly signed SSL
+certificate, so you may find that you get a blank screen when you
+connect to a Zulip server using a self-signed certificate.
+
+The Zulip Android app in the Google Play store doesn't yet support
+talking to non-zulip.com servers (and the iOS one doesn't support
+Google auth SSO against non-zulip.com servers; there's a design for
+how to fix that which wouldn't be a ton of work to implement).  If you
+are interested in helping out with the Zulip mobile apps, shoot an
+email to zulip-devel@googlegroups.com and the maintainers can guide
+you on how to help.
+
+For announcements about improvements to the apps, make sure to join
+the zulip-announce@googlegroups.com list so that you can receive the
+announcements when these become available.
+
+## All other features
+
+Hotkeys, emoji, search filters,
+@-mentions, etc.  Zulip has lots of great features, make sure your
+team knows they exist and how to use them effectively.
+
+## Enjoy your Zulip installation!  
+
+If you discover things that you
+wish had been documented, please contribute documentation suggestions
+either via a GitHub issue or pull request; we love even small
+contributions, and we'd love to make the Zulip documentation cover
+everything anyone might want to know about running Zulip in
+production.
+
+Next: [Maintaining and upgrading Zulip in
+production](prod-maintain-secure-upgrade.html).

docs/prod-install.md

@@ -0,0 +1,164 @@
+# Installation
+
+Ensure you have an Ubuntu system that satisfies [the installation
+requirements](prod-requirements.html).  In short, you should have an
+Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server instance,
+with at least 4GB RAM, 2 CPUs, and 10 GB disk space.  You should also
+have a domain name available and have updated its DNS record to point
+to the server.
+
+## Step 0: Subscribe
+
+Please subscribe to low-traffic [the Zulip announcements Google
+Group](https://groups.google.com/forum/#!forum/zulip-announce) to get
+announcements about new releases, security issues, etc.
+
+## Step 1: Install SSL Certificates
+
+Zulip runs over https only and requires ssl certificates in order to
+work. It looks for the certificates in `/etc/ssl/private/zulip.key`
+and `/etc/ssl/certs/zulip.combined-chain.crt`.  Note that Zulip uses
+`nginx` as its webserver and thus [expects a chained certificate
+bundle](http://nginx.org/en/docs/http/configuring_https_servers.html)
+
+If you need an SSL certificate, see [our SSL certificate
+documentation](ssl-certificates.html).  If you already have an SSL
+certificate, just install (or symlink) them into place at the above
+paths, and move on to the next step.
+
+## Step 2: Download and install latest release
+
+If you haven't already, download and unpack [the latest built server
+tarball](https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz)
+with the following commands:
+
+```
+sudo -i  # If not already root
+wget https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz
+rm -rf /root/zulip && mkdir /root/zulip
+tar -xf zulip-server-latest.tar.gz --directory=/root/zulip --strip-components=1
+```
+
+Then, run the Zulip install script:
+```
+/root/zulip/scripts/setup/install
+```
+
+This may take a while to run, since it will install a large number of
+dependencies.
+
+The Zulip install script is designed to be idempotent, so if it fails,
+you can just rerun it after correcting the issue that caused it to
+fail.  Also note that it automatically logs a transcript to
+`/var/log/zulip/install.log`; please include a copy of that file in
+any bug reports.
+
+## Step 3: Configure Zulip
+
+Configure the Zulip server instance by editing `/etc/zulip/settings.py` and
+providing values for the mandatory settings, which are all found under the
+heading `### MANDATORY SETTINGS`.
+
+These settings include:
+
+- `EXTERNAL_HOST`: the user-accessible Zulip domain name for your Zulip
+  installation. This will be the domain for which you have DNS A records
+  pointing to this server and for which you configured SSL certificates.
+
+- `ZULIP_ADMINISTRATOR`: the email address of the person or team maintaining
+  this installation and who will get support emails.
+
+- `AUTHENTICATION_BACKENDS`: a list of enabled authentication
+  mechanisms.  You'll need to enable at least one authentication
+  mechanism by uncommenting its corresponding line, and then also do
+  any additional configuration required for that backend as documented
+  in the `settings.py` file.  See the [section on
+  Authentication](prod-auth-first-login.html) for more detail on the
+  available authentication backends and how to configure them.
+
+- `EMAIL_*`, `DEFAULT_FROM_EMAIL`, and `NOREPLY_EMAIL_ADDRESS`:
+  Regardless of which authentication backends you enable, you must
+  provide settings for an outgoing SMTP server so Zulip can send
+  emails when needed.  We highly recommend testing your configuration
+  using `manage.py send_test_email` to confirm your outgoing email
+  configuration is working correctly.
+
+- `ALLOWED_HOSTS`: Replace `*` with the fully qualified DNS name for
+  your Zulip server here.
+
+## Step 4: Initialize Zulip database
+
+At this point, you are done doing things as root.  To initialize the
+Zulip database for your production install, run:
+
+```
+su zulip -c /home/zulip/deployments/current/scripts/setup/initialize-database
+```
+
+The `initialize-database` script will report an error if you did not
+fill in all the mandatory settings from `/etc/zulip/settings.py`.  It
+is safe to rerun it after correcting the problem if that happens.
+
+This completes the process of installing Zulip on your server.
+However, in order to use Zulip, you'll need to create an organization
+in your Zulip installation.
+
+## Step 5: Create a Zulip organization and login
+
+* If you haven't already, verify that your server can send email using
+`./manage.py send_test_email username@example.com`.  You'll need
+working outgoing email to complete the setup process.
+
+* Run the organization (realm) creation [management
+command](prod-maintain-secure-upgrade.html#management-commands) :
+
+  ```
+  su zulip # If you weren't already the zulip user
+  cd /home/zulip/deployments/current
+  ./manage.py generate_realm_creation_link
+  ```
+
+  This will print out a secure 1-time use link that allows creation of a
+  new Zulip organization on your server.  For most servers, you will
+  only ever do this once, but you can run `manage.py
+  generate_realm_creation_link` again if you want to host another
+  organization on your Zulip server.
+
+* Open the link generated with your web browser. You'll see the create
+organization page ([screenshot here](_images/zulip-create-realm.png)).
+Enter your email address and click *Create organization*.
+
+* Check your email to find the confirmation email and click the
+link. You'll be prompted to finish setting up your organization and
+initial administrator user ([screenshot
+here](_images/zulip-create-user-and-org.png)).  Complete this form and
+log in!
+
+**Congratulations!** You are logged in as an organization
+administrator for your new Zulip organization.  After getting
+oriented, we recommend visiting the special "Administration" tab
+linked to from the upper-right gear menu in the Zulip app to configure
+important policy settings like how users can join your new
+organization.  By default, your organization will be configured as
+follows ([screenshot here](_images/zulip-admin-settings.png)):
+
+* `restricted_to_domain=True`: Only people with emails with the same ending as yours can join.
+* `invite_required=False`: An invitation is not required to join the realm.
+* `invite_by_admin_only=False`: You don't need to be an admin user to invite other users.
+
+Next, you'll likely want to do one of the following:
+
+* [Customize your Zulip organization](prod-customize.html).
+* [Learn about managing a production Zulip server](prod-maintain-secure-upgrade.html).
+
+## Troubleshooting
+
+If you get an error after `scripts/setup/install` completes, check
+`/var/log/zulip/errors.log` for a traceback, and consult the
+[troubleshooting section](prod-troubleshooting.html) for advice on
+how to debug.  If that doesn't help, please visit [the "installation
+help" stream in the Zulip developers'
+chat](https://zulip.tabbott.net/#narrow/stream/installation.20help)
+for realtime help or email zulip-help@googlegroups.com with the
+traceback and we'll try to help you out!
+

docs/prod-maintain-secure-upgrade.md

@@ -0,0 +1,576 @@
+# Secure, maintain, and upgrade
+
+This page covers topics that will help you maintain a healthy, up-to-date, and
+secure Zulip installation, including:
+
+- [Upgrading](#upgrading)
+- [Upgrading from a git repository](#upgrading-from-a-git-repository)
+- [Backups](#backups)
+- [Monitoring](#monitoring)
+- [Scalability](#scalability)
+- [Security Model](#security-model)
+- [Management commands](#management-commands)
+
+
+## Upgrading
+
+**We recommend reading this entire section before doing your first
+upgrade.**
+
+To upgrade to a new version of the zulip server, download the appropriate
+release tarball from
+[https://www.zulip.com/dist/releases/](https://www.zulip.com/dist/releases/)
+
+You also have the option of creating your own release tarballs from a
+copy of zulip.git repository using `tools/build-release-tarball`. And,
+starting with Zulip version 1.4, you can upgrade Zulip [to a version
+in a Git repository directly](#upgrade-from-a-git-repository).
+
+Next, run as root:
+
+```
+/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
+```
+
+The upgrade process will shut down the Zulip service and then run `apt-get upgrade`, a
+puppet apply, any database migrations, and then bring the Zulip service back
+up. Upgrading will result in some brief downtime for the service, which should be
+under 30 seconds unless there is an expensive transition involved. Unless you
+have tested the upgrade in advance, we recommend doing upgrades at off hours.
+
+Note that upgrading an existing Zulip production server from Ubuntu
+14.04 Trusty to Ubuntu 16.04 Xenial will require significant manual
+intervention on your part to migrate the data in the database from
+Postgres 9.3 to Postgres 9.5.  Contributions on testing and
+documenting this process are welcome!
+
+### Preserving local changes to configuration files
+
+**Warning**: If you have modified configuration files installed by
+Zulip (e.g. the nginx configuration), the Zulip upgrade process will
+overwrite your configuration when it does the `puppet apply`.
+
+You can test whether this will happen assuming no upstream changes to
+the configuration using `scripts/zulip-puppet-apply` (without the
+`-f` option), which will do a test puppet run and output and changes
+it would make. Using this list, you can save a copy of any files
+that you've modified, do the upgrade, and then restore your
+configuration.
+
+If you need to do this, please report the issue so
+that we can make the Zulip puppet configuration flexible enough to
+handle your setup.
+
+### Troubleshooting with the upgrade log
+
+The Zulip upgrade script automatically logs output to
+`/var/log/zulip/upgrade.log`. Please use those logs to include output
+that shows all errors in any bug reports.
+
+After the upgrade, we recommend checking `/var/log/zulip/errors.log`
+to confirm that your users are not experiencing errors after the
+upgrade.
+
+### Rolling back to a prior version
+
+The Zulip upgrade process works by creating a new deployment under
+`/home/zulip/deployments/` containing a complete copy of the Zulip server code,
+and then moving the symlinks at `/home/zulip/deployments/current` and
+`/root/zulip` as part of the upgrade process.
+
+This means that if the new version isn't working,
+you can quickly downgrade to the old version by using
+`/home/zulip/deployments/<date>/scripts/restart-server` to return to
+a previous version that you've deployed (the version is specified
+via the path to the copy of `restart-server` you call).
+
+### Updating settings
+
+If required, you can update your settings by editing `/etc/zulip/settings.py`
+and then run `/home/zulip/deployments/current/scripts/restart-server` to
+restart the server.
+
+### Applying Ubuntu system updates
+
+While the Zulip upgrade script runs `apt-get upgrade`, you are responsible for
+running this on your system on a regular basis between Zulip upgrades to
+ensure that it is up to date with the latest security patches.
+
+### API and your Zulip URL
+
+To use the Zulip API with your Zulip server, you will need to use the
+API endpoint of e.g. `https://zulip.example.com/api`.  Our Python
+API example scripts support this via the
+`--site=https://zulip.example.com` argument.  The API bindings
+support it via putting `site=https://zulip.example.com` in your
+.zuliprc.
+
+Every Zulip integration supports this sort of argument (or e.g. a
+`ZULIP_SITE` variable in a zuliprc file or the environment), but this
+is not yet documented for some of the integrations (the included
+integration documentation on `/integrations` will properly document
+how to do this for most integrations).  We welcome pull requests for
+integrations that don't discuss this!
+
+Similarly, you will need to instruct your users to specify the URL
+for your Zulip server when using the Zulip desktop and mobile apps.
+
+### Memory leak mitigation
+
+As a measure to mitigate the impact of potential memory leaks in one
+of the Zulip daemons, the service automatically restarts itself
+every Sunday early morning.  See `/etc/cron.d/restart-zulip` for the
+precise configuration.
+
+## Upgrading from a git repository
+
+Starting with version 1.4, the Zulip server supports doing deployments
+from a Git repository.  To configure this, you will need to add
+`zulip::static_asset_compiler` to your `/etc/zulip/zulip.conf` file's
+`puppet_classes` entry, like this:
+
+```
+puppet_classes = zulip::voyager, zulip::static_asset_compiler
+```
+
+Then, run `scripts/zulip-puppet-apply` to install the dependencies for
+building Zulip's static assets.  You can configure the `git`
+repository that you'd like to use by adding a section like this to
+`/etc/zulip/zulip.conf`; by default it uses the main `zulip`
+repository (shown below).
+
+```
+[deployment]
+git_repo_url = https://github.com/zulip/zulip.git
+```
+
+Once that is done (and assuming the currently installed version of
+Zulip is new enough that this script exists), you can do deployments
+by running as root:
+
+```
+/home/zulip/deployments/current/scripts/upgrade-zulip-from-git <branch>
+```
+
+and Zulip will automatically fetch the relevant branch from the
+specified repository, build the static assets, and deploy that
+version.  Currently, the upgrade process is slow, but it doesn't need
+to be; there is ongoing work on optimizing it.
+
+## Backups
+
+There are several pieces of data that you might want to back up:
+
+* The postgres database.  That you can back up like any postgres
+database; we have some example tooling for doing that incrementally
+into S3 using [wal-e](https://github.com/wal-e/wal-e) in
+`puppet/zulip_internal/manifests/postgres_common.pp` (that's what we
+use for zulip.com's database backups).  Note that this module isn't
+part of the Zulip server releases since it's part of the zulip.com
+configuration (see https://github.com/zulip/zulip/issues/293 for a
+ticket about fixing this to make life easier for running backups).
+
+* Any user-uploaded files.  If you're using S3 as storage for file
+uploads, this is backed up in S3, but if you have instead set
+LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars)
+will be stored in that directory and you'll want to back it up.
+
+* Your Zulip configuration including secrets from /etc/zulip/.
+E.g. if you lose the value of secret_key, all users will need to login
+again when you setup a replacement server since you won't be able to
+verify their cookies; if you lose avatar_salt, any user-uploaded
+avatars will need to be re-uploaded (since avatar filenames are
+computed using a hash of avatar_salt and user's email), etc.
+
+* The logs under /var/log/zulip can be handy to have backed up, but
+they do get large on a busy server, and it's definitely
+lower-priority.
+
+### Restore from backups
+
+To restore from backups, the process is basically the reverse of the above:
+
+* Install new server as normal by downloading a Zulip release tarball
+  and then using `scripts/setup/install`, you don't need
+  to run the `initialize-database` second stage which puts default
+  data into the database.
+
+* Unpack to /etc/zulip the settings.py and secrets.conf files from your backups.
+
+* Restore your database from the backup using wal-e; if you ran
+  `initialize-database` anyway above, you'll want to first
+  `scripts/setup/postgres-init-db` to drop the initial database first.
+
+* If you're using local file uploads, restore those files to the path
+  specified by `settings.LOCAL_UPLOADS_DIR` and (if appropriate) any
+  logs.
+
+* Start the server using scripts/restart-server
+
+This restoration process can also be used to migrate a Zulip
+installation from one server to another.
+
+We recommend running a disaster recovery after you setup backups to
+confirm that your backups are working; you may also want to monitor
+that they are up to date using the Nagios plugin at:
+`puppet/zulip_internal/files/nagios_plugins/check_postgres_backup`.
+
+Contributions to more fully automate this process or make this section
+of the guide much more explicit and detailed are very welcome!
+
+
+### Postgres streaming replication
+
+Zulip has database configuration for using Postgres streaming
+replication; you can see the configuration in these files:
+
+* puppet/zulip_internal/manifests/postgres_slave.pp
+* puppet/zulip_internal/manifests/postgres_master.pp
+* puppet/zulip_internal/files/postgresql/*
+
+Contribution of a step-by-step guide for setting this up (and moving
+this configuration to be available in the main `puppet/zulip/` tree)
+would be very welcome!
+
+
+## Monitoring
+
+The complete Nagios configuration (sans secret keys) used to
+monitor zulip.com is available under `puppet/zulip_internal` in the
+Zulip Git repository (those files are not installed in the release
+tarballs).
+
+The Nagios plugins used by that configuration are installed
+automatically by the Zulip installation process in subdirectories
+under `/usr/lib/nagios/plugins/`.  The following is a summary of the
+various Nagios plugins included with Zulip and what they check:
+
+Application server and queue worker monitoring:
+
+* check_send_receive_time (sends a test message through the system
+  between two bot users to check that end-to-end message sending works)
+
+* check_rabbitmq_consumers and check_rabbitmq_queues (checks for
+  rabbitmq being down or the queue workers being behind)
+
+* check_queue_worker_errors (checks for errors reported by the queue workers)
+
+* check_worker_memory (monitors for memory leaks in queue workers)
+
+* check_email_deliverer_backlog and check_email_deliverer_process
+  (monitors for whether outgoing emails are being sent)
+
+Database monitoring:
+
+* check_postgres_replication_lag (checks streaming replication is up
+  to date).
+
+* check_postgres (checks the health of the postgres database)
+
+* check_postgres_backup (checks backups are up to date; see above)
+
+* check_fts_update_log (monitors for whether full-text search updates
+  are being processed)
+
+Standard server monitoring:
+
+* check_website_response.sh (standard HTTP check)
+
+* check_debian_packages (checks apt repository is up to date)
+
+If you're using these plugins, bug reports and pull requests to make
+it easier to monitor Zulip and maintain it in production are
+encouraged!
+
+## Scalability
+
+This section attempts to address the considerations involved with
+running Zulip with a large team (>1000 users).
+
+* We recommend using a [remote postgres
+  database](#postgres-database-details) for isolation, though it is
+  not required.  In the following, we discuss a relatively simple
+  configuration with two types of servers: application servers
+  (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and
+  database servers.
+
+* You can scale to a pretty large installation (O(~1000) concurrently
+  active users using it to chat all day) with just a single reasonably
+  large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB
+  of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB
+  RAM actively in use).  You can probably get away with half that
+  (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale.
+  Beyond a 1000 active users, you will eventually want to increase the
+  memory cap in `memcached.conf` from the default 512MB to avoid high
+  rates of memcached misses.
+
+* For the database server, we highly recommend SSD disks, and RAM is
+  the primary resource limitation.  We have not aggressively tested
+  for the minimum resources required, but 8 cores with 30GB of RAM
+  (e.g. AWS's m3.2xlarge) should suffice; you may be able to get away
+  with less especially on the CPU side.  The database load per user is
+  pretty optimized as long as `memcached` is working correctly.  This
+  has not been tested, but from extrapolating the load profile, it
+  should be possible to scale a Zulip installation to 10,000s of
+  active users using a single large database server without doing
+  anything complicated like sharding the database.
+
+* For reasonably high availability, it's easy to run a hot spare
+  application server and a hot spare database (using Postgres
+  streaming replication; see the section on configuring this).  Be
+  sure to check out the section on backups if you're hoping to run a
+  spare application server; in particular you probably want to use the
+  S3 backend for storing user-uploaded files and avatars and will want
+  to make sure secrets are available on the hot spare.
+
+* Zulip does not support dividing traffic for a given Zulip realm
+  between multiple application servers.  There are two issues: you
+  need to share the memcached/redis/rabbitmq instance (these should
+  can be moved to a network service shared by multiple servers with a
+  bit of configuration) and the Tornado event system for pushing to
+  browsers currently has no mechanism for multiple frontend servers
+  (or event processes) talking to each other.  One can probably get a
+  factor of 10 in a single server's scalability by [supporting
+  multiple tornado processes on a single
+  server](https://github.com/zulip/zulip/issues/372), which is also
+  likely the first part of any project to support exchanging events
+  amongst multiple servers.
+
+Questions, concerns, and bug reports about this area of Zulip are very
+welcome!  This is an area we are hoping to improve.
+
+## Security Model
+
+This section attempts to document the Zulip security model.  Since
+this is new documentation, it likely does not cover every issue; if
+there are details you're curious about, please feel free to ask
+questions on the Zulip development mailing list (or if you think
+you've found a security bug, please report it to
+security@googlegroups.com so we can do a responsible security
+announcement).
+
+### Secure your Zulip server like your email server
+
+* It's reasonable to think about security for a Zulip server like you
+  do security for a team email server -- only trusted administrators
+  within an organization should have shell access to the server.
+
+  In particular, anyone with root access to a Zulip application server
+  or Zulip database server, or with access to the `zulip` user on a
+  Zulip application server, has complete control over the Zulip
+  installation and all of its data (so they can read messages, modify
+  history, etc.).  It would be difficult or impossible to avoid this,
+  because the server needs access to the data to support features
+  expected of a group chat system like the ability to search the
+  entire message history, and thus someone with control over the
+  server has access to that data as well.
+
+### Encryption and Authentication
+
+* Traffic between clients (web, desktop and mobile) and the Zulip is
+  encrypted using HTTPS.  By default, all Zulip services talk to each
+  other either via a localhost connection or using an encrypted SSL
+  connection.
+
+* The preferred way to login to Zulip is using an SSO solution like
+  Google Auth, LDAP, or similar.  Zulip stores user passwords using
+  the standard PBKDF2 algorithm.  Password strength is checked and
+  weak passwords are visually discouraged using the zxcvbn library,
+  but Zulip does not by default have strong requirements on user
+  password strength.  Modify `static/js/common.js` to adjust the
+  password strength requirements (Patches welcome to make controlled
+  by an easy setting!).
+
+* Zulip requires CSRF tokens in all interactions with the web API to
+  prevent CSRF attacks.
+
+### Messages and History
+
+* Zulip message content is rendering using a specialized Markdown
+  parser which escapes content to protect against cross-site scripting
+  attacks.
+
+* Zulip supports both public streams and private ("invite-only")
+  streams.  Any Zulip user can join any public stream in the realm
+  (and can view the complete message of any public stream history
+  without joining the stream).
+
+* Users who are not members of a private stream cannot read messages
+  on the stream, send messages to the stream, or join the stream, even
+  if they are a Zulip administrator.  However, any member of a private
+  stream can invite other users to the stream.  When a new user joins
+  a private stream, they can see future messages sent to the stream,
+  but they do not receive access to the stream's message history.
+
+* Zulip supports editing the content or topics of messages that have
+  already been sent (and even updating the topic of messages sent by
+  other users when editing the topic of the overall thread).
+
+  While edited messages are synced immediately to open browser
+  windows, editing messages is not a safe way to redact secret content
+  (e.g. a password) unintentionally shared via Zulip, because other
+  users may have seen and saved the content of the original message
+  (for example, they could have taken a screenshot immediately after
+  you sent the message, or have an API tool recording all messages
+  they receive).
+
+  Zulip stores and sends to clients the content of every historical
+  version of a message, so that future versions of Zulip could support
+  displaying the diffs between previous versions.
+
+### Users and Bots
+
+* There are three types of users in a Zulip realm: Administrators,
+  normal users, and bots.  Administrators have the ability to
+  deactivate and reactivate other human and bot users, delete streams,
+  add/remove administrator privileges, as well as change configuration
+  for the overall realm (e.g. whether an invitation is required to
+  join the realm).  Being a Zulip administrator does not provide the
+  ability to interact with other users' private messages or the
+  messages sent to private streams to which the administrator is not
+  subscribed.  However, a Zulip administrator subscribed to a stream
+  can toggle whether that stream is public or private.  Also, Zulip
+  realm administrators have administrative access to the API keys of
+  all bots in the realm, so a Zulip administrator may be able to
+  access messages sent to private streams that have bots subscribed,
+  by using the bot's credentials.
+
+  In the future, Zulip's security model may change to allow realm
+  administrators to access private messages (e.g. to support auditing
+  functionality).
+
+* Every Zulip user has an API key, available on the settings page.
+  This API key can be used to do essentially everything the user can
+  do; for that reason, users should keep their API key safe.  Users
+  can rotate their own API key if it is accidentally compromised.
+
+* To properly remove a user's access to a Zulip team, it does not
+  suffice to change their password or deactivate their account in the
+  SSO system, since neither of those prevents authenticating with the
+  user's API key or those of bots the user has created.  Instead, you
+  should deactivate the user's account in the Zulip administration
+  interface (/#administration); this will automatically also
+  deactivate any bots the user had created.
+
+* The Zulip mobile apps authenticate to the server by sending the
+  user's password and retrieving the user's API key; the apps then use
+  the API key to authenticate all future interactions with the site.
+  Thus, if a user's phone is lost, in addition to changing passwords,
+  you should rotate the user's Zulip API key.
+
+* Zulip bots are used for integrations.  A Zulip bot can do everything
+  a normal user in the realm can do including reading other, with a
+  few exceptions (e.g. a bot cannot login to the web application or
+  create other bots).  In particular, with the API key for a Zulip
+  bot, one can read any message sent to a public stream in that bot's
+  realm.  A likely future feature for Zulip is [limited bots that can
+  only send messages](https://github.com/zulip/zulip/issues/373).
+
+* Certain Zulip bots can be marked as "API super users"; these special
+  bots have the ability to send messages that appear to have been sent
+  by another user (an important feature for implementing integrations
+  like the Jabber, IRC, and Zephyr mirrors).
+
+### User-uploaded content
+
+* Zulip supports user-uploaded files; ideally they should be hosted
+  from a separate domain from the main Zulip server to protect against
+  various same-domain attacks (e.g. zulip-user-content.example.com)
+  using the S3 integration.
+
+  The URLs of user-uploaded files are secret; if you are using the
+  "local file upload" integration, anyone with the URL of an uploaded
+  file can access the file.  This means the local uploads integration
+  is vulnerable to a subtle attack where if a user clicks on a link in
+  a secret .PDF or .HTML file that had been uploaded to Zulip, access
+  to the file might be leaked to the other server via the Referrer
+  header (see https://github.com/zulip/zulip/issues/320).
+
+  The Zulip S3 file upload integration is relatively safe against that
+  attack, because the URLs of files presented to users don't host the
+  content.  Instead, the S3 integration checks the user has a valid
+  Zulip session in the relevant realm, and if so then redirects the
+  browser to a one-time S3 URL that expires a short time later.
+  Keeping the URL secret is still important to avoid other users in
+  the Zulip realm from being able to access the file.
+
+* Zulip supports using the Camo image proxy to proxy content like
+  inline image previews that can be inserted into the Zulip message
+  feed by other users over HTTPS.
+
+* By default, Zulip will provide image previews inline in the body of
+  messages when a message contains a link to an image.  You can
+  control this using the `INLINE_IMAGE_PREVIEW` setting.
+
+### Final notes and security response
+
+If you find some aspect of Zulip that seems inconsistent with this
+security model, please report it to zulip-security@googlegroups.com so that we can
+investigate and coordinate an appropriate security release if needed.
+
+Zulip security announcements will be sent to
+zulip-announce@googlegroups.com, so you should subscribe if you are
+running Zulip in production.
+
+## Management commands
+
+Zulip has a large library of [Django management
+commands](https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-and-manage-py).
+To use them, you will want to be logged in as the `zulip` user and for
+the purposes of this documentation, we assume the current working
+directory is `/home/zulip/deployments/current`.
+
+Below, we should several useful examples, but there are more than 100
+in total.  We recommend skimming the usage docs (or if there are none,
+the code) of a management command before using it, since they are
+generally less polished and more designed for expert use than the rest
+of the Zulip system.
+
+### manage.py shell
+
+You can get an iPython shell with full access to code within the Zulip
+project using `manage.py shell`, e.g. you can do the following to
+change an email address:
+
+```
+$ /home/zulip/deployments/current/manage.py shell
+In [1]: user_profile = get_user_profile_by_email("email@example.com")
+In [2]: do_change_user_email(user_profile, "new_email@example.com")
+```
+
+#### manage.py dbshell
+
+This will start a postgres shell connected to the Zulip database.
+
+### Grant administrator access
+
+You can make any user a realm administrator on the command line with
+the `knight` management command:
+
+```
+./manage.py knight username@example.com -f
+```
+
+#### Creating api super users with manage.py
+
+If you need to manage the IRC, Jabber, or Zephyr mirrors, you will
+need to create api super users.  To do this, use `./manage.py knight`
+with the `--permission=api_super_user` argument.  See
+`bots/irc-mirror.py` and `bots/jabber_mirror.py` for further detail on
+these.
+
+
+### Other useful manage.py commands
+
+There are a large number of useful management commands under
+`zerver/manangement/commands/`; you can also see them listed using
+`./manage.py` with no arguments.
+
+One such command worth highlighting because it's a valuable feature
+with no UI in the Administration page is `./manage.py realm_filters`,
+which allows you to configure certain patterns in messages to be
+automatically linkified, e.g., whenever someone mentions "T1234", it
+could be auto-linkified to ticket 1234 in your team's Trac instance.
+
+
+Next: [Remote User SSO Authentication.](prod-remote-user-sso-auth.html)

docs/prod-postgres.md

@@ -0,0 +1,133 @@
+Postgres database details
+=========================
+
+#### Remote Postgres database
+
+This is a bit annoying to setup, but you can configure Zulip to use a
+dedicated postgres server by setting the `REMOTE_POSTGRES_HOST`
+variable in /etc/zulip/settings.py, and configuring Postgres
+certificate authentication (see
+http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and
+http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
+documentation on how to set this up and deploy the certificates) to
+make the DATABASES configuration in `zproject/settings.py` work (or
+override that configuration).
+
+If you want to use a remote Postgresql database, you should configure
+the information about the connection with the server. You need a user
+called "zulip" in your database server. You can configure these
+options in /etc/zulip/settings.py:
+
+* REMOTE_POSTGRES_HOST: Name or IP address of the remote host
+* REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are:
+  * disable: I don't care about security, and I don't want to pay the overhead of encryption.
+  * allow: I don't care about security, but I will pay the overhead of encryption if the server insists on it.
+  * prefer: I don't care about encryption, but I wish to pay the overhead of encryption if the server supports it.
+  * require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want.
+  * verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust.
+  * verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it's the one I specify.
+
+Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf:
+
+```
+postgres_password = xxxx
+```
+
+Finally, you can stop your database on the Zulip server via:
+
+```
+sudo service postgresql stop
+sudo update-rc.d postgresql disable
+```
+
+In future versions of this feature, we'd like to implement and
+document how to the remote postgres database server itself
+automatically by using the Zulip install script with a different set
+of puppet manifests than the all-in-one feature; if you're interested
+in working on this, post to the Zulip development mailing list and we
+can give you some tips.
+
+#### Debugging postgres database issues
+
+When debugging postgres issues, in addition to the standard `pg_top`
+tool, often it can be useful to use this query:
+
+```
+SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY procpid;
+```
+
+which shows the currently running backends and their activity. This is
+similar to the pg_top output, with the added advantage of showing the
+complete query, which can be valuable in debugging.
+
+To stop a runaway query, you can run `SELECT pg_cancel_backend(pid
+int)` or `SELECT pg_terminate_backend(pid int)` as the 'postgres'
+user. The former cancels the backend's current query and the latter
+terminates the backend process. They are implemented by sending SIGINT
+and SIGTERM to the processes, respectively.  We recommend against
+sending a Postgres process SIGKILL. Doing so will cause the database
+to kill all current connections, roll back any pending transactions,
+and enter recovery mode.
+
+#### Stopping the Zulip postgres database
+
+To start or stop postgres manually, use the pg_ctlcluster command:
+
+```
+pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
+```
+
+By default, using stop uses "smart" mode, which waits for all clients
+to disconnect before shutting down the database. This can take
+prohibitively long. If you use the --force option with stop,
+pg_ctlcluster will try to use the "fast" mode for shutting
+down. "Fast" mode is described by the manpage thusly:
+
+  With the --force option the "fast" mode is used which rolls back all
+  active transactions, disconnects clients immediately and thus shuts
+  down cleanly. If that does not work, shutdown is attempted again in
+  "immediate" mode, which can leave the cluster in an inconsistent state
+  and thus will lead to a recovery run at the next start. If this still
+  does not help, the postmaster process is killed. Exits with 0 on
+  success, with 2 if the server is not running, and with 1 on other
+  failure conditions. This mode should only be used when the machine is
+  about to be shut down.
+
+Many database parameters can be adjusted while the database is
+running. Just modify /etc/postgresql/9.1/main/postgresql.conf and
+issue a reload. The logs will note the change.
+
+#### Debugging issues starting postgres
+
+pg_ctlcluster often doesn't give you any information on why the
+database failed to start. It may tell you to check the logs, but you
+won't find any information there. pg_ctlcluster runs the following
+command underneath when it actually goes to start Postgres:
+
+```
+/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o  '-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
+```
+
+Since pg_ctl doesn't redirect stdout or stderr, running the above can
+give you better diagnostic information. However, you might want to
+stop Postgres and restart it using pg_ctlcluster after you've debugged
+with this approach, since it does bypass some of the work that
+pg_ctlcluster does.
+
+
+#### Postgres Vacuuming alerts
+
+The `autovac_freeze` postgres alert from `check_postgres` is
+particularly important.  This alert indicates that the age (in terms
+of number of transactions) of the oldest transaction id (XID) is
+getting close to the `autovacuum_freeze_max_age` setting.  When the
+oldest XID hits that age, Postgres will force a VACUUM operation,
+which can often lead to sudden downtime until the operation finishes.
+If it did not do this and the age of the oldest XID reached 2 billion,
+transaction id wraparound would occur and there would be data loss.
+To clear the nagios alert, perform a `VACUUM` in each indicated
+database as a database superuser (`postgres`).
+
+See
+http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
+for more details on postgres vacuuming.

docs/prod-requirements.md

@@ -0,0 +1,57 @@
+# Requirements
+
+Note that if you just want to play around with Zulip and see what it looks
+like, it is easier to install it in a development environment
+following [these
+instructions](readme-symlink.html#installing-the-zulip-development-environment),
+since then you don't need to worry about setting up SSL certificates and an
+authentication mechanism. Or, you can check out the
+[developers' chatroom](http://zulip.tabbott.net/) (a public, running Zulip
+instance).
+
+## Server
+
+#### Hardware Specifications
+
+* CPU and Memory: For installations with 100+ users you'll need a minimum of
+  **2 CPUs** and **4GB RAM**. For installations with fewer users, 1 CPU and 2GB
+  RAM might be sufficient. We strong recommend against installing with less
+  than 2GB of RAM, as you will likely experience out of memory issues.
+
+* Disk space: You'll need at least 10GB of free disk space. If you intend to
+  store uploaded files locally rather than on S3 you will likely need more.
+
+#### Network and Security Specifications
+
+* Outgoing HTTP(S) access to the public Internet. If you want to be able to
+  send email from Zulip, you'll also need SMTP access.
+
+#### Operating System
+
+Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running
+Zulip in production. 64-bit is recommended.
+
+#### Domain name
+
+You should already have a domain name available for your Zulip
+production instance. In order to generate valid SSL certificates with Let's
+Encrypt, and to enable other services such as Google Authentication, you'll
+need to update the domains A record to point to your production server.
+
+## Credentials needed
+
+#### SSL Certificate
+
+* SSL Certificate for the host you're putting this on (e.g. zulip.example.com).
+  The installation instructions contain documentation for how to get an SSL
+  certificate for free using [LetsEncrypt](https://letsencrypt.org/).
+
+#### Outgoing email
+
+* Email credentials Zulip can use to send outgoing emails to users
+  (e.g. email address confirmation emails during the signup process,
+  missed message notifications, password reminders if you're not using
+  SSO, etc.).
+
+Once you have met these requirements, see [full instructions for installing
+Zulip in production](prod-install.html).

docs/prod-troubleshooting.md

@@ -0,0 +1,122 @@
+# Troubleshooting
+
+Zulip uses [Supervisor](http://supervisord.org/index.html) to monitor
+and control its many Python services. Read the next section, [Using
+supervisorctl](#using-supervisorctl), to learn how to use the
+Supervisor client to monitor and manage services.
+
+If you haven't already, now might be a good time to read Zulip's [architectural
+overview](architecture-overview.html), particularly the
+[Components](architecture-overview.html#components) section. This will help you
+understand the many services Zulip uses.
+
+If you encounter issues while running Zulip, take a look at Zulip's logs, which
+are located in  `/var/log/zulip/`. That directory contains one log file for
+each service, plus `errors.log` (has all errors), `server.log` (has logs from
+the Django and Tornado servers), and `workers.log` (has combined logs from the
+queue workers).
+
+The section [troubleshooting services](#troubleshooting-services)
+on this page includes details about how to fix common issues with Zulip services.
+
+If you run into additional problems, [please report
+them](https://github.com/zulip/zulip/issues) so that we can update
+this page!  The Zulip installation scripts logs its full output to
+`/var/log/zulip/install.log`, so please include the context for any
+tracebacks from that log.
+
+## Using supervisorctl
+
+To see what Zulip-related services are configured to
+use Supervisor, look at `/etc/supervisor/conf.d/zulip.conf` and
+`/etc/supervisor/conf.d/zulip-db.conf`.
+
+Use the supervisor client `supervisorctl` to list the status of, stop, start,
+and restart various services.
+
+### Checking status with `supervisorctl status`
+
+You can check if the zulip application is running using:
+```
+supervisorctl status
+```
+
+When everything is running as expected, you will see something like this:
+
+```
+process-fts-updates                                             RUNNING   pid 2194, uptime 1:13:11
+zulip-django                                                    RUNNING   pid 2192, uptime 1:13:11
+zulip-senders:zulip-events-message_sender-0                     RUNNING   pid 2209, uptime 1:13:11
+zulip-senders:zulip-events-message_sender-1                     RUNNING   pid 2210, uptime 1:13:11
+zulip-senders:zulip-events-message_sender-2                     RUNNING   pid 2211, uptime 1:13:11
+zulip-senders:zulip-events-message_sender-3                     RUNNING   pid 2212, uptime 1:13:11
+zulip-senders:zulip-events-message_sender-4                     RUNNING   pid 2208, uptime 1:13:11
+zulip-tornado                                                   RUNNING   pid 2193, uptime 1:13:11
+zulip-workers:zulip-deliver-enqueued-emails                     STARTING
+zulip-workers:zulip-events-confirmation-emails                  RUNNING   pid 2199, uptime 1:13:11
+zulip-workers:zulip-events-digest_emails                        RUNNING   pid 2205, uptime 1:13:11
+zulip-workers:zulip-events-email_mirror                         RUNNING   pid 2203, uptime 1:13:11
+zulip-workers:zulip-events-error_reports                        RUNNING   pid 2200, uptime 1:13:11
+zulip-workers:zulip-events-feedback_messages                    RUNNING   pid 2207, uptime 1:13:11
+zulip-workers:zulip-events-missedmessage_mobile_notifications   RUNNING   pid 2204, uptime 1:13:11
+zulip-workers:zulip-events-missedmessage_reminders              RUNNING   pid 2206, uptime 1:13:11
+zulip-workers:zulip-events-signups                              RUNNING   pid 2198, uptime 1:13:11
+zulip-workers:zulip-events-slowqueries                          RUNNING   pid 2202, uptime 1:13:11
+zulip-workers:zulip-events-user-activity                        RUNNING   pid 2197, uptime 1:13:11
+zulip-workers:zulip-events-user-activity-interval               RUNNING   pid 2196, uptime 1:13:11
+zulip-workers:zulip-events-user-presence                        RUNNING   pid 2195, uptime 1:13:11
+```
+
+### Restarting services with `supervisorctl restart all`
+
+After you change configuration in `/etc/zulip/settings.py` or fix a
+misconfiguration, you will often want to restart the Zulip application.
+You can restart Zulip using:
+
+```
+supervisorctl restart all
+```
+
+### Stopping services with `supervisorctl stop all`
+
+Similarly, you can stop Zulip using:
+
+```
+supervisorctl stop all
+```
+
+## Troubleshooting services
+
+The Zulip application uses several major open source services to store
+and cache data, queue messages, and otherwise support the Zulip
+application:
+
+* postgresql
+* rabbitmq-server
+* nginx
+* redis
+* memcached
+
+If one of these services is not installed or functioning correctly,
+Zulip will not work.  Below we detail some common configuration
+problems and how to resolve them:
+
+* An AMQPConnectionError traceback or error running rabbitmqctl
+  usually means that RabbitMQ is not running; to fix this, try:
+  ```
+  service rabbitmq-server restart
+  ```
+  If RabbitMQ fails to start, the problem is often that you are using
+  a virtual machine with broken DNS configuration; you can often
+  correct this by configuring `/etc/hosts` properly.
+
+* If your browser reports no webserver is running, that is likely
+  because nginx is not configured properly and thus failed to start.
+  nginx will fail to start if you configured SSL incorrectly or did
+  not provide SSL certificates.  To fix this, configure them properly
+  and then run:
+  ```
+  service nginx restart
+  ```
+
+Next: [Making your Zulip instance awesome.](prod-customize.html)

docs/queuing.md

@@ -1,4 +1,4 @@
-# RabbitMQ queues
+# Queue processors
 
 Zulip uses RabbitMQ to manage a system of internal queues.  These are
 used for a variety of purposes:
@@ -41,10 +41,11 @@ To add a new queue processor:
 * Define the processor in `zerver/worker/queue_processors.py` using
   the `@assign_queue` decorator; it's pretty easy to get the template
   for an existing similar queue processor.  This suffices to test your
-  queue worker in the Zulip development environment, though you'll
-  need to restart `tools/run-dev.py` in order to run your new queue
-  processor.  You can also run a single queue processor manually using
-  e.g. `./manage.py process_queue --queue=user_activity`.
+  queue worker in the Zulip development environment
+  (`tools/run-dev.py` will automatically restart the queue processors
+  and start running your new queue processor code).  You can also run
+  a single queue processor manually using e.g. `./manage.py
+  process_queue --queue=user_activity`.
 
 * So that supervisord will known to run the queue processor in
   production, you will need to define a program entry for it in

docs/roadmap.md

@@ -10,8 +10,8 @@ Zulip community.  From when Zulip was released as open source in late
 September 2015 through today (mid-April, 2016), over 300 pull requests
 have been submitted to the various Zulip repositories (and over 250
 have been merged!), the vast majority of which are submitted by
-Zulip's users around the world (as opposed to the small core team who
-review and merge the pull requests).
+Zulip's users around the world (as opposed to the small core team that
+reviews and merges the pull requests).
 
 In any project, there can be a lot of value in periodically putting
 together a roadmap detailing the major areas where the project is
@@ -36,26 +36,33 @@ community (if you're looking for a starter project, see the [guide to
 getting involved with
 Zulip](https://github.com/zulip/zulip#how-to-get-involved-with-contributing-to-zulip)).
 
+We occasionally update this roadmap by adding strikethrough for issues
+that have been resolved.
+
 Without further ado, below is the Zulip 2016 roadmap.
 
 ## Burning problems
 
 The top problem for the Zulip project is the state of the mobile apps.
 The Android app has started seeing rapid progress thanks to a series
-of contributions by Lisa Neigut of Recurse Center, and we believe to
-be on a good path.  The iOS app has fewer features than Android and
+of contributions by Lisa Neigut of Recurse Center, and we believe it
+is on a good path.  The iOS app has fewer features than Android and
 has more bugs, but more importantly is in need of an experienced iOS
 developer who has time to drive the project.
 
+Update: Neeraj Wahi is leading an effort on to write a [new React
+Native iOS app for Zulip](https://github.com/zulip/zulip-mobile) to
+replace the old iOS app.
+
 ## Core User Experience
 
 This category includes important improvements to the core user
 experience that will benefit all users.
 
-* [Improve missed message notifications to make "reply" work nicely](https://github.com/zulip/zulip/issues/612)
+* <strike>[Improve missed message notifications to make "reply" work nicely](https://github.com/zulip/zulip/issues/612)</strike>
 * [Add support for showing "user is typing" notifications](https://github.com/zulip/zulip/issues/150)
 * [Add pretty bubbles for recipients in the compose box](https://github.com/zulip/zulip/issues/595)
-* [Finish and merge support for pinning a few important streams](https://github.com/zulip/zulip/issues/285)
+* <strike>[Finish and merge support for pinning a few important streams](https://github.com/zulip/zulip/issues/285)</strike>
 * [Display stream descriptions more prominently](https://github.com/zulip/zulip/issues/164)
 * [Integration inline URL previews](https://github.com/zulip/zulip/issues/406)
 * [Add support for managing uploaded files](https://github.com/zulip/zulip/issues/454)
@@ -78,10 +85,10 @@ The core Zulip UI has been mostly translated into 5 languages;
 however, more work is required to make those translations actually
 displayed in the Zulip UI for the users who would benefit from them.
 
-* [Merge support for using translations in Django templates](https://github.com/zulip/zulip/pull/607)
-* [Add text in handlebars templates to translatable string database](https://github.com/zulip/zulip/issues/726)
-* [Merge support for translating text in handlebars](https://github.com/zulip/zulip/issues/726)
-* [Add text in error messages to translatable strings](https://github.com/zulip/zulip/issues/727)
+* <strike>[Merge support for using translations in Django templates](https://github.com/zulip/zulip/pull/607)</strike>
+* <strike>[Add text in handlebars templates to translatable string database](https://github.com/zulip/zulip/issues/726)</strike>
+* <strike>[Merge support for translating text in handlebars](https://github.com/zulip/zulip/issues/726)</strike>
+* <strike>[Add text in error messages to translatable strings](https://github.com/zulip/zulip/issues/727)</strike>
 
 ## User Experience at scale
 
@@ -93,15 +100,15 @@ teams.
 * [Improve @-mentioning syntax based on stronger unique identifiers](https://github.com/zulip/zulip/issues/374)
 * [Show subscriber counts on streams](https://github.com/zulip/zulip/pull/525)
 * [Make the streams page easier to navigate with 100s of streams](https://github.com/zulip/zulip/issues/563)
-* [Add support for filtering long lists of streams](https://github.com/zulip/zulip/issues/565)
+* <strike>[Add support for filtering long lists of streams](https://github.com/zulip/zulip/issues/565)</strike>
 
 ## Administration and management
 
 Currently, Zulip has a number of administration features that can be
 controlled only via the command line.
 
-* [Make default streams web-configurable](https://github.com/zulip/zulip/issues/665)
-* [Make realm emoji web-configurable](https://github.com/zulip/zulip/pull/543)
+* <strike>[Make default streams web-configurable](https://github.com/zulip/zulip/issues/665)</strike>
+* <strike>[Make realm emoji web-configurable](https://github.com/zulip/zulip/pull/543)</strike>
 * [Make realm filters web-configurable](https://github.com/zulip/zulip/pull/544)
 * [Make realm aliases web-configurable](https://github.com/zulip/zulip/pull/651)
 * [Enhance the LDAP integration and make it web-configurable](https://github.com/zulip/zulip/issues/715)
@@ -122,12 +129,12 @@ initial goal is working well with only 2GB of RAM).
 ## Performance
 
 Performance is essential for a communication tool.  While some things
-are already quite good (E.g. narrowing and message sending is speedy),
+are already quite good (e.g. narrowing and message sending is speedy),
 this is an area where one can always improve.  There are a few known
 performance opportunities:
 
-* [Migrate to faster jinja2 templating engine](https://github.com/zulip/zulip/issues/620)
-* [Don't load zxcvbn when it isn't needed](https://github.com/zulip/zulip/issues/263)
+* <strike>[Migrate to faster jinja2 templating engine](https://github.com/zulip/zulip/issues/620)</strike>
+* <strike>[Don't load zxcvbn when it isn't needed](https://github.com/zulip/zulip/issues/263)</strike>
 * [Optimize the frontend performance of loading the Zulip webapp using profiling](https://github.com/zulip/zulip/issues/714)
 
 ## Technology improvements
@@ -147,7 +154,7 @@ While the Zulip server has a great codebase compared to most projects
 of its size, it takes work to keep it that way.
 
 * [Migrate most web routes to REST API](https://github.com/zulip/zulip/issues/611)
-* [Finish purging global variables from the Zulip javascript](https://github.com/zulip/zulip/issues/610)
+* [Finish purging global variables from the Zulip JavaScript](https://github.com/zulip/zulip/issues/610)
 * [Finish deprecating and remove the pre-REST Zulip /send_message API](https://github.com/zulip/zulip/issues/730)
 * [Split Tornado subsystem into a separate Django app](https://github.com/zulip/zulip/issues/729)
 * [Clean up clutter in the root of the zulip.git repository](https://github.com/zulip/zulip/issues/707)
@@ -155,14 +162,14 @@ of its size, it takes work to keep it that way.
 
 ## Deployment and upgrade process
 
-* [Support backwards-incompatible upgrades to Python libraries](https://github.com/zulip/zulip/issues/717)
-* [Minimize the downtime required in Zulip upgrade process](https://github.com/zulip/zulip/issues/646)
+* <strike>[Support backwards-incompatible upgrades to Python libraries](https://github.com/zulip/zulip/issues/717)</strike>
+* [Minimize the downtime required in the Zulip upgrade process](https://github.com/zulip/zulip/issues/646)
 
 ## Security
 
 * [Add support for 2-factor authentication on all platforms](https://github.com/zulip/zulip/pull/451)
 * [Add a retention policy feature that automatically deletes old messages](https://github.com/zulip/zulip/issues/106)
-* [Upgrade every Zulip dependency to a modern version](https://github.com/zulip/zulip/issues/717)
+* [Upgrade every Zulip dependency to a modern version](https://github.com/zulip/zulip/issues/1331)
 * [The LOCAL_UPLOADS_DIR file uploads backend only supports world-readable uploads](https://github.com/zulip/zulip/issues/320)
 * [Add support for stronger security controls for uploaded files](https://github.com/zulip/zulip/issues/320)
 
@@ -171,19 +178,18 @@ of its size, it takes work to keep it that way.
 * [Extend Zulip's automated test coverage to include all API endpoints](https://github.com/zulip/zulip/issues/732)
 * [Build automated tests for the client API bindings](https://github.com/zulip/zulip/issues/713)
 * [Add Python static type-checking to Zulip using mypy](https://github.com/zulip/zulip/issues/733)
-* [Improve the runtime of Zulip's backend test suite](https://github.com/zulip/zulip/issues/441)
-* [Use caching to make Travis CI runtimes faster](https://github.com/zulip/zulip/issues/712)
+* <strike>[Improve the runtime of Zulip's backend test suite](https://github.com/zulip/zulip/issues/441)</strike>
+* <strike>[Use caching to make Travis CI runtimes faster](https://github.com/zulip/zulip/issues/712)</strike>
 * [Add automated tests for the production upgrade process](https://github.com/zulip/zulip/issues/306)
-* [Improve Travis CI "production" test suite to catch more regressions](https://github.com/zulip/zulip/issues/598)
+* <strike>[Improve Travis CI "production" test suite to catch more regressions](https://github.com/zulip/zulip/issues/598)</strike>
 
 ## Development environment
 
 * [Migrate from jslint to eslint](https://github.com/zulip/zulip/issues/535)
-* [Figure out a nice upgrade process for Zulip Vagrant VMs](https://github.com/zulip/zulip/issues/264)
-* [Overhaul new contributor documentation](https://github.com/zulip/zulip/issues/677)
+* <strike>[Figure out a nice upgrade process for Zulip Vagrant VMs](https://github.com/zulip/zulip/issues/264)</strike>
 * [Replace closure-compiler with a faster minifier toolchain](https://github.com/zulip/zulip/issues/693)
 * [Add support for building frontend features in React](https://github.com/zulip/zulip/issues/694)
-* [Use a javascript bundler like webpack](https://github.com/zulip/zulip/issues/695)
+* [Use a JavaScript bundler like webpack](https://github.com/zulip/zulip/issues/695)
 
 ## Documentation
 
@@ -204,28 +210,29 @@ reasonably good framework for writing new webhook integrations for
 getting notifications into Zulip, it'd be great to streamline that
 process and make bots that receive messages just as easy to build.
 
-* [Make it super easy to take screenshots for new webhook integrations](https://github.com/zulip/zulip/issues/658)
+* <strike>[Make it super easy to take screenshots for new webhook integrations](https://github.com/zulip/zulip/issues/658)</strike>
 * [Add an outgoing webhook integration system](https://github.com/zulip/zulip/issues/735)
-* [Build a framework to cut duplicated code in new webhook integrations](https://github.com/zulip/zulip/issues/660)
+* <strike>[Build a framework to cut duplicated code in new webhook integrations](https://github.com/zulip/zulip/issues/660)</strike>
 * [Make setting up a new integration a smooth flow](https://github.com/zulip/zulip/issues/692)
-* [Optimize the integration writing documentation to make writing new
-   ones really easy.](https://github.com/zulip/zulip/issues/70)
+* <strike>[Optimize the integration writing documentation to make writing new
+   ones really easy.](https://github.com/zulip/zulip/issues/70)</strike>
 
 ## Android app
 
 The Zulip Android app is ahead of the iOS app in terms of feature set,
-so this section serves to document the goals for Zulip on mobile.
+but there is still a lot of work to do. Most of the things listed below
+will eventually apply to the iOS app as well.
 
-* [Support using a non-zulip.com server](https://github.com/zulip/zulip-android/issues/1)
+* <strike>[Support using a non-zulip.com server](https://github.com/zulip/zulip-android/issues/1)</strike>
 * [Support Google authentication with a non-Zulip.com server](https://github.com/zulip/zulip-android/issues/49)
 * [Add support for narrowing to @-mentions](https://github.com/zulip/zulip-android/issues/39)
 * [Support having multiple Zulip realms open simultaneously](https://github.com/zulip/zulip-android/issues/47)
-* [Build a slick development login page to simplify testing (similar to
-  the development homepage on web)](https://github.com/zulip/zulip-android/issues/48)
-* [Improve the compose box to let you see what you're replying to](https://github.com/zulip/zulip-android/issues/8)
+* <strike>[Build a slick development login page to simplify testing (similar to
+  the development homepage on web)](https://github.com/zulip/zulip-android/issues/48)</strike>
+* <strike>[Improve the compose box to let you see what you're replying to](https://github.com/zulip/zulip-android/issues/8)</strike>
 * [Make it easy to compose messages with mentions, emoji, etc.](https://github.com/zulip/zulip-android/issues/11)
 * [Display unread counts and improve navigation](https://github.com/zulip/zulip-android/issues/57)
-* [Hide messages sent to muted topics](https://github.com/zulip/zulip-android/issues/9)
+* <strike>[Hide messages sent to muted topics](https://github.com/zulip/zulip-android/issues/9)</strike>
 * [Fill out documentation to make it easy to get started](https://github.com/zulip/zulip-android/issues/58)
 
 ## iOS app
@@ -237,17 +244,22 @@ iOS app.  Once we have that resolved, we'll expand our ambitions for
 the app with more specific improvements.
 
 * [iOS app needs maintainer](https://github.com/zulip/zulip-ios/issues/12)
-* [APNS notifications are broken](https://github.com/zulip/zulip/issues/538)
+* <strike>[APNS notifications are broken](https://github.com/zulip/zulip/issues/538)</strike>
 
 ## Desktop apps
 
-The top goal for the desktop apps is to rebuild it in modern toolchain
-(probably Electron) so that it's easy for a wide range of developers
-to contribute to the apps.
+The top goal for the desktop apps is to rebuild it in a modern
+toolchain so that it's easy for a wide range of developers to
+contribute to the apps. The new [cross-platform
+app](https://github.com/zulip/zulip-electron) is implemented in
+[Electron](http://electron.atom.io/), a framework (maintained by
+GitHub) that uses Chromium and Node.js, so Zulip developers only need
+to write HTML, CSS, and JavaScript. The new Zulip app is in alpha as of
+early August 2016.
 
 * Migrate platform from QT/webkit to Electron
 * Desktop app doesn't recover well from entering the wrong Zulip server
-* Support having multiple Zulip realms open simultaneously
+* [Support having multiple Zulip realms open simultaneously](https://github.com/zulip/zulip-electron/issues/1)
 * Build an efficient process for testing and releasing new versions of
   the desktop apps
 
@@ -256,7 +268,7 @@ to contribute to the apps.
 These don't get GitHub issues since they're not technical projects,
 but they are important goals for the project.
 
-* Setup a Zulip server for the Zulip development community
+* <strike>Setup a Zulip server for the Zulip development community</strike>
 * Expand the number of core developers able to do code reviews
-* Expand the number of contributors regularly adding features to Zulip
+* <strike>Expand the number of contributors regularly adding features to Zulip</strike>
 * Have a successful summer with Zulip's 3 GSOC students

docs/schema-migrations.md

@@ -2,8 +2,8 @@
 
 Zulip uses the [standard Django system for doing schema
 migrations](https://docs.djangoproject.com/en/1.8/topics/migrations/).
-There is some example usage in the Zulip new feature tutorial on
-readthedocs.
+There is some example usage in the [new feature
+tutorial](new-feature-tutorial.html).
 
 This page documents some important issues related to writing schema
 migrations.
@@ -18,6 +18,8 @@ migrations.
 * **Numbering conflicts across branches**: If you've done your schema
   change in a branch, and meanwhile another schema change has taken
   place, Django will now have two migrations with the same number. To
-  fix this, you can just rename the file, as long as no other
-  migrations depend on it (in which case you also need to update the
-  dependencies).
+  fix this, you need to renumber your migration(s), fix up
+  the "dependencies" entries in your migration(s), and rewrite your
+  git history as needed.  There is a tutorial
+  [here](migration-renumbering.html) that walks you though that
+  process.

docs/settings.md

@@ -1,4 +1,4 @@
-# Zulip settings
+# Settings system
 
 The page documents the Zulip settings system, and hopefully should
 help you decide how to correctly implement new settings you're adding
@@ -9,6 +9,16 @@ set via the /#administration page in the Zulip web application) and
 apply to a single Zulip realm/organization (which for most Zulip
 servers is the only realm on the server).
 
+Philosophically, the goals of the settings system are to make it
+convenient for:
+
+* Zulip server administrations to configure
+Zulip's featureset for their server without needing to patch Zulip
+* Realm administrators to configure settings for their organization
+independently without needing to talk with the server administrator.
+* Secrets (passwords, API keys, etc.) to be stored in a separate place
+from shareable configuration.
+
 ## Server settings
 
 Zulip uses the [Django settings
@@ -36,8 +46,8 @@ of settings needed by the Zulip Django app.  As a result, there are a
 few files involved in the Zulip settings for server administrations.
 In a production environment, we have:
 
-* `/etc/zulip/settings.py` (generated from
-  `zproject/local_settings_template.py`) is the main system
+* `/etc/zulip/settings.py` (the template is in the Zulip repo at
+  `zproject/prod_settings_template.py`) is the main system
   administration facing settings file for Zulip.  It contains all the
   server-specific settings, such as how to send outgoing email, the
   hostname of the Postgres database, etc., but does not contain any
@@ -50,7 +60,10 @@ In a production environment, we have:
   `scripts/setup/generate-secrets.py` as part of installation)
   contains secrets used by the Zulip installation.  These are read
   using the standard Python `ConfigParser`, and accessed in
-  `zproject/settings.py` by the `get_secret` function.
+  `zproject/settings.py` by the `get_secret` function.  All
+  secrets/API keys/etc. used by the Zulip Django application should be
+  stored here, and read using the `get_secret` function in
+  `zproject/settings.py`.
 
 * `zproject/settings.py` is the main Django settings file for Zulip.
   It contains all the settings that are constant for all Zulip
@@ -58,15 +71,15 @@ In a production environment, we have:
   middleware, etc.), as well as default values for the settings the
   user would set in `/etc/zulip/settings.py` (you can look at the
   `DEFAULT_SETTINGS` dictionary to easily review the settings
-  available).  `zproject/settings.py` has a line `from local_settings
+  available).  `zproject/settings.py` has a line `from prod_settings
   import *`, which has the effect of importing
-  `/etc/zulip/settings.py`.
+  `/etc/zulip/settings.py` in a prod environment (via a symlink).
 
 In a development environment, we have `zproject/settings.py`, and
 additionally:
 
 * `zproject/dev_settings.py` has the settings for the Zulip development
-  environment; it mostly just imports local_settings_template.py.
+  environment; it mostly just imports prod_settings_template.py.
 
 * `zproject/dev-secrets.conf` replaces `/etc/zulip/zulip-secrets.conf`.
 
@@ -78,9 +91,11 @@ When adding a new server setting to Zulip, you will typically add it
 in two or three places:
 
 * In DEFAULT_SETTINGS in `zproject/settings.py`, with a default value
-  for production environments.
+  for production environments.  If the settings has a secret key,
+  you'll add a `get_secret` call in `zproject/settings.py` (and the
+  user will add the value when they configure the feature).
 
-* In an appropriate section of `zproject/local_settings_template.py`,
+* In an appropriate section of `zproject/prod_settings_template.py`,
   with documentation in the comments explaining the settings's
   purpose and effect.
 
@@ -124,4 +139,4 @@ replaced with realm settings:
   server, and in the realm settings indicating which methods the realm
   administrator wants to allow users to login with.
 
-[doc-newfeat]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
+[doc-newfeat]: new-feature-tutorial.html

docs/ssl-certificates.md

@@ -0,0 +1,71 @@
+### Using Let's Encrypt
+
+If you have a domain name and you've configured DNS to point to the
+server where you want to install Zulip, you can use [Let's
+Encrypt](https://letsencrypt.org/) to generate a valid, properly
+signed SSL certificates, for free.
+
+Run all of these commands as root. If you're not already logged in as root, use
+`sudo -i` to start an interactive root shell.
+
+First, install the Let's Encrypt client [Certbot](https://certbot.eff.org/) and
+then generate the certificate:
+
+```
+wget https://dl.eff.org/certbot-auto
+chmod a+x certbot-auto
+./certbot-auto certonly --standalone
+```
+
+Note: If you already had a webserver installed on this system (e.g. you
+previously installed Zulip and are now getting a cert), you will
+need to stop the webserver (e.g. `service nginx stop`) and start it
+again after (e.g. `service nginx start`) running the certbot command above.
+
+Next, symlink the certificates to make them available where Zulip expects them.
+Be sure to replace YOUR_DOMAIN with your domain name.
+
+```
+ln -s /etc/letsencrypt/live/YOUR_DOMAIN/privkey.pem /etc/ssl/private/zulip.key
+ln -s /etc/letsencrypt/live/YOUR_DOMAIN/fullchain.pem /etc/ssl/certs/zulip.combined-chain.crt
+```
+
+Note: Certificates provided by Let's Encrypt are valid for 90 days and then
+need to be [renewed](https://certbot.eff.org/docs/using.html#renewal). You can
+renew with this command:
+
+```
+./certbot-auto renew
+```
+
+### Generating a self-signed certificate
+
+If you aren't able to use Let's Encrypt, you can generate a
+self-signed ssl certificate.  We recommend getting a real certificate
+using LetsEncrypt over this approach because your browser (and some of
+the Zulip clients) will complain when connecting to your server that
+the certificate isn't signed.
+
+Run all of these commands as root. If you're not already logged in as root, use
+`sudo -i` to start an interactive root shell.
+
+```
+apt-get install openssl
+openssl genrsa -des3 -passout pass:x -out server.pass.key 4096
+openssl rsa -passin pass:x -in server.pass.key -out zulip.key
+rm server.pass.key
+openssl req -new -key zulip.key -out server.csr
+openssl x509 -req -days 365 -in server.csr -signkey zulip.key -out zulip.combined-chain.crt
+rm server.csr
+cp zulip.key /etc/ssl/private/zulip.key
+cp zulip.combined-chain.crt /etc/ssl/certs/zulip.combined-chain.crt
+```
+
+You will eventually want to get a properly signed SSL certificate, but
+this will let you finish the installation process.
+
+### If you are using a self-signed certificate with an IP address (no domain)
+
+Finally, if you want to proceed with just an IP address, it is
+possible to finish a Zulip installation that way; just set
+EXTERNAL_HOST to be the IP address.

docs/testing-with-casper.md

@@ -0,0 +1,124 @@
+# Web frontend black-box casperjs tests
+
+These live in `frontend_tests/casper_tests/`. This is a "black box"
+test; we load the frontend in a real (headless) browser, from a real dev
+server, and simulate UI interactions like sending messages, narrowing,
+etc.
+
+Since this is interacting with a real dev server, it can catch backend
+bugs as well.
+
+You can run this with `./tools/test-js-with-casper` or as
+`./tools/test-js-with-casper 06-settings.js` to run a single test file
+from `frontend_tests/casper_tests/`.
+
+## Debugging Casper.JS
+
+Casper.js (via PhantomJS) has support for remote debugging. However, it
+is not perfect. Here are some steps for using it and gotchas you might
+want to know.
+
+To turn on remote debugging, pass `--remote-debug` to the
+`./frontend_tests/run-casper` script. This will run the tests with port
+`7777` open for remote debugging. You can now connect to
+`localhost:7777` in a Webkit browser. Somewhat recent versions of Chrome
+or Safari might be required.
+
+-   When connecting to the remote debugger, you will see a list of
+    pages, probably 2. One page called `about:blank` is the headless
+    page in which the CasperJS test itself is actually running in. This
+    is where your test code is.
+-   The other page, probably `localhost:9981`, is the Zulip page that
+    the test is testing---that is, the page running our app that our
+    test is exercising.
+
+Since the tests are now running, you can open the `about:blank` page,
+switch to the Scripts tab, and open the running `0x-foo.js` test. If you
+set a breakpoint and it is hit, the inspector will pause and you can do
+your normal JS debugging. You can also put breakpoints in the Zulip
+webpage itself if you wish to inspect the state of the Zulip frontend.
+
+You can also check the screenshots of failed tests at `/tmp/casper-failure*.png`.
+
+If you need to use print debugging in casper, you can do using
+`casper.log`; see <http://docs.casperjs.org/en/latest/logging.html> for
+details.
+
+An additional debugging technique is to enable verbose mode in the
+Casper tests; you can do this by adding to the top of the relevant test
+file the following:
+
+>     var casper = require('casper').create({
+>        verbose: true,
+>        logLevel: "debug"
+>     });
+
+This can sometimes give insight into exactly what's happening.
+
+## Writing Casper tests
+
+Probably the easiest way to learn how to write Casper tests is to study
+some of the existing test files. There are a few tips that can be useful
+for writing Casper tests in addition to the debugging notes below:
+
+-   Run just the file containing your new tests as described above to
+    have a fast debugging cycle.
+-   With frontend tests in general, it's very important to write your
+    code to wait for the right events. Before essentially every action
+    you take on the page, you'll want to use `waitForSelector`,
+    `waitUntilVisible`, or a similar function to make sure the page or
+    elemant is ready before you interact with it. For instance, if you
+    want to click a button that you can select via `#btn-submit`, and
+    then check that it causes `success-elt` to appear, you'll want to
+    write something like:
+
+        casper.waitForSelector("#btn-submit", function () {
+           casper.click('#btn-submit')
+           casper.test.assertExists("#success-elt");
+         });
+
+    This will ensure that the element is present before the interaction
+    is attempted. The various wait functions supported in Casper are
+    documented in the Casper here:
+    <http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector>
+    and the various assert statements available are documented here:
+    <http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype>
+
+- The 'waitFor' style functions (waitForSelector, etc.) cannot be
+    chained together in certain conditions without creating race
+    conditions where the test may fail nondeterministically. For
+    example, don't do this:
+
+        casper.waitForSelector('tag 1');
+        casper.waitForSelector('tag 2');
+
+    Instead, if you want to avoid race condition, wrap the second
+    `waitFor` in a `then` function like this:
+
+        casper.waitForSelector('tag 1');
+        casper.then(function () {
+            casper.waitForSelector('tag 2');
+        });
+
+-   Casper uses CSS3 selectors; you can often save time by testing and
+    debugging your selectors on the relevant page of the Zulip
+    development app in the Chrome JavaScript console by using e.g.
+    `$$("#settings-dropdown")`.
+-   The test suite uses a smaller set of default user accounts and other
+    data initialized in the database than the development environment;
+    to see what differs check out the section related to
+    `options["test_suite"]` in
+    `zilencer/management/commands/populate_db.py`.
+-   Casper effectively runs your test file in two phases -- first it
+    runs the code in the test file, which for most test files will just
+    collect a series of steps (each being a `casper.then` or
+    `casper.wait...` call). Then, usually at the end of the test file,
+    you'll have a `casper.run` call which actually runs that series of
+    steps. This means that if you write code in your test file outside a
+    `casper.then` or `casper.wait...` method, it will actually run
+    before all the Casper test steps that are declared in the file,
+    which can lead to confusing failures where the new code you write in
+    between two `casper.then` blocks actually runs before either of
+    them. See this for more details about how Casper works:
+    <http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work>
+

docs/testing-with-django.md

@@ -0,0 +1,245 @@
+# Backend Django tests
+
+## Overview
+
+Zulip uses the Django framework for its Python back end.  We
+use the testing framework from
+[django.test](https://docs.djangoproject.com/en/1.10/topics/testing/)
+to test our code.  We have over a thousand automated tests that verify that
+our backend works as expected.
+
+All changes to the Zulip backend code should be supported by tests.  We
+enforce our testing culture during code review, and we also use
+coverage tools to measure how well we test our code.  We mostly use
+tests to prevent regressions in our code, but the tests can have
+ancillary benefits such as documenting interfaces and influencing
+the design of our software.
+
+If you have worked on other Django projects that use unit testing, you
+will probably find familiar patterns in Zulip's code.  This document
+describes how to write tests for the Zulip back end, with a particular
+emphasis on areas where we have either wrapped Django's test framework
+or just done things that are kind of unique in Zulip.
+
+## Running tests
+
+Our tests live in `zerver/tests/`. You can run them with
+`./tools/test-backend`. It generally takes about a minute to run
+the entire test suite.  When you are in iterative mode, you
+can run individual tests or individual modules, following the
+dotted.test.name convention below:
+
+    cd /srv/zulip
+    ./tools/test-backend zerver.tests.tests.WorkerTest
+
+There are many command line options for running Zulip tests, such
+as a `--verbose` option.  The
+best way to learn the options is to use the online help:
+
+    ./tools/test-backend -h
+
+We also have ways to instrument our tests for finding code coverage,
+URL coverage, and slow tests.  Use the `-h` option to discover these
+features.  We also have a `--profile` option to facilitate profiling
+tests.
+
+Another thing to note is that our tests generally "fail fast," i.e. they
+stop at the first sign of trouble.  This is generally a good thing for
+iterative development, but you can override this behavior with the
+`--nonfatal-errors` option.
+
+## How to write tests.
+
+Before you write your first tests of Zulip, it is worthwhile to read
+the rest of this document, and you can also read some of the existing tests
+in `zerver/tests` to get a feel for the patterns we use.
+
+A good practice is to get a "failing test" before you start to implement
+your feature.  First, it is a useful exercise to understand what needs to happen
+in your tests before you write the code, as it can help drive out simple
+design or help you make incremental progress on a large feature.  Second,
+you want to avoid introducing tests that give false positives.  Ensuring
+that a test fails before you implement the feature ensures that if somebody
+accidentally regresses the feature in the future, the test will catch
+the regression.
+
+Another important file to skim is
+[zerver/lib/test_helpers.py](https://github.com/zulip/zulip/blob/master/zerver/lib/test_helpers.py),
+which contains test helpers and our `ZulipTestCase` class.
+
+### Setting up data for tests
+
+All tests start with the same fixture data.  (The tests themselves
+update the database, but they do so inside a transaction that gets
+rolled back after each of the tests complete. For more details on how the
+fixture data gets set up, refer to `tools/setup/generate-fixtures`.)
+
+The fixture data includes a few users that are named after
+Shakesepeare characters, and they are part of the "zulip.com" realm.
+
+Generally, you will also do some explicit data setup of your own. Here
+are a couple useful methods in ZulipTestCase:
+
+- common_subscribe_to_streams
+- send_message
+- subscribe_to_stream
+
+More typically, you will use methods directly from the backend code.
+(This ensures more end-to-end testing, and avoids false positives from
+tests that might not consider ancillary parts of data setup that could
+influence tests results.)
+
+Here are some example action methods that tests may use for data setup:
+
+- check_send_message
+- create_stream_if_needed
+- do_add_subscription
+- do_change_is_admin
+- do_create_user
+- do_make_stream_private
+
+## Zulip Testing Philosophy
+
+If there is one word to describe Zulip's philosophy for writing tests,
+it is probably "flexible."  (Hopefully "thorough" goes without saying.)
+
+When in doubt, unless speed concerns are prohibitive,
+you usually want your tests to be somewhat end-to-end, particularly
+for testing endpoints.
+
+These are some of the testing strategies that you will see in the Zulip
+test suite...
+
+### Endpoint tests
+
+We strive to test all of our URL endpoints.  The vast majority of Zulip
+endpoints support a JSON interface.  Regardless of the interface, an
+endpoint test generally follows this pattern:
+
+- Set up the data.
+- Login with `self.login()` or set up an API key.
+- Use a Zulip test helper to hit the endpoint.
+- Assert that the result was either a success or failure.
+- Check the data that comes back from the endpoint.
+
+Generally, if you are doing endpoint tests, you will want to create a
+test class that is a subclass of `ZulipTestCase`, which will provide
+you helper methods like the following:
+
+- api_auth
+- assert_json_error
+- assert_json_success
+- client_get
+- client_post
+- get_api_key
+- get_streams
+- login
+- send_message
+
+### Library tests
+
+For certain Zulip library functions, especially the ones that are
+not intrinsically tied to Django, we use a classic unit testing
+approach of calling the function and inspecting the results.
+
+For these types of tests, you will often use methods like
+`self.assertEqual()`, `self.assertTrue()`, etc., which come with
+[unittest](https://docs.python.org/3/library/unittest.html#unittest.TestCase)
+via Django.
+
+### Fixture-driven tests
+
+Particularly for testing Zulip's integrations with third party systems,
+we strive to have a highly data-driven approach to testing.  To give a
+specific example, when we test our GitHub integration, the test code
+reads a bunch of sample inputs from a JSON fixture file, feeds them
+to our Github integration code, and then verifies the output against
+expected values from the same JSON fixture file.
+
+Our fixtures live in `zerver/fixtures`.
+
+### Mocks and stubs
+
+We use mocks and stubs for all the typical reasons:
+
+- to more precisely test the target code
+- to stub out calls to third-party services
+- to make it so that you can run your tests on the airplane without wifi
+
+For mocking we generally use the "mock" library and use `mock.patch` as
+a context manager or decorator.  We also take advantage of some context managers
+from Django as well as our own custom helpers.  Here is an example:
+
+
+        with self.settings(RATE_LIMITING=True):
+            with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock:
+                api_result = my_webhook(request)
+
+        self.assertTrue(rate_limit_mock.called)
+
+Follow [this link](settings.html#testing-non-default-settings) for more
+information on the "settings" context manager.
+
+### Template tests
+
+In [zerver/tests/test_templates.py](https://github.com/zulip/zulip/blob/master/zerver/tests/test_templates.py)
+we have a test that renders all of our back end templates with
+a "dummy" context, to make sure the templates don't have obvious
+errors.  (These tests won't catch all types of errors; they are
+just a first line of defense.)
+
+### SQL performance tests
+
+A common class of bug with Django systems is to handle bulk data in
+an inefficient way, where the back end populates objects for join tables
+with a series of individual queries that give O(N) latency.  (The
+remedy is often just to call `select_related()`, but sometimes it
+requires a more subtle restructuring of the code.)
+
+We try to prevent these bugs in our tests by using a context manager
+called `queries_captured()` that captures the SQL queries used by
+the back end during a particular operation.  We make assertions about
+those queries, often simply asserting that the number of queries is
+below some threshold.
+
+### Event-based tests
+
+The Zulip back end has a mechanism where it will fetch initial data
+for a client from the database, and then it will subsequently apply
+some queued up events to that data to the data structure before notifying
+the client.  The `EventsRegisterTest.do_test()` helper helps tests
+verify that the application of those events via apply_events() produces
+the same data structure as performing an action that generates said event.
+
+This is a bit esoteric, but if you read the tests, you will see some of
+the patterns.  You can also learn more about our event system in the
+[new feature tutorial](new-feature-tutorial.html#handle-database-interactions).
+
+### Negative tests
+
+It is important to verify error handling paths for endpoints, particularly
+situations where we need to ensure that we don't return results to clients
+with improper authentication or with limited authorization.  A typical test
+will call the endpoint with either a non-logged in client, an invalid API
+key, or missing input fields.  Then the test will call `assert_json_error()`
+to verify that the endpoint is properly failing.
+
+## Testing considerations
+
+Here are some things to consider when writing new tests:
+
+- **Duplication** We try to avoid excessive duplication in tests.
+If you have several tests repeating the same type of test setup,
+consider making a setUp() method or a test helper.
+
+- **Network independence** Our tests should still work if you don't
+have an internet connection.  For third party clients, you can simulate
+their behavior using fixture data.  For third party servers, you can
+typically simulate their behavior using mocks.
+
+- **Coverage** We have 100% line coverage on several of our backend
+modules.  You can use the `--coverage` option to generate coverage
+reports, and new code should have 100% coverage, which generally requires
+testing not only the "happy path" but also error handling code and
+edge cases.
+

docs/testing-with-node.md

@@ -0,0 +1,132 @@
+# Web frontend unit tests
+
+As an alternative to the black-box whole-app testing, you can unit test
+individual JavaScript files that use the module pattern. For example, to
+test the `foobar.js` file, you would first add the following to the
+bottom of `foobar.js`:
+
+>     if (typeof module !== 'undefined') {
+>         module.exports = foobar;
+>     }
+
+This makes `foobar.js` follow the CommonJS module pattern, so it can be
+required in Node.js, which runs our tests.
+
+Now create `frontend_tests/node_tests/foobar.js`. At the top, require
+the [Node.js assert module](http://nodejs.org/api/assert.html), and the
+module you're testing, like so:
+
+>     var assert = require('assert');
+>     var foobar = require('js/foobar.js');
+
+(If the module you're testing depends on other modules, or modifies
+global state, you need to also read [the next
+section](handling-dependencies_).)
+
+Define and call some tests using the [assert
+module](http://nodejs.org/api/assert.html). Note that for "equal"
+asserts, the *actual* value comes first, the *expected* value second.
+
+>     (function test_somefeature() {
+>         assert.strictEqual(foobar.somefeature('baz'), 'quux');
+>         assert.throws(foobar.somefeature('Invalid Input'));
+>     }());
+
+The test runner (`index.js`) automatically runs all .js files in the
+frontend\_tests/node directory.
+
+## HTML output
+
+The JavaScript unit tests can generate output to be viewed in the
+browser.  The best examples of this are in `frontend_tests/node_tests/templates.js`.
+
+The main use case for this mechanism is to be able to unit test
+templates and see how they are rendered without the complications
+of the surrounding app.  (Obviously, you still need to test the
+app itself!)  The HTML output can also help to debug the unit tests.
+
+Each test calls a method named `write_handlebars_output` after it
+renders a template with similar data.  This API is still evolving,
+but you should be able to look at existing code for patterns.
+
+When you run `tools/test-js-with-node`, it will present you with a
+message like "To see more output, open var/test-js-with-node/index.html."
+Basically, you just need to open the file in the browser.  (If you are
+running a VM, this might require switching to another terminal window
+to launch the `open` command.)
+
+## Coverage reports
+
+You can automatically generate coverage reports for the JavaScript unit
+tests like this:
+
+>     tools/test-js-with-node cover
+
+Then open `coverage/lcov-report/js/index.html` in your browser. Modules
+we don't test *at all* aren't listed in the report, so this tends to
+overstate how good our overall coverage is, but it's accurate for
+individual files. You can also click a filename to see the specific
+statements and branches not tested. 100% branch coverage isn't
+necessarily possible, but getting to at least 80% branch coverage is a
+good goal.
+
+## Handling dependencies in unit tests
+
+The following scheme helps avoid tests leaking globals between each
+other.
+
+First, if you can avoid globals, do it, and the code that is directly
+under test can simply be handled like this:
+
+>     var search = require('js/search_suggestion.js');
+
+For deeper dependencies, you want to categorize each module as follows:
+
+-   Exercise the module's real code for deeper, more realistic testing?
+-   Stub out the module's interface for more control, speed, and
+    isolation?
+-   Do some combination of the above?
+
+For all the modules where you want to run actual code, add a statement
+like the following to the top of your test file:
+
+>     add_dependencies({
+>         _: 'third/underscore/underscore.js',
+>         util: 'js/util.js',
+>         Dict: 'js/dict.js',
+>         Handlebars: 'handlebars',
+>         Filter: 'js/filter.js',
+>         typeahead_helper: 'js/typeahead_helper.js',
+>         stream_data: 'js/stream_data.js',
+>         narrow: 'js/narrow.js'
+>     });
+
+For modules that you want to completely stub out, please use a pattern
+like this:
+
+>     set_global('page_params', {
+>         email: 'bob@zulip.com'
+>     });
+>
+>     // then maybe further down
+>     global.page_params.email = 'alice@zulip.com';
+
+Finally, there's the hybrid situation, where you want to borrow some of
+a module's real functionality but stub out other pieces. Obviously, this
+is a pretty strong smell that the other module might be lacking in
+cohesion, but that code might be outside your jurisdiction. The pattern
+here is this:
+
+>     // Use real versions of parse/unparse
+>     var narrow = require('js/narrow.js');
+>     set_global('narrow', {
+>         parse: narrow.parse,
+>         unparse: narrow.unparse
+>     });
+>
+>     // But later, I want to stub the stream without having to call super-expensive
+>     // real code like narrow.activate().
+>     global.narrow.stream = function () {
+>         return 'office';
+>     };
+

docs/testing.md

@@ -1,22 +1,88 @@
-Testing and writing tests
-=========================
+# Testing and writing tests
 
-Running tests
--------------
+## Overview
 
-To run everything, just use `./tools/test-all`. This runs lint checks,
-web frontend / whole-system blackbox tests, and backend Django tests.
+Zulip has a full test suite that includes many components.  The most
+important components are documented in depth in their own sections:
 
-If you want to run individual parts, see the various commands inside
-that script.
+- [Django](testing-with-django.html): backend Python tests
+- [Casper](testing-with-casper.html): end-to-end UI tests
+- [Node](testing-with-node.html): unit tests for JS front end code
+- [Linters](linters.html)
 
-### Schema and initial data changes
+This document covers more general testing issues, such as how to run the
+entire test suite, how to troubleshoot database issues, how to manually
+test the front end, and how to plan for the future upgrade to Python3.
+
+## Running tests
+
+Zulip tests must be run inside a Zulip development environment; if
+you're using Vagrant, you will need to enter the Vagrant environment
+before running the tests:
+
+```
+vagrant ssh
+cd /srv/zulip
+```
+
+Then, to run the full Zulip test suite, do this:
+```
+./tools/test-all
+```
+
+This runs the linter (`tools/lint-all`) plus all of our test suites;
+they can all be run separately (just read `tools/test-all` to see
+them).  You can also run individual tests which can save you a lot of
+time debugging a test failure, e.g.:
+
+```
+./tools/lint-all # Runs all the linters in parallel
+./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube
+./tools/test-js-with-casper 09-navigation.js
+./tools/test-js-with-node utils.js
+```
+The above setup instructions include the first-time setup of test
+databases, but you may need to rebuild the test database occasionally
+if you're working on new database migrations.  To do this, run:
+
+```
+./tools/do-destroy-rebuild-test-database
+```
+
+### Possible testing issues
+
+- When running the test suite, if you get an error like this:
+
+  ```
+      sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not   exist
+      LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
+                                                                   ^
+  ```
+
+  … then you need to install tsearch-extras, described
+  above. Afterwards, re-run the `init*-db` and the
+  `do-destroy-rebuild*-database` scripts.
+
+- When building the development environment using Vagrant and the LXC
+  provider, if you encounter permissions errors, you may need to
+  `chown -R 1000:$(whoami) /path/to/zulip` on the host before running
+  `vagrant up` in order to ensure that the synced directory has the
+  correct owner during provision. This issue will arise if you run `id
+  username` on the host where `username` is the user running Vagrant
+  and the output is anything but 1000.
+  This seems to be caused by Vagrant behavior; for more information,
+  see [the vagrant-lxc FAQ entry about shared folder permissions][lxc-sf].
+
+[lxc-sf]: https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner
+
+
+## Schema and initial data changes
 
 If you change the database schema or change the initial test data, you
 have to regenerate the pristine test database by running
 `tools/do-destroy-rebuild-test-database`.
 
-### Wiping the test databases
+## Wiping the test databases
 
 You should first try running: `tools/do-destroy-rebuild-test-database`
 
@@ -28,7 +94,7 @@ If that fails you should try to do:
 
 and then run `tools/do-destroy-rebuild-test-database`
 
-#### Recreating the postgres cluster
+### Recreating the postgres cluster
 
 > **warning**
 >
@@ -42,239 +108,7 @@ it. On Ubuntu:
     sudo pg_dropcluster --stop 9.1 main
     sudo pg_createcluster --locale=en_US.utf8 --start 9.1 main
 
-### Backend Django tests
-
-These live in `zerver/tests/tests.py` and `zerver/tests/test_*.py`. Run
-them with `tools/test-backend`.
-
-### Web frontend black-box casperjs tests
-
-These live in `frontend_tests/casper_tests/`. This is a "black box"
-test; we load the frontend in a real (headless) browser, from a real dev
-server, and simulate UI interactions like sending messages, narrowing,
-etc.
-
-Since this is interacting with a real dev server, it can catch backend
-bugs as well.
-
-You can run this with `./tools/test-js-with-casper` or as
-`./tools/test-js-with-casper 05-settings.js` to run a single test file
-from `frontend_tests/casper_tests/`.
-
-#### Debugging Casper.JS
-
-Casper.js (via PhantomJS) has support for remote debugging. However, it
-is not perfect. Here are some steps for using it and gotchas you might
-want to know.
-
-To turn on remote debugging, pass `--remote-debug` to the
-`./frontend_tests/run-casper` script. This will run the tests with port
-`7777` open for remote debugging. You can now connect to
-`localhost:7777` in a Webkit browser. Somewhat recent versions of Chrome
-or Safari might be required.
-
--   When connecting to the remote debugger, you will see a list of
-    pages, probably 2. One page called `about:blank` is the headless
-    page in which the CasperJS test itself is actually running in. This
-    is where your test code is.
--   The other page, probably `localhost:9981`, is the Zulip page that
-    the test is testing---that is, the page running our app that our
-    test is exercising.
-
-Since the tests are now running, you can open the `about:blank` page,
-switch to the Scripts tab, and open the running `0x-foo.js` test. If you
-set a breakpoint and it is hit, the inspector will pause and you can do
-your normal JS debugging. You can also put breakpoints in the Zulip
-webpage itself if you wish to inspect the state of the Zulip frontend.
-
-You can also check the screenshots of failed tests at `/tmp/casper-failure*.png`.
-
-If you need to use print debugging in casper, you can do using
-`casper.log`; see <http://docs.casperjs.org/en/latest/logging.html> for
-details.
-
-An additional debugging technique is to enable verbose mode in the
-Casper tests; you can do this by adding to the top of the relevant test
-file the following:
-
->     var casper = require('casper').create({
->        verbose: true,
->        logLevel: "debug"
->     });
-
-This can sometimes give insight into exactly what's happening.
-
-### Web frontend unit tests
-
-As an alternative to the black-box whole-app testing, you can unit test
-individual JavaScript files that use the module pattern. For example, to
-test the `foobar.js` file, you would first add the following to the
-bottom of `foobar.js`:
-
->     if (typeof module !== 'undefined') {
->         module.exports = foobar;
->     }
-
-This makes `foobar.js` follow the CommonJS module pattern, so it can be
-required in Node.js, which runs our tests.
-
-Now create `frontend_tests/node_tests/foobar.js`. At the top, require
-the [Node.js assert module](http://nodejs.org/api/assert.html), and the
-module you're testing, like so:
-
->     var assert = require('assert');
->     var foobar = require('js/foobar.js');
-
-(If the module you're testing depends on other modules, or modifies
-global state, you need to also read [the next
-section](handling-dependencies_).)
-
-Define and call some tests using the [assert
-module](http://nodejs.org/api/assert.html). Note that for "equal"
-asserts, the *actual* value comes first, the *expected* value second.
-
->     (function test_somefeature() {
->         assert.strictEqual(foobar.somefeature('baz'), 'quux');
->         assert.throws(foobar.somefeature('Invalid Input'));
->     }());
-
-The test runner (index.js) automatically runs all .js files in the
-frontend\_tests/node directory.
-
-#### Coverage reports
-
-You can automatically generate coverage reports for the JavaScript unit
-tests. To do so, install istanbul:
-
->     sudo npm install -g istanbul
-
-And run test-js-with-node with the 'cover' parameter:
-
->     tools/test-js-with-node cover
-
-Then open `coverage/lcov-report/js/index.html` in your browser. Modules
-we don't test *at all* aren't listed in the report, so this tends to
-overstate how good our overall coverage is, but it's accurate for
-individual files. You can also click a filename to see the specific
-statements and branches not tested. 100% branch coverage isn't
-necessarily possible, but getting to at least 80% branch coverage is a
-good goal.
-
-Writing tests
--------------
-
-### Writing Casper tests
-
-Probably the easiest way to learn how to write Casper tests is to study
-some of the existing test files. There are a few tips that can be useful
-for writing Casper tests in addition to the debugging notes below:
-
--   Run just the file containing your new tests as described above to
-    have a fast debugging cycle.
--   With frontend tests in general, it's very important to write your
-    code to wait for the right events. Before essentially every action
-    you take on the page, you'll want to use `waitForSelector`,
-    `waitUntilVisible`, or a similar function to make sure the page or
-    elemant is ready before you interact with it. For instance, if you
-    want to click a button that you can select via `#btn-submit`, and
-    then check that it causes `success-elt` to appear, you'll want to
-    write something like:
-
-        casper.waitForSelector("#btn-submit", function () {
-           casper.click('#btn-submit')
-           casper.test.assertExists("#success-elt");
-         });
-
-    This will ensure that the element is present before the interaction
-    is attempted. The various wait functions supported in Casper are
-    documented in the Casper here:
-    <http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector>
-    and the various assert statements available are documented here:
-    <http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype>
--   Casper uses CSS3 selectors; you can often save time by testing and
-    debugging your selectors on the relevant page of the Zulip
-    development app in the Chrome javascript console by using e.g.
-    `$$("#settings-dropdown")`.
--   The test suite uses a smaller set of default user accounts and other
-    data initialized in the database than the development environment;
-    to see what differs check out the section related to
-    `options["test_suite"]` in
-    `zilencer/management/commands/populate_db.py`.
--   Casper effectively runs your test file in two phases -- first it
-    runs the code in the test file, which for most test files will just
-    collect a series of steps (each being a `casper.then` or
-    `casper.wait...` call). Then, usually at the end of the test file,
-    you'll have a `casper.run` call which actually runs that series of
-    steps. This means that if you write code in your test file outside a
-    `casper.then` or `casper.wait...` method, it will actually run
-    before all the Casper test steps that are declared in the file,
-    which can lead to confusing failures where the new code you write in
-    between two `casper.then` blocks actually runs before either of
-    them. See this for more details about how Casper works:
-    <http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work>
-
-### Handling dependencies in unit tests
-
-The following scheme helps avoid tests leaking globals between each
-other.
-
-First, if you can avoid globals, do it, and the code that is directly
-under test can simply be handled like this:
-
->     var search = require('js/search_suggestion.js');
-
-For deeper dependencies, you want to categorize each module as follows:
-
--   Exercise the module's real code for deeper, more realistic testing?
--   Stub out the module's interface for more control, speed, and
-    isolation?
--   Do some combination of the above?
-
-For all the modules where you want to run actual code, add a statement
-like the following to the top of your test file:
-
->     add_dependencies({
->         _: 'third/underscore/underscore.js',
->         util: 'js/util.js',
->         Dict: 'js/dict.js',
->         Handlebars: 'handlebars',
->         Filter: 'js/filter.js',
->         typeahead_helper: 'js/typeahead_helper.js',
->         stream_data: 'js/stream_data.js',
->         narrow: 'js/narrow.js'
->     });
-
-For modules that you want to completely stub out, please use a pattern
-like this:
-
->     set_global('page_params', {
->         email: 'bob@zulip.com'
->     });
->
->     // then maybe further down
->     global.page_params.email = 'alice@zulip.com';
-
-Finally, there's the hybrid situation, where you want to borrow some of
-a module's real functionality but stub out other pieces. Obviously, this
-is a pretty strong smell that the other module might be lacking in
-cohesion, but that code might be outside your jurisdiction. The pattern
-here is this:
-
->     // Use real versions of parse/unparse
->     var narrow = require('js/narrow.js');
->     set_global('narrow', {
->         parse: narrow.parse,
->         unparse: narrow.unparse
->     });
->
->     // But later, I want to stub the stream without having to call super-expensive
->     // real code like narrow.activate().
->     global.narrow.stream = function () {
->         return 'office';
->     };
-
-Manual testing (local app + web browser)
-----------------------------------------
+## Manual testing (local app + web browser)
 
 ### Clearing the manual testing database
 
@@ -290,10 +124,10 @@ return to a clean state for testing.
 
 ### JavaScript manual testing
 
-debug.js has some tools for profiling Javascript code, including:
+`debug.js` has some tools for profiling JavaScript code, including:
 
 -   \`print\_elapsed\_time\`: Wrap a function with it to print the time
-    that function takes to the javascript console.
+    that function takes to the JavaScript console.
 -   \`IterationProfiler\`: Profile part of looping constructs (like a
     for loop or \$.each). You mark sections of the iteration body and
     the IterationProfiler will sum the costs of those sections over all
@@ -305,25 +139,26 @@ Chrome's is a sampling profiler while Firebug's is an instrumenting
 profiler. Using them both can be helpful because they provide different
 information.
 
-Python 3 Compatibility
-----------------------
+## Python 3 Compatibility
 
 Zulip is working on supporting Python 3, and all new code in Zulip
 should be Python 2+3 compatible. We have converted most of the codebase
 to be compatible with Python 3 using a suite of 2to3 conversion tools
 and some manual work. In order to avoid regressions in that
-compatibility as we continue to develop new features in zulip, we have a
-special tool, tools/check-py3, which checks all code for Python 3
+compatibility as we continue to develop new features in Zulip, we have a
+special tool, `tools/check-py3`, which checks all code for Python 3
 syntactic compatibility by running a subset of the automated migration
-tools and checking if they trigger any changes. tools/check-py3 is run
-automatically in Zulip's Travis CI tests to avoid any regressions, but
-is not included in test-all since it is quite slow.
+tools and checking if they trigger any changes. `tools/check-py3` is run
+automatically in Zulip's Travis CI tests (in the 'static-analysis'
+build) to avoid any regressions, but is not included in `test-all` since
+it is quite slow.
 
-To run tooks/check-py3, you need to install the modernize and future
-python packages (which are included in requirements/py3k.txt, which
-itself is included in requirements/dev.txt, so you probably already
-have these packages installed).
+To run `tools/check-py3`, you need to install the `modernize` and
+`future` Python packages (which are included in
+`requirements/py3k.txt`, which itself is included in
+`requirements/dev.txt`, so you probably already have these packages
+installed).
 
-To run check-py3 on just the python files in a particular directory, you
-can change the current working directory (e.g. cd zerver/) and run
-check-py3 from there.
+To run `check-py3` on just the Python files in a particular directory, you
+can change the current working directory (e.g. `cd zerver/`) and run
+`check-py3` from there.

docs/translating.md

@@ -18,17 +18,71 @@ email to zulip-core@googlegroups.com when you request to join the
 project or add a language so that we can be sure to accept your
 request to contribute.
 
-## Translation Tags
+## Setting Default Language in Zulip
 
-All user-facing text in the Zulip UI should be generated by a HTML
+Zulip allows you to set the default language through the settings
+page under 'Display Settings' section.
+
+## Translation Resource Files
+
+All the translation magic happens through resource files which hold
+the translated text. Backend resource files are located at
+`static/locale/<lang_code>/LC_MESSAGES/django.po`, while frontend
+resource files are located at
+`static/locale/<lang_code>/translations.json`.  These files are
+uploaded to Transifex using `tx push`, where they can be
+translated. Once translated, they are downloaded back into the
+codebase using `tx pull`.
+
+## Transifex Config
+
+The config file that maps the resources from Zulip to Transifex is
+located at `.tx/config`. Django  recognizes `zh_CN` instead of `zh-HANS`
+for simplified Chinese language (this is fixed in Django 1.9). This
+idiosyncrasy is also handled in the Transifex config file.
+
+## Translation Process
+
+The end-to-end process to get the translations working is as follows:
+
+1. Mark the strings for translations (see sections for backend and
+   frontend translations for details on this).
+
+2. Create JSON formatted [resource][] files using the `python manage makemessages`
+   command. This command will create a resource file called `translations.json`
+   for frontend and `django.po` for backend for every language under
+   `static/locale`. The location for frontend resource file can be
+   changed by passing an argument to the command (see the help for the
+   command for further details). However, make sure that the location
+   is publicly accessible since frontend files are loaded through XHR
+   in the frontend which will only work with publicly accessible resources.
+
+   The `makemessages` command is idempotent in that:
+
+   - It will only delete singular keys in the resource file when they
+     are no longer used in Zulip code.
+   - It will only delete plural keys (see below for the documentation
+     on plural translations) when the corresponding singular key is
+     absent.
+   - It will not override the value of a singular key if that value
+     contains a translated text.
+
+3. Upload the resource files to Transifex using the `tx push -s -a`
+   command.
+
+4. Download the updated resource files from Transifex using the
+   `tx pull -a` command. This command will download the resource files
+   from Transifex and replace your local resource files with them.
+
+## Backend Translations
+
+All user-facing text in the Zulip UI should be generated by an HTML
 template so that it can be translated.
 
 Zulip uses two types of templates: backend templates (powered by the
 [Jinja2][] template engine, though the original [Django][] template
 engine is still supported) and frontend templates (powered by
-[Handlebars][]).  At present, the frontend templates don't support
-translation (though we're working on fixing this!), so the rest of
-this discussion will be about the backend templates.
+[Handlebars][]).
 
 To mark a string for translation in the Jinja2 and Django template
 engines, you can use the `_()` function in the templates like this:
@@ -39,7 +93,7 @@ engines, you can use the `_()` function in the templates like this:
 
 If a string contains both a literal string component and variables,
 you can use a block translation, which makes use of placeholders to
-help translators to translated an entire sentence.  To translate a
+help translators to translate an entire sentence.  To translate a
 block, Jinja2 uses the [trans][] tag while Django uses the
 [blocktrans][] tag.  So rather than writing something ugly and
 confusing for translators like this:
@@ -58,8 +112,8 @@ You can instead use:
 {% blocktrans %}This string will have {{ value }} inside.{% endblocktrans %}
 ```
 
-Zulip expects all the error messages to be translatable.  To ensure
-this, the error message passed to `json_error` and `JsonableError`
+Zulip expects all the error messages to be translatable as well.  To
+ensure this, the error message passed to `json_error` and `JsonableError`
 should always be a literal string enclosed by `_()` function, e.g:
 
 ```
@@ -71,18 +125,76 @@ To ensure we always internationalize our JSON errors messages, the
 Zulip linter (`tools/lint-all`) checks for correct usage.
 
 ## Frontend Translations
-The first step in translating the frontend is to create the translation
-files using `python manage makemessages`. This command will create
-translation files under `static/locale`, the location can be changed by
-passing an argument to the command, however make sure that the location is
-publicly accessible since these files are loaded through XHR in the
-frontend which will only work with publicly accessible resources.
 
-The second step is to upload the translatable strings to Transifex using
-`tx push -s -a`.
+Zulip uses the [i18next][] library for frontend translations. There
+are two types of files in Zulip frontend which can hold translatable
+strings, JavaScript code files and Handlebar templates. To mark a
+string translatable in JavaScript files pass it to the `i18n.t` function.
 
-The final step is to get the translated files from Transifex using
-`tx pull -a`.
+```
+i18n.t('English Text', context);
+i18n.t('English text with a __variable__', {'variable': 'Variable value'});
+```
+
+Note: In the second example above, instead of enclosing the variable with
+handlebars, `{{ }}`, we enclose it with `__` because we need to
+differentiate the variable from the Handlebar tags. The symbol which is
+used to enclose the variables can be changed in `/static/js/src/main.js`.
+
+`i18next` also supports plural translations. To support plurals make
+sure your resource file contatins the related keys:
+
+```
+{
+    "en": {
+        "translation": {
+            "key": "item",
+            "key_plural": "items",
+            "keyWithCount": "__count__ item",
+            "keyWithCount_plural": "__count__ items"
+        }
+    }
+}
+```
+
+With this resource you can show plurals like this:
+
+```
+i18n.t('key', {count: 0}); // output: 'items'
+i18n.t('key', {count: 1}); // output: 'item'
+i18n.t('key', {count: 5}); // output: 'items'
+i18n.t('key', {count: 100}); // output: 'items'
+i18n.t('keyWithCount', {count: 0}); // output: '0 items'
+i18n.t('keyWithCount', {count: 1}); // output: '1 item'
+i18n.t('keyWithCount', {count: 5}); // output: '5 items'
+i18n.t('keyWithCount', {count: 100}); // output: '100 items'
+```
+
+For further reading on plurals, read the [official] documentation.
+
+To mark the strings as translatable in the Handlebar templates, Zulip
+registers two Handlebar [helpers][]. The syntax for simple strings is:
+
+```
+{{t 'English Text' }}
+```
+
+The syntax for block strings or strings containing variables is:
+
+```
+{{tr context}}
+    Block of English text.
+{{/tr}}
+
+var context = {'variable': 'variable value'};
+{{tr context}}
+    Block of English text with a __variable__.
+{{/tr}}
+```
+
+The rules for plurals are same as for JavaScript files. You just have
+to declare the appropriate keys in the resource file and then include the
+`count` in the context.
 
 
 [Django]: https://docs.djangoproject.com/en/1.9/topics/templates/#the-django-template-language
@@ -90,6 +202,10 @@ The final step is to get the translated files from Transifex using
 [Handlebars]: http://handlebarsjs.com/
 [trans]: http://jinja.pocoo.org/docs/dev/templates/#i18n
 [blocktrans]: https://docs.djangoproject.com/en/1.8/topics/i18n/translation/#std:templatetag-blocktrans
+[i18next]: http://i18next.com
+[official]: http://i18next.com/translate/pluralSimple/
+[helpers]: http://handlebarsjs.com/block_helpers.html
+[resource]: http://i18next.com/translate/
 
 ## Testing Translations
 
@@ -100,7 +216,7 @@ Django figures out the effective language by going through the
 following steps:
 
 1. It looks for the language code in the url.
-2. It loooks for the LANGUGE_SESSION_KEY key in the current user's
+2. It looks for the LANGUGE_SESSION_KEY key in the current user's
 session.
 3. It looks for the cookie named 'django_language'. You can set a
 different name through LANGUAGE_COOKIE_NAME setting.

docs/using-dev-environment.md

@@ -0,0 +1,52 @@
+Using the Development Environment
+=================================
+
+Once the development environment is running, you can visit
+<http://localhost:9991/> in your browser.  By default, the development
+server homepage just shows a list of the users that exist on the
+server and you can login as any of them by just clicking on a user.
+This setup saves time for the common case where you want to test
+something other than the login process; to test the login process
+you'll want to change `AUTHENTICATION_BACKENDS` in the not-PRODUCTION
+case of `zproject/settings.py` from zproject.backends.DevAuthBackend
+to use the auth method(s) you'd like to test.
+
+While developing, it's helpful to watch the `run-dev.py` console
+output, which will show any errors your Zulip development server
+encounters.
+
+When you make a change, here's a guide for what you need to do in
+order to see your change take effect in Development:
+
+* If you change JavaScript, CSS, or Jinja2 backend templates (under
+`templates/`), you'll just need to reload the browser window to see
+changes take effect.  The Handlebars frontend HTML templates
+(`static/templates`) are automatically recompiled by the
+`tools/compile-handlebars-templates` job, which runs as part of
+`tools/run-dev.py`.
+
+* If you change Python code used by the the main Django/Tornado server
+processes, these services are run on top of Django's [manage.py
+runserver][django-runserver] which will automatically restart the
+Zulip Django and Tornado servers whenever you save changes to Python
+code.  You can watch this happen in the `run-dev.py` console to make
+sure the backend has reloaded.
+
+* The Python queue workers will also automatically restart when you
+save changes.  However, you may need to ctrl-C and then restart
+`run-dev.py` manually if a queue worker has crashed.
+
+* If you change the database schema, you'll need to use the standard
+Django migrations process to create and then run your migrations; see
+the [new feature tutorial][new-feature-tutorial] for an example.
+Additionally you should check out the [detailed testing
+docs][testing-docs] for how to run the tests properly after doing a
+migration.
+
+(In production, everything runs under supervisord and thus will
+restart if it crashes, and `upgrade-zulip` will take care of running
+migrations and then cleanly restaring the server for you).
+
+[django-runserver]: https://docs.djangoproject.com/en/1.8/ref/django-admin/#runserver-port-or-address-port
+[new-feature-tutorial]: http://zulip.readthedocs.io/en/latest/new-feature-tutorial.html
+[testing-docs]: http://zulip.readthedocs.io/en/latest/testing.html

docs/version-control.md

@@ -0,0 +1,115 @@
+# Version control
+
+## Commit Discipline
+
+We follow the Git project's own commit discipline practice of "Each
+commit is a minimal coherent idea". This discipline takes a bit of work,
+but it makes it much easier for code reviewers to spot bugs, and
+makes the commit history a much more useful resource for developers
+trying to understand why the code works the way it does, which also
+helps a lot in preventing bugs.
+
+Coherency requirements for any commit:
+
+-   It should pass tests (so test updates needed by a change should be
+    in the same commit as the original change, not a separate "fix the
+    tests that were broken by the last commit" commit).
+-   It should be safe to deploy individually, or comment in detail in
+    the commit message as to why it isn't (maybe with a [manual] tag).
+    So implementing a new API endpoint in one commit and then adding the
+    security checks in a future commit should be avoided -- the security
+    checks should be there from the beginning.
+-   Error handling should generally be included along with the code that
+    might trigger the error.
+-   TODO comments should be in the commit that introduces the issue or
+    functionality with further work required.
+
+When you should be minimal:
+
+-   Significant refactorings should be done in a separate commit from
+    functional changes.
+-   Moving code from one file to another should be done in a separate
+    commits from functional changes or even refactoring within a file.
+-   2 different refactorings should be done in different commits.
+-   2 different features should be done in different commits.
+-   If you find yourself writing a commit message that reads like a list
+    of somewhat dissimilar things that you did, you probably should have
+    just done 2 commits.
+
+When not to be overly minimal:
+
+-   For completely new features, you don't necessarily need to split out
+    new commits for each little subfeature of the new feature. E.g. if
+    you're writing a new tool from scratch, it's fine to have the
+    initial tool have plenty of options/features without doing separate
+    commits for each one. That said, reviewing a 2000-line giant blob of
+    new code isn't fun, so please be thoughtful about submitting things
+    in reviewable units.
+-   Don't bother to split back end commits from front end commits, even
+    though the backend can often be coherent on its own.
+
+Other considerations:
+
+-   Overly fine commits are easily squashed, but not vice versa, so err
+    toward small commits, and the code reviewer can advise on squashing.
+-   If a commit you write doesn't pass tests, you should usually fix
+    that by amending the commit to fix the bug, not writing a new "fix
+    tests" commit on top of it.
+
+Zulip expects you to structure the commits in your pull requests to form
+a clean history before we will merge them; it's best to write your
+commits following these guidelines in the first place, but if you don't,
+you can always fix your history using git rebase -i.
+
+It can take some practice to get used to writing your commits with a
+clean history so that you don't spend much time doing interactive
+rebases. For example, often you'll start adding a feature, and discover
+you need to a refactoring partway through writing the feature. When that
+happens, we recommend stashing your partial feature, do the refactoring,
+commit it, and then finish implementing your feature.
+
+## Commit Messages
+
+-   The first line of commit messages should be written in the
+    imperative and be kept relatively short while concisely explaining
+    what the commit does. For example:
+
+Bad:
+
+    bugfix
+    gather_subscriptions was broken
+    fix bug #234.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Use present-tense action verbs in your commit messages.
+
+Bad:
+
+    Fixing gather_subscriptions throwing an exception when given bad input.
+    Fixed gather_subscriptions throwing an exception when given bad input.
+
+Good:
+
+    Fix gather_subscriptions throwing an exception when given bad input.
+
+-   Please use a complete sentence in the summary, ending with a period.
+-   The rest of the commit message should be written in full prose and
+    explain why and how the change was made. If the commit makes
+    performance improvements, you should generally include some rough
+    benchmarks showing that it actually improves the performance.
+-   When you fix a GitHub issue, [mark that you've fixed the issue in
+    your commit
+    message](https://help.github.com/articles/closing-issues-via-commit-messages/)
+    so that the issue is automatically closed when your code is merged.
+    Zulip's preferred style for this is to have the final paragraph of
+    the commit message read e.g. "Fixes: \#123."
+-   Any paragraph content in the commit message should be line-wrapped
+    to less than 76 characters per line, so that your commit message
+    will be reasonably readable in git log in a normal terminal.
+-   In your commit message, you should describe any manual testing you
+    did in addition to running the automated tests, and any aspects of
+    the commit that you think are questionable and you'd like special
+    attention applied to.

docs/writing-views.md

@@ -0,0 +1,366 @@
+# Writing views in Zulip
+
+## What this covers
+
+This page documents how views work in Zulip. You may want to read the
+[new feature tutorial](https://zulip.readthedocs.io/en/latest/new-feature-tutorial.html)
+or the [integration guide](https://zulip.readthedocs.io/en/latest/integration-guide.html),
+and treat this as a reference.
+
+If you have experience with Django, much of this will be familiar, but
+you may want to read about how REST requests are dispatched, and how
+request authentication works.
+
+This document supplements the [new feature tutorial](https://zulip.readthedocs.io/en/latest/new-feature-tutorial.html)
+and the [testing](https://zulip.readthedocs.io/en/latest/testing.html)
+documentation.
+
+## What is a view?
+
+A view in Zulip is everything that helps implement a server endpoint.
+Every path that the Zulip server supports (doesn't show a 404 page
+for) is a view. The obvious ones are those you can visit in your
+browser, for example
+[/integrations](https://zulipchat.com/integrations/), which shows the
+integration documentation. These paths show up in the address bar of
+the browser. There are other views that are only seen by software,
+namely the API views. They are used to build the various clients that
+Zulip has, namely the web client (which is also used by the desktop
+client) and the mobile clients.
+
+## Modifying urls.py
+
+A view is anything with an entry in the appropriate urls.py, usually
+`zproject/urls.py`. Zulip views either serve HTML (pages for browsers)
+or JSON (data for Zulip clients on all platforms, custom bots, and
+integrations).
+
+The format of the URL patterns in Django is [documented
+here](https://docs.djangoproject.com/en/1.8/topics/http/urls/), and
+the Zulip specific details for these are discussed in detail in the
+[life of a request doc](life-of-a-request.html#options).
+
+We have two Zulip-specific conventions we use for internationalization and for
+our REST API, respectively.
+
+## Writing human-readable views
+
+If you're writing a new page for the website, make sure to add it
+to `i18n_urls` in `zproject/urls.py`
+
+```diff
+     i18n_urls = [
+     ...
++    url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html')),
++    url(r'^postcards/$', 'zerver.views.postcards'),
+]
+```
+
+As an example, if a request comes in for Spanish, language code `es`,
+the server path will be something like: `es/features/`.
+
+### Decorators used for webpage views
+
+This section documents a few simple decorators that we use for webpage
+views, as an introduction to view decorators.
+
+`require_post`:
+
+```py
+
+@require_post
+def accounts_register(request):
+    # type: (HttpRequest) -> HttpResponse
+```
+
+This decorator ensures that the requst was a POST--here, we're
+checking that the registration submission page is requested with a
+post, and inside the function, we'll check the form data. If you
+request this page with GET, you'll get a HTTP 405 METHOD NOT ALLOWED
+error.
+
+`zulip_login_required`:
+
+This decorator verifies that the browser is logged in (i.e. has a
+valid session cookie) before providing the view for this route, or
+redirects the browser to a login page. This is used in the root path
+(`/`) of the website for the web client. If a request comes from a
+browser without a valid session cookie, they are redirected to a login
+page.  It is a small fork of Django's
+[login_required](https://docs.djangoproject.com/en/1.8/topics/auth/default/#django.contrib.auth.decorators.login_required),
+adding a few extra checks specific to Zulip.
+
+```py
+@zulip_login_required
+def home(request):
+    # type: (HttpRequest) -> HttpResponse
+```
+
+### Writing a template
+
+Templates for the main website are found in
+[templates/zerver](https://github.com/zulip/zulip/blob/master/templates/zerver).
+
+
+## Writing API REST endpoints
+
+These are code-parseable views that take x-www-form-urlencoded or JSON
+request bodies, and return JSON-string responses.  Almost all Zulip
+view code is in the implementations of API REST endpoints.
+
+The REST API does authentication of the user through `rest_dispatch`,
+which is documented in detail at [zerver/lib/rest.py](https://github.com/zulip/zulip/blob/master/zerver/lib/rest.py).
+This method will authenticate the user either through a session token
+from a cookie on the browser, or from a base64 encoded `email:api-key`
+string given via HTTP Basic Auth for API clients.
+
+``` py
+>>> import requests
+>>> r = requests.get('https://api.github.com/user', auth=('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV'))
+>>> r.status_code
+-> 200
+```
+
+### Request variables
+
+Most API views will have some arguments that are passed as part of the
+request to control the behavior of the view.  In any well-engineered
+view, you need to write code to parse and validate that the arguments
+exist and have the correct form.  For many applications, this leads to
+one of serveral bad outcomes:
+
+* The code isn't written, so arguments aren't validated, leading to
+  bugs and confusing error messages for users of the API.
+* Every function starts with a long list of semi-redundant validation
+  code, usually with highly inconsistent error messages.
+* Every view function comes with another function that does the
+  validation that has the problems from the last bullet point.
+
+In Zulip, we solve this problem with a the special decorator called
+`has_request_variables` which allows a developer to declare the
+arguments a view function takes and validate their types all within
+the `def` line of the function.  We like this framework because we
+have found it makes the validation code compact, readable, and
+conveniently located in the same place as the method it is validating
+arguments for.
+
+Here's an example:
+
+``` py
+from zerver.decorator import has_request_variables, REQ, JsonableError, \
+    require_realm_admin
+
+@require_realm_admin
+@has_request_variables
+def create_user_backend(request, user_profile, email=REQ(), password=REQ(),
+                        full_name=REQ(), short_name=REQ()):
+    # ... code here
+```
+
+You will notice the special `REQ()` in the keyword arguments to
+`create_user_backend`.  `has_request_variables` parses the declared
+keyword arguments of the decorated function, and for each that has an
+instance of `REQ` as the default value, it extracts the HTTP parameter
+with that name from the request, parses it as JSON, and passes it to
+the function.  It will return an nicely JSON formatted HTTP 400 error
+in the event that an argument is missing, doesn't parse as JSON, or
+otherwise is invalid.
+
+`require_realm_admin` is another decorator which checks the
+authorization of the given `user_profile` to make sure it belongs to a
+realm administrator (and thus has permission to create a user); we
+show it here primarily to show how `has_request_variables` should be
+the inner decorator.
+
+The implementation of `has_request_variables` is documented in detail
+in
+[zerver/lib/request.py](https://github.com/zulip/zulip/blob/master/zerver/lib/request.py))
+
+REQ also helps us with request variable validation. For example:
+
+* `msg_ids = REQ(validator=check_list(check_int))` will check that the
+  `msg_ids` HTTP parameter is a list of integers, marshalled as JSON,
+  and pass it into the function as the `msg_ids` Python keyword
+  argument.
+
+* `streams_raw = REQ("subscriptions",
+  validator=check_list(check_string))` will check that the
+  "subscriptions" HTTP parameter is a list of strings, marshalled as
+  JSON, and pass it into the function with the Python keyword argument
+  `streams_raw`.
+
+* `message_id=REQ(converter=to_non_negative_int)` will check that the
+  `message_id` HTTP parameter is a string containing a non-negative
+  integer (`converter` differs from `validator` in that it does not
+  automatically marshall the input from JSON).
+
+See [zerver/lib/validator.py](https://github.com/zulip/zulip/blob/master/zerver/lib/validator.py) for more validators and their documentation.
+
+### Deciding which HTTP verb to use
+
+When writing a new API view, you should writing a view to do just one
+type of thing.  Usually that's either a read or write operation.
+
+If you're reading data, GET is the best option. Other read-only verbs
+are HEAD, which should be used for testing if a resource is available to
+be read with GET, without the expense of the full GET. OPTIONS is also
+read-only, and used by clients to determine which HTTP verbs are
+available for a given path. This isn't something you need to write, as
+it happens automatically in the implementation of `rest_dispatch`--see
+[zerver/lib/rest.py](https://github.com/zulip/zulip/blob/master/zerver/lib/rest.py)
+for more.
+
+If you're creating new data, try to figure out if the thing you are
+creating is uniquely identifiable. For example, if you're creating a
+user, there's only one user per email. If you can find a unique ID,
+you should use PUT for the view. If you want to create the data multiple
+times for multiple requests (for example, requesting the send_message
+view multiple times with the same content should send multiple
+messages), you should use POST.
+
+If you're updating existing data, use PATCH.
+
+If you're removing data, use DELETE.
+
+### Idempotency
+
+When writing a new API endpoint, with the exception of things like
+sending messages, requests should be safe to repeat, without impacting
+the state of the server. This is *idempotency*.
+
+You will often want to return an error if a request to change
+something would do nothing because the state is already as desired, to
+make debugging Zulip clients easier. This means that the response for
+repeated requests may not be the same, but the repeated requests won't
+change the server more than once or cause unwanted side effects.
+
+### Making changes to the database
+
+If the view does any modification to the database, that change is done
+in a helper function in `zerver/lib/actions.py`.  Those functions are
+responsible for doing a complete update to the state of the server,
+which often entails both updating the database and sending any events
+to notify clients about the state change.  When possible, we prefer to
+design a clean boundary between the view function and the actions
+function is such that all user input validation happens in the view
+code (i.e. all 400 type errors are thrown there), and the actions code
+is responsible for atomically executing the change (this is usually
+signalled by having the actions function have a name starting with
+`do_`.  So in most cases, errors in an actions function will be the
+result of an operational problem (e.g. lost connection to the
+database) and lead to a 500 error.  If an actions function is
+responsible for validation as well, it should have a name starting
+with `check_`.
+
+For example, in [zerver/views/__init__.py](https://github.com/zulip/zulip/blob/master/zerver/views/__init__.py):
+
+```py
+@require_realm_admin
+@has_request_variables
+def update_realm(request, user_profile, name=REQ(validator=check_string, default=None), ...)):
+    # type: (HttpRequest, UserProfile, ...) -> HttpResponse
+    realm = user_profile.realm
+    data = {} # type: Dict[str, Any]
+    if name is not None and realm.name != name:
+        do_set_realm_name(realm, name)
+        data['name'] = 'updated'
+```
+
+and in [zerver/lib/actions.py](https://github.com/zulip/zulip/blob/master/zerver/lib/actions.py):
+
+```py
+def do_set_realm_name(realm, name):
+    # type: (Realm, text_type) -> None
+    realm.name = name
+    realm.save(update_fields=['name'])
+    event = dict(
+        type="realm",
+        op="update",
+        property='name',
+        value=name,
+    )
+    send_event(event, active_user_ids(realm))
+```
+
+`realm.save()` actually saves the changes to the realm to the
+database, and `send_event` sends the event to active clients belonging
+to the provided list of users (in this case, all altive users in the
+Zulip realm).
+
+### Calling from the web application
+
+You should always use channel.<method> to make an `HTTP <method>` call
+to the Zulip JSON API. As an example, in
+[static/js/admin.js](https://github.com/zulip/zulip/blob/master/static/js/admin.js)
+
+```js
+var url = "/json/realm";
+var data = {
+    name: JSON.stringify(new_name),
+}
+channel.patch({
+    url: url,
+    data: data,
+    success: function (response_data) {
+        if (response_data.name !== undefined) {
+            ui.report_success(i18n.t("Name changed!"), name_status);
+        }
+        ...
+```
+
+### Calling from an API client
+
+Here's how you might manually make a call from python:
+
+```py
+payload = {'name': new_name}
+
+# email and API key
+api_auth = ('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV')
+
+r = requests.patch(SERVER_URL + 'api/v1/realm',
+                   data=json.dumps(payload),
+                   auth=api_auth,
+                  )
+```
+
+This is simply an illustration; we recommend making use of the [Zulip
+Python API bindings](https://www.zulipchat.com/api) since they provide
+a nice interface for accessing the API.
+
+## Legacy endpoints used by the web client
+
+New features should conform the REST API style. The legacy, web-only
+endpoints can't effectively enforce usage of a browser, so they aren't
+preferable from a security perspective, and it is generally a good idea
+to make your feature available to other clients, especially the mobile
+clients.
+
+These endpoints make use of some older authentication decorators,
+`authenticated_json_api_view`, `authenticated_json_post_view`, and
+`authenticated_json_view`, so you may see them in the code.
+
+## Webhook integration endpoints
+
+Webhooks are called by other services, often to send a message as part
+of those services' integrations. They are most often POST requests, and
+often there is very little you can customize about them. Usually you can
+expect that the webhook for a service will allow specification for the
+target server for the webhook, and an API key.
+
+If the webhook does not have an option to provide a bot email, use the
+`api_key_only_webhook_view` decorator, to fill in the `user_profile` and
+`client` fields of a request:
+
+``` py
+@api_key_only_webhook_view('PagerDuty')
+@has_request_variables
+def api_pagerduty_webhook(request, user_profile, client,
+                          payload=REQ(argument_type='body'),
+                          stream=REQ(default='pagerduty'),
+                          topic=REQ(default=None)):
+```
+The `client` will be the result of `get_client("ZulipPagerDutyWebhook")`
+in this example.
+
+

frontend_tests/casper_lib/common.js

@@ -2,7 +2,7 @@ var common = (function () {
 
 var exports = {};
 
-var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
+var test_credentials = require('../../var/casper/test_credentials.js').test_credentials;
 
 function timestamp() {
     return new Date().getTime();
@@ -32,7 +32,7 @@ exports.initialize_casper = function (viewport) {
     // casper.start has been called.
 
     // Set default viewport size to something reasonable
-    casper.page.viewportSize = viewport || {width: 1280, height: 768};
+    casper.page.viewportSize = viewport || {width: 1280, height: 1024};
 
     // Fail if we get a JavaScript error in the page's context.
     // Based on the example at http://phantomjs.org/release-1.5.html
@@ -83,7 +83,7 @@ exports.then_log_in = function (credentials) {
 };
 
 exports.start_and_log_in = function (credentials, viewport) {
-    casper.start('http://localhost:9981/accounts/login', function () {
+    casper.start('http://127.0.0.1:9981/accounts/login', function () {
         exports.initialize_casper(viewport);
         log_in(credentials);
     });

frontend_tests/casper_tests/00-realm-creation.js

@@ -0,0 +1,91 @@
+var common = require('../casper_lib/common.js').common;
+
+var email = 'alice@test.example.com';
+var domain = 'test.example.com';
+var organization_name = 'Awesome Organization';
+
+casper.start('http://127.0.0.1:9981/create_realm/');
+
+casper.then(function () {
+    // Submit the email for realm creation
+    this.waitForSelector('form[action^="/create_realm/"]', function () {
+        this.fill('form[action^="/create_realm/"]', {
+            email: email
+        }, true);
+    });
+    // Make sure confirmation email is send
+    this.waitWhileSelector('form[action^="/create_realm/"]', function () {
+         var regex = new RegExp('^http:\/\/[^\/]+\/accounts\/send_confirm\/' + email);
+         this.test.assertUrlMatch(regex, 'Confirmation mail send');
+    });
+});
+
+// Special endpoint enabled only during tests for extracting confirmation key
+casper.thenOpen('http://127.0.0.1:9981/confirmation_key/');
+
+// Open the confirmation URL
+casper.then(function () {
+    var confirmation_key = JSON.parse(this.getPageContent()).confirmation_key;
+    var confirmation_url = 'http://127.0.0.1:9981/accounts/do_confirm/' + confirmation_key;
+    this.thenOpen(confirmation_url);
+});
+
+// Make sure the realm creation page is loaded correctly
+casper.then(function () {
+    this.waitForSelector('.pitch', function () {
+        this.test.assertSelectorHasText('.pitch', "You're almost there. We just need you to do one last thing.");
+    });
+
+    this.waitForSelector('.controls.fakecontrol', function () {
+        this.test.assertSelectorHasText('.controls.fakecontrol', email);
+    });
+
+    this.waitForSelector('label[for=id_team_name]', function () {
+        this.test.assertSelectorHasText('label[for=id_team_name]', 'Organization name');
+    });
+});
+
+casper.then(function () {
+    this.waitForSelector('form[action^="/accounts/register/"]', function () {
+        this.fill('form[action^="/accounts/register/"]', {
+            full_name: 'Alice',
+            realm_name: organization_name,
+            password: 'password',
+            terms: true
+        }, true);
+    });
+
+    this.waitWhileSelector('form[action^="/accounts/register/"]', function () {
+        casper.test.assertUrlMatch('http://127.0.0.1:9981/invite/', 'Invite more users page loaded');
+    });
+});
+
+// Tests for invite more users page
+casper.then(function () {
+    this.waitForSelector('.app-main.portico-page-container', function () {
+        this.test.assertSelectorHasText('.app-main.portico-page-container', "You're the first one here!");
+    });
+
+    this.waitForSelector('.invite_row', function () {
+        this.test.assertSelectorHasText('.invite_row', domain);
+    });
+
+    this.waitForSelector('#submit_invitation', function () {
+        this.click('#submit_invitation');
+    });
+
+    this.waitWhileSelector('#submit_invitation', function () {
+        this.test.assertUrlMatch('http://127.0.0.1:9981/', 'Realm created and logged in');
+    });
+});
+
+casper.then(function () {
+    // The user is logged in to the newly created realm
+    this.test.assertTitle('home - ' + organization_name + ' - Zulip');
+});
+
+common.then_log_out();
+
+casper.run(function () {
+    casper.test.done();
+});

frontend_tests/casper_tests/01-login.js

@@ -1,7 +1,7 @@
 var common = require('../casper_lib/common.js').common;
 
 // Start of test script.
-casper.start('http://localhost:9981/', common.initialize_casper);
+casper.start('http://127.0.0.1:9981/', common.initialize_casper);
 
 casper.then(function () {
     casper.test.assertHttpStatus(302);

frontend_tests/casper_tests/03-narrow.js

@@ -2,7 +2,7 @@ var common = require('../casper_lib/common.js').common;
 
 common.start_and_log_in();
 
-// We could use the messages sent by 01-site.js, but we want to
+// We could use the messages sent by 02-site.js, but we want to
 // make sure each test file can be run individually (which the
 // 'run' script provides for).
 
@@ -193,6 +193,28 @@ function search_and_check(str, item, check, narrow_title) {
     un_narrow();
 }
 
+function search_silent_user(str, item) {
+    common.select_item_via_typeahead('#search_query', str, item);
+    casper.waitUntilVisible('#silent_user', function () {
+        casper.test.info("Empty feed for silent user visible.");
+        var expected_message = "\n        You haven't received any messages sent by this user yet!"+
+                                "\n      ";
+        this.test.assertEquals(casper.fetchText('#silent_user'), expected_message);
+    });
+    un_narrow();
+}
+
+function search_non_existing_user(str, item) {
+    common.select_item_via_typeahead('#search_query', str, item);
+    casper.waitUntilVisible('#non_existing_user', function () {
+        casper.test.info("Empty feed for non existing user visible.");
+        var expected_message = "\n        This user does not exist!"+
+                                "\n      ";
+        this.test.assertEquals(casper.fetchText('#non_existing_user'), expected_message);
+    });
+    un_narrow();
+}
+
 casper.waitUntilVisible('#zhome', expect_home);
 
 // Test stream / recipient autocomplete in the search bar
@@ -209,6 +231,9 @@ search_and_check('stream:Verona subject:frontend+test', 'Narrow', expect_stream_
 search_and_check('subject:frontend+test', 'Narrow', expect_subject,
                 'home - Zulip Dev - Zulip');
 
+search_silent_user('sender:emailgateway@zulip.com', 'Narrow');
+search_non_existing_user('sender:dummyuser@zulip.com', 'Narrow');
+
 // Narrow by clicking the left sidebar.
 casper.then(function () {
     casper.test.info('Narrowing with left sidebar');
@@ -222,6 +247,66 @@ casper.then(check_narrow_title('private - Zulip Dev - Zulip'));
 un_narrow();
 
 
+// Make sure stream search filters the stream list
+casper.then(function () {
+    casper.test.info('Search streams using left sidebar');
+});
+
+casper.then(function () {
+    casper.test.assertExists('.stream-list-filter.notdisplayed', 'Stream filter box not visible initially');
+});
+
+casper.thenClick('#streams_header .sidebar-title');
+
+casper.then(function () {
+    casper.test.assertDoesntExist('.stream-list-filter.notdisplayed', 'Stream filter box visible after click');
+});
+
+casper.then(function () {
+    casper.test.assertExists('#stream_filters [data-name="Denmark"]', 'Original stream list contains Denmark');
+    casper.test.assertExists('#stream_filters [data-name="Scotland"]', 'Original stream list contains Scotland');
+    casper.test.assertExists('#stream_filters [data-name="Verona"]', 'Original stream list contains Verona');
+});
+
+// We search for the beginning of "Verona", not case sensitive
+casper.then(function () {
+    casper.evaluate(function () {
+        $('.stream-list-filter').expectOne()
+            .focus()
+            .val('ver')
+            .trigger($.Event('input'));
+    });
+});
+casper.then(function () {
+    casper.test.assertDoesntExist('#stream_filters [data-name="Denmark"]', 'Filtered stream list does not contain Denmark');
+    casper.test.assertDoesntExist('#stream_filters [data-name="Scotland"]', 'Filtered stream list does not contain Scotland');
+    casper.test.assertExists('#stream_filters [data-name="Verona"]', 'Filtered stream list does contain Verona');
+});
+
+// Clearing the list should give us back all the streams in the list
+casper.then(function () {
+    casper.evaluate(function () {
+        $('.stream-list-filter').expectOne()
+            .focus()
+            .val('')
+            .trigger($.Event('input'));
+    });
+});
+casper.then(function () {
+    casper.test.assertExists('#stream_filters [data-name="Denmark"]', 'Restored stream list contains Denmark');
+    casper.test.assertExists('#stream_filters [data-name="Scotland"]', 'Restored stream list contains Scotland');
+    casper.test.assertExists('#stream_filters [data-name="Verona"]', 'Restored stream list contains Verona');
+});
+
+casper.thenClick('#streams_header .sidebar-title');
+
+casper.then(function () {
+    casper.test.assertExists('.stream-list-filter.notdisplayed', 'Stream filter box not visible after second click');
+});
+
+un_narrow();
+
+
 common.then_log_out();
 
 // Run the above queued actions.

frontend_tests/casper_tests/05-settings.js

@@ -1,114 +0,0 @@
-var common = require('../casper_lib/common.js').common;
-var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
-
-common.start_and_log_in();
-
-var form_sel = 'form[action^="/json/settings/change"]';
-
-casper.then(function () {
-    casper.test.info('Settings page');
-    casper.click('a[href^="#settings"]');
-});
-
-casper.waitForSelector("#settings-change-box", function () {
-    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#settings/, 'URL suggests we are on settings page');
-    casper.test.assertExists('#settings.tab-pane.active', 'Settings page is active');
-
-    casper.test.assertNotVisible("#old_password");
-
-    casper.click(".change_password_button");
-});
-
-casper.waitUntilVisible("#old_password", function () {
-    casper.test.assertVisible("#old_password");
-    casper.test.assertVisible("#new_password");
-    casper.test.assertVisible("#confirm_password");
-
-    casper.test.assertEqual(casper.getFormValues(form_sel).full_name, "Iago");
-
-    casper.fill(form_sel, {
-        "full_name": "IagoNew",
-        "old_password": test_credentials.default_user.password,
-        "new_password": "qwertyuiop",
-        "confirm_password": "qwertyuiop"
-    });
-    casper.click('input[name="change_settings"]');
-});
-
-casper.waitUntilVisible('#settings-status', function () {
-    casper.test.assertSelectorHasText('#settings-status', 'Updated settings!');
-
-    casper.click('#api_key_button');
-});
-
-casper.waitUntilVisible('#get_api_key_password', function () {
-    casper.fill('form[action^="/json/fetch_api_key"]', {'password':'qwertyuiop'});
-    casper.click('input[name="view_api_key"]');
-});
-
-casper.waitUntilVisible('#api_key_value', function () {
-    casper.test.assertMatch(casper.fetchText('#api_key_value'), /[a-zA-Z0-9]{32}/, "Looks like an API key");
-
-    // Change it all back so the next test can still log in
-    casper.fill(form_sel, {
-        "full_name": "Iago",
-        "old_password": "qwertyuiop",
-        "new_password": test_credentials.default_user.password,
-        "confirm_password": test_credentials.default_user.password
-    });
-    casper.click('input[name="change_settings"]');
-});
-
-casper.waitUntilVisible('#settings-status', function () {
-    casper.test.assertSelectorHasText('#settings-status', 'Updated settings!');
-});
-
-
-casper.then(function create_bot() {
-    casper.test.info('Filling out the create bot form');
-
-    casper.fill('#create_bot_form',{
-        bot_name: 'Bot 1',
-        bot_short_name: '1',
-        bot_default_sending_stream: 'Denmark',
-        bot_default_events_register_stream: 'Rome'
-    });
-
-    casper.test.info('Submiting the create bot form');
-    casper.click('#create_bot_button');
-});
-
-casper.waitUntilVisible('.open_edit_bot_form[data-email="1-bot@zulip.com"]', function open_edit_bot_form() {
-    casper.test.info('Opening edit bot form');
-    casper.click('.open_edit_bot_form[data-email="1-bot@zulip.com"]');
-});
-
-casper.waitUntilVisible('.edit_bot_form[data-email="1-bot@zulip.com"]', function test_edit_bot_form_values() {
-    var form_sel = '.edit_bot_form[data-email="1-bot@zulip.com"]';
-    casper.test.info('Testing edit bot form values');
-
-//     casper.test.assertEqual(
-//         common.get_form_field_value(form_sel + ' [name=bot_name]'),
-//         'Bot 1'
-//     );
-//     casper.test.assertEqual(
-//         common.get_form_field_value(form_sel + ' [name=bot_default_sending_stream]'),
-//         'Denmark'
-//     );
-//     casper.test.assertEqual(
-//         common.get_form_field_value(form_sel + ' [name=bot_default_events_register_stream]'),
-//         'Rome'
-//     );
-    casper.test.assertEqual(
-        common.get_form_field_value(form_sel + ' [name=bot_name]'),
-        'Bot 1'
-    );
-});
-
-// TODO: test the "Declare Zulip Bankruptcy option"
-
-common.then_log_out();
-
-casper.run(function () {
-    casper.test.done();
-});

frontend_tests/casper_tests/05-subscriptions.js

@@ -10,13 +10,48 @@ casper.then(function () {
     // subscriptions need to load; if they have *any* subs,
     // the word "Unsubscribe" will appear
 });
-casper.waitForText('Subscribed', function () {
+casper.waitForSelector('.sub_unsub_button.subscribed-button', function () {
     casper.test.assertTextExists('Subscribed', 'Initial subscriptions loaded');
     casper.fill('form#add_new_subscription', {stream_name: 'Waseemio'});
     casper.click('form#add_new_subscription input.btn');
 });
 casper.waitForText('Waseemio', function () {
     casper.test.assertTextExists('Create stream Waseemio', 'Modal for specifying new stream users');
+});
+casper.then(function () {
+    casper.test.assertExists('#user-checkboxes [for="cordelia@zulip.com"]', 'Original user list contains Cordelia');
+    casper.test.assertExists('#user-checkboxes [for="hamlet@zulip.com"]', 'Original user list contains King Hamlet');
+});
+casper.then(function () {
+    casper.test.info("Filtering user list with keyword 'cor'");
+    casper.fill('form#stream_creation_form', {user_list_filter: 'cor'});
+});
+casper.then(function () {
+    casper.test.assertEquals(casper.visible('#user-checkboxes [for="cordelia@zulip.com"]'),
+                             true,
+                             "Cordelia is visible"
+    );
+    casper.test.assertEquals(casper.visible('#user-checkboxes [for="hamlet@zulip.com"]'),
+                             false,
+                             "King Hamlet is not visible"
+    );
+});
+casper.then(function () {
+    casper.test.info("Clearing user filter search box");
+    casper.fill('form#stream_creation_form', {user_list_filter: ''});
+});
+casper.then(function () {
+    casper.test.assertEquals(casper.visible('#user-checkboxes [for="cordelia@zulip.com"]'),
+                             true,
+                             "Cordelia is visible again"
+    );
+    casper.test.assertEquals(casper.visible('#user-checkboxes [for="hamlet@zulip.com"]'),
+                             true,
+                             "King Hamlet is visible again"
+    );
+});
+casper.then(function () {
+    casper.test.assertTextExists('Create stream Waseemio', 'Create a new stream');
     casper.click('form#stream_creation_form button.btn.btn-primary');
 });
 casper.waitFor(function () {

frontend_tests/casper_tests/06-settings.js

@@ -0,0 +1,198 @@
+var common = require('../casper_lib/common.js').common;
+var test_credentials = require('../../var/casper/test_credentials.js').test_credentials;
+
+common.start_and_log_in();
+
+var form_sel = 'form[action^="/json/settings/change"]';
+
+casper.waitForSelector('a[href^="#settings"]', function () {
+    casper.test.info('Settings page');
+    casper.click('a[href^="#settings"]');
+});
+
+casper.waitForSelector("#settings-change-box", function () {
+    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#settings/, 'URL suggests we are on settings page');
+    casper.test.assertExists('#settings.tab-pane.active', 'Settings page is active');
+
+    casper.test.assertNotVisible("#old_password");
+
+    casper.click(".change_password_button");
+});
+
+casper.waitUntilVisible("#old_password", function () {
+    casper.waitForResource("zxcvbn.js", function () {
+        casper.test.assertVisible("#old_password");
+        casper.test.assertVisible("#new_password");
+        casper.test.assertVisible("#confirm_password");
+
+        casper.test.assertEqual(casper.getFormValues(form_sel).full_name, "Iago");
+
+        casper.fill(form_sel, {
+            "full_name": "IagoNew",
+            "old_password": test_credentials.default_user.password,
+            "new_password": "qwertyuiop",
+            "confirm_password": "qwertyuiop"
+        });
+        casper.click('input[name="change_settings"]');
+    });
+});
+
+casper.waitUntilVisible('#settings-status', function () {
+    casper.test.assertSelectorHasText('#settings-status', 'Updated settings!');
+
+    casper.click('#api_key_button');
+});
+
+casper.waitUntilVisible('#get_api_key_password', function () {
+    casper.fill('form[action^="/json/fetch_api_key"]', {'password':'qwertyuiop'});
+    casper.click('input[name="view_api_key"]');
+});
+
+casper.waitUntilVisible('#api_key_value', function () {
+    casper.test.assertMatch(casper.fetchText('#api_key_value'), /[a-zA-Z0-9]{32}/, "Looks like an API key");
+
+    // Change it all back so the next test can still log in
+    casper.fill(form_sel, {
+        "full_name": "Iago",
+        "old_password": "qwertyuiop",
+        "new_password": test_credentials.default_user.password,
+        "confirm_password": test_credentials.default_user.password
+    });
+    casper.click('input[name="change_settings"]');
+});
+
+casper.waitUntilVisible('#settings-status', function () {
+    casper.test.assertSelectorHasText('#settings-status', 'Updated settings!');
+});
+
+
+casper.then(function create_bot() {
+    casper.test.info('Filling out the create bot form');
+
+    casper.fill('#create_bot_form',{
+        bot_name: 'Bot 1',
+        bot_short_name: '1',
+        bot_default_sending_stream: 'Denmark',
+        bot_default_events_register_stream: 'Rome'
+    });
+
+    casper.test.info('Submiting the create bot form');
+    casper.click('#create_bot_button');
+});
+
+casper.waitUntilVisible('.open_edit_bot_form[data-email="1-bot@zulip.com"]', function open_edit_bot_form() {
+    casper.test.info('Opening edit bot form');
+    casper.click('.open_edit_bot_form[data-email="1-bot@zulip.com"]');
+});
+
+casper.waitUntilVisible('.edit_bot_form[data-email="1-bot@zulip.com"]', function test_edit_bot_form_values() {
+    var form_sel = '.edit_bot_form[data-email="1-bot@zulip.com"]';
+    casper.test.info('Testing edit bot form values');
+
+//     casper.test.assertEqual(
+//         common.get_form_field_value(form_sel + ' [name=bot_name]'),
+//         'Bot 1'
+//     );
+//     casper.test.assertEqual(
+//         common.get_form_field_value(form_sel + ' [name=bot_default_sending_stream]'),
+//         'Denmark'
+//     );
+//     casper.test.assertEqual(
+//         common.get_form_field_value(form_sel + ' [name=bot_default_events_register_stream]'),
+//         'Rome'
+//     );
+    casper.test.assertEqual(
+        common.get_form_field_value(form_sel + ' [name=bot_name]'),
+        'Bot 1'
+    );
+});
+
+/*
+   This test needs a modification. As it stands now, it will cause a race
+   condition with all subsequent tests which access the UserProfile object
+   this test modifies. Currently, if we modify alert words, we don't get
+   any notification from the server, issue reported at
+   https://github.com/zulip/zulip/issues/1269. Consequently, we can't wait
+   on any condition to avoid the race condition.
+
+casper.waitForSelector('#create_alert_word_form', function () {
+    casper.test.info('Attempting to submit an empty alert word');
+    casper.click('#create_alert_word_button');
+    casper.test.info('Checking that an error is displayed');
+    casper.test.assertVisible('#empty_alert_word_error');
+
+    casper.test.info('Closing the error message');
+    casper.click('.close-empty-alert-word-error');
+    casper.test.info('Checking the error is hidden');
+    casper.test.assertNotVisible('#empty_alert_word_error');
+
+    casper.test.info('Filling out the alert word input');
+    casper.sendKeys('#create_alert_word_name', 'some phrase');
+    casper.click('#create_alert_word_button');
+
+    casper.test.info('Checking that an element was created');
+    casper.test.assertExists('div.alert-word-information-box');
+    casper.test.assertSelectorHasText('span.value', 'some phrase');
+
+    casper.test.info('Deleting element');
+    casper.click('button.remove-alert-word');
+    casper.test.info('Checking that the element was deleted');
+    casper.test.assertDoesntExist('div.alert-word-information-box');
+});
+*/
+
+casper.then(function change_default_language() {
+    casper.test.info('Changing the default language');
+    casper.waitForSelector('#default_language');
+});
+
+casper.thenClick('#default_language');
+
+casper.waitUntilVisible('#default_language_modal');
+
+casper.thenClick('a[data-code="zh_CN"]');
+
+casper.waitUntilVisible('#display-settings-status', function () {
+    casper.test.assertSelectorHasText('#display-settings-status', '简体中文 is now the default language');
+    casper.test.info("Reloading the page.");
+    casper.reload();
+});
+
+casper.waitForSelector("#default_language", function () {
+    casper.test.info("Checking if we are on Chinese page.");
+    casper.test.assertEvalEquals(function () {
+        return $('#default_language_name').text();
+    }, '简体中文');
+    casper.test.info("Opening German page through i18n url.");
+});
+
+casper.thenOpen('http://127.0.0.1:9981/de/#settings');
+
+casper.waitForSelector("#settings-change-box", function check_url_preference() {
+    casper.test.info("Checking the i18n url language precedence.");
+    casper.test.assertEvalEquals(function () {
+        return document.documentElement.lang;
+    }, 'de');
+    casper.test.info("Changing language back to English.");
+});
+
+casper.thenClick('#default_language');
+
+casper.waitUntilVisible('#default_language_modal');
+
+casper.thenClick('a[data-code="en"]');
+
+/*
+ * Changing the language back to English so that subsequent tests pass.
+ */
+casper.waitUntilVisible('#display-settings-status', function () {
+    casper.test.assertSelectorHasText('#display-settings-status', 'English is now the default language');
+});
+
+// TODO: test the "Declare Zulip Bankruptcy option"
+
+common.then_log_out();
+
+casper.run(function () {
+    casper.test.done();
+});

frontend_tests/casper_tests/08-edit.js

@@ -43,7 +43,7 @@ casper.then(function () {
     });
 });
 
-casper.waitForSelector(".selected_message .message_edit_notice", function () {
+casper.waitWhileVisible("textarea.message_edit_content", function () {
     casper.test.assertSelectorHasText(".last_message .message_content", "test edited");
 });
 
@@ -67,7 +67,7 @@ casper.then(function () {
     });
 });
 
-casper.waitForSelector(".selected_message .message_edit_notice", function () {
+casper.waitWhileVisible("textarea.message_edit_content", function () {
     casper.test.assertSelectorHasText(".last_message .sender-status", "test edited one line with me");
 });
 
@@ -88,7 +88,7 @@ casper.then(function () {
     });
 });
 
-casper.waitForSelector(".private-message .message_edit_notice", function () {
+casper.waitWhileVisible("textarea.message_edit_content", function () {
     casper.test.assertSelectorHasText(".last_message .message_content", "test edited pm");
 });
 

frontend_tests/casper_tests/10-admin.js

@@ -0,0 +1,517 @@
+var common = require('../casper_lib/common.js').common;
+var test_credentials = require('../../var/casper/test_credentials.js').test_credentials;
+var stream_name = "Scotland";
+
+common.start_and_log_in();
+
+casper.then(function () {
+    casper.test.info('Administration page');
+    casper.click('a[href^="#administration"]');
+});
+
+casper.waitForSelector('#administration.tab-pane.active', function () {
+    casper.test.info('Administration page is active');
+    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#administration/, 'URL suggests we are on administration page');
+});
+
+// Test only admins may create streams Setting
+casper.waitForSelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]', function () {
+    casper.click('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]');
+    casper.click('form.admin-realm-form input.btn');
+
+});
+
+casper.then(function () {
+    // Test setting was activated
+    casper.waitUntilVisible('#admin-realm-create-stream-by-admins-only-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-create-stream-by-admins-only-status', 'Only Admins may now create new streams!');
+        casper.test.assertEval(function () {
+            return document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked;
+        }, 'Only admins may create streams Setting activated');
+    });
+});
+
+casper.then(function () {
+    // Leave the page and return
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#subscriptions"]');
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+});
+
+casper.waitForSelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]', function () {
+    // Test Setting was saved
+    casper.test.assertEval(function () {
+        return document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked;
+    }, 'Only admins may create streams Setting saved');
+
+    // Deactivate setting
+    casper.click('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]');
+    casper.click('form.admin-realm-form input.btn');
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-create-stream-by-admins-only-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-create-stream-by-admins-only-status', 'Any user may now create new streams!');
+        casper.test.assertEval(function () {
+            return !(document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked);
+        }, 'Only admins may create streams Setting deactivated');
+    });
+});
+
+// Test user deactivation and reactivation
+casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
+    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
+    casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
+    casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
+    casper.click('#do_deactivate_user_button');
+});
+
+casper.then(function () {
+    casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"].deactivated_user', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
+        casper.click('.user_row[id="user_cordelia@zulip.com"] .reactivate');
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]:not(.deactivated_user)', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    });
+});
+
+casper.then(function () {
+    // Test Deactivated users section of admin page
+    casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+        casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
+        casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
+        casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
+        casper.click('#do_deactivate_user_button');
+    });
+});
+
+casper.then(function () {
+    // Leave the page and return
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#subscriptions"]');
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+
+    casper.test.assertSelectorHasText("#administration a[aria-controls='deactivated-users']", "Deactivated Users");
+    casper.click("#administration a[aria-controls='deactivated-users']");
+
+
+    casper.waitForSelector('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] .reactivate', function () {
+        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
+        casper.click('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] .reactivate');
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] button:not(.reactivate)', function () {
+        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
+    });
+
+    casper.test.assertSelectorHasText("#administration a[aria-controls='organization']", "Organization");
+    casper.click("#administration a[aria-controls='organization']");
+});
+
+casper.then(function () {
+    // Test bot deactivation and reactivation
+    casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
+        casper.click('.user_row[id="user_new-user-bot@zulip.com"] .deactivate');
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"].deactivated_user', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Reactivate');
+        casper.click('.user_row[id="user_new-user-bot@zulip.com"] .reactivate');
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]:not(.deactivated_user)', function () {
+        casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
+    });
+});
+
+casper.then(function () {
+    // Test custom realm emoji
+    casper.waitForSelector('.admin-emoji-form', function () {
+        casper.fill('form.admin-emoji-form', {
+            'name': 'MouseFace',
+            'url': 'http://127.0.0.1:9991/static/images/integrations/logos/jenkins.png'
+        });
+        casper.click('form.admin-emoji-form input.btn');
+    });
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('div#admin-emoji-status', function () {
+        casper.test.assertSelectorHasText('div#admin-emoji-status', 'Custom emoji added!');
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('.emoji_row', function () {
+        casper.test.assertSelectorHasText('.emoji_row .emoji_name', 'MouseFace');
+        casper.test.assertExists('.emoji_row img[src="http://127.0.0.1:9991/static/images/integrations/logos/jenkins.png"]');
+        casper.click('.emoji_row button.delete');
+    });
+});
+
+casper.then(function () {
+    casper.waitWhileSelector('.emoji_row', function () {
+        casper.test.assertDoesntExist('.emoji_row');
+    });
+});
+
+function get_suggestions(str) {
+    casper.then(function () {
+        casper.evaluate(function (str) {
+            $('.create_default_stream')
+            .focus()
+            .val(str)
+            .trigger($.Event('keyup', { which: 0 }));
+        }, str);
+    });
+}
+
+function select_from_suggestions(item) {
+    casper.then(function () {
+        casper.evaluate(function (item) {
+            var tah = $('.create_default_stream').data().typeahead;
+            tah.mouseenter({
+                currentTarget: $('.typeahead:visible li:contains("'+item+'")')[0]
+            });
+            tah.select();
+        }, {item: item});
+    });
+}
+
+// Test default stream creation and addition
+casper.then(function () {
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#subscriptions"]');
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+    // It matches with all the stream names which has 'O' as a substring (Rome, Scotland, Verona etc).
+    // I used 'O' to make sure that it works even if there are multiple suggestions.
+    // Capital 'O' is used instead of small 'o' to make sure that the suggestions are not case sensitive.
+    get_suggestions("O");
+    select_from_suggestions(stream_name);
+    casper.waitForSelector('.default_stream_row[id='+stream_name+']', function () {
+        casper.test.assertSelectorHasText('.default_stream_row[id='+stream_name+'] .default_stream_name', stream_name);
+    });
+});
+
+casper.then(function () {
+    casper.waitForSelector('.default_stream_row[id='+stream_name+']', function () {
+        casper.test.assertSelectorHasText('.default_stream_row[id='+stream_name+'] .default_stream_name', stream_name);
+        casper.click('.default_stream_row[id='+stream_name+'] button.remove-default-stream');
+    });
+});
+
+casper.then(function () {
+    casper.waitWhileSelector('.default_stream_row[id='+stream_name+']', function () {
+        casper.test.assertDoesntExist('.default_stream_row[id='+stream_name+']');
+    });
+});
+
+// TODO: Test stream deletion
+
+// Test turning message editing off and on
+// go to home page
+casper.then(function () {
+    casper.click('.global-filter[data-name="home"]');
+});
+
+// send two messages
+common.then_send_message('stream', {
+    stream:  'Verona',
+    subject: 'edits',
+    content: 'test editing 1'
+});
+common.then_send_message('stream', {
+    stream:  'Verona',
+    subject: 'edits',
+    content: 'test editing 2'
+});
+casper.waitForText("test editing 1");
+casper.waitForText("test editing 2");
+
+// wait for message to be sent
+casper.waitFor(function () {
+    return casper.evaluate(function () {
+        return current_msg_list.last().local_id === undefined;
+    });
+});
+
+// edit the last message just sent
+casper.then(function () {
+    casper.evaluate(function () {
+        var msg = $('#zhome .message_row:last');
+        msg.find('.info').click();
+        $('.popover_edit_message').click();
+    });
+});
+
+casper.waitForSelector(".message_edit_content", function () {
+    casper.evaluate(function () {
+        var msg = $('#zhome .message_row:last');
+        msg.find('.message_edit_content').val("test edited");
+        msg.find('.message_edit_save').click();
+    });
+});
+
+casper.then(function () {
+    // check that the message was indeed edited
+    casper.waitWhileVisible("textarea.message_edit_content", function () {
+        casper.test.assertSelectorHasText(".last_message .message_content", "test edited");
+    });
+});
+
+// Commented out due to Issue #1243
+// // edit the same message, but don't hit save this time
+// casper.then(function () {
+//     casper.evaluate(function () {
+//         var msg = $('#zhome .message_row:last');
+//         msg.find('.info').click();
+//         $('.popover_edit_message').click();
+//     });
+// });
+// casper.waitForSelector(".message_edit_content", function () {
+//     casper.evaluate(function () {
+//         var msg = $('#zhome .message_row:last');
+//         msg.find('.message_edit_content').val("test RE-edited");
+//     });
+// });
+
+// go to admin page
+casper.then(function () {
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+});
+
+// deactivate "allow message editing"
+casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+    casper.click('input[type="checkbox"][id="id_realm_allow_message_editing"]');
+    casper.click('form.admin-realm-form input.btn');
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can no longer edit their past messages!');
+        casper.test.assertEval(function () {
+            return !(document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked);
+        }, 'Allow message editing Setting de-activated');
+    });
+});
+
+// go back to home page
+casper.then(function () {
+    casper.click('.global-filter[data-name="home"]');
+});
+
+// Commented out due to Issue #1243
+// // try to save the half-finished edit
+// casper.waitForSelector('.message_table', function () {
+//     casper.then(function () {
+//         casper.evaluate(function () {
+//             var msg = $('#zhome .message_row:last');
+//             msg.find('.message_edit_save').click();
+//         });
+//     });
+// });
+
+// // make sure we get the right error message, and that the message hasn't actually changed
+// casper.waitForSelector("div.edit_error", function () {
+//     casper.test.assertSelectorHasText('div.edit_error', 'Error saving edit: Your organization has turned off message editing.');
+//     casper.test.assertSelectorHasText(".last_message .message_content", "test edited");
+// });
+
+// Check that edit link no longer appears in the popover menu
+// TODO: also check that the edit icon no longer appears next to the message
+casper.then(function () {
+    casper.waitForSelector('.message_row');
+    // Note that this could have a false positive, e.g. if all the messages aren't
+    // loaded yet. See Issue #1243
+    casper.evaluate(function () {
+        var msg = $('#zhome .message_row:last');
+        msg.find('.info').click();
+    });
+    casper.test.assertDoesntExist('.popover_edit_message');
+    casper.evaluate(function () {
+        var msg = $('#zhome .message_row:last');
+        msg.find('.info').click();
+    });
+});
+
+// go back to admin page, and reactivate "allow message editing"
+casper.then(function () {
+    casper.click('#settings-dropdown');
+    casper.click('a[href^="#administration"]');
+});
+casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+    casper.click('input[type="checkbox"][id="id_realm_allow_message_editing"]');
+    casper.click('form.admin-realm-form input.btn');
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can now edit topics for all their messages, and the content of messages which are less than 10 minutes old.');
+        casper.test.assertEval(function () {
+            return document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked;
+        }, 'Allow message editing Setting re-activated');
+    });
+});
+
+// Commented out due to Issue #1243
+// go back home
+// casper.then(function () {
+//     casper.click('.global-filter[data-name="home"]');
+// });
+
+// // save our edit
+// casper.waitForSelector('.message_table', function () {
+//     casper.then(function () {
+//         casper.evaluate(function () {
+//             var msg = $('#zhome .message_row:last');
+//             msg.find('.message_edit_save').click();
+//         });
+//     });
+// });
+
+// // check that edit went through
+// casper.waitWhileVisible("textarea.message_edit_content", function () {
+//     casper.test.assertSelectorHasText(".last_message .message_content", "test RE-edited");
+// });
+
+// check that the edit link reappears in popover menu
+// TODO check for edit icon next to message on hover
+// casper.then(function () {
+//     casper.evaluate(function () {
+//         var msg = $('#zhome .message_row:last');
+//         msg.find('.info').click();
+//     });
+//     casper.test.assertExists('.popover_edit_message');
+//     casper.evaluate(function () {
+//         var msg = $('#zhome .message_row:last');
+//         msg.find('.info').click();
+//     });
+// });
+
+// go to admin page
+casper.then(function () {
+    casper.test.info('Administration page');
+    casper.click('a[href^="#administration"]');
+    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#administration/, 'URL suggests we are on administration page');
+    casper.test.assertExists('#administration.tab-pane.active', 'Administration page is active');
+});
+
+casper.waitForSelector('form.admin-realm-form input.btn');
+
+// deactivate message editing
+casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+    casper.evaluate(function () {
+        $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val('4');
+    });
+    casper.click('input[type="checkbox"][id="id_realm_allow_message_editing"]');
+    casper.click('form.admin-realm-form input.btn');
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can no longer edit their past messages!');
+        casper.test.assertEval(function () {
+            return !(document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked);
+        }, 'Allow message editing Setting de-activated');
+        casper.test.assertEval(function () {
+            return $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val() === '4';
+        }, 'Message content edit limit now 4');
+    });
+});
+
+casper.then(function () {
+    // allow message editing again, and check that the old edit limit is still there
+    casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+        casper.click('input[type="checkbox"][id="id_realm_allow_message_editing"]');
+        casper.click('form.admin-realm-form input.btn');
+    });
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can now edit topics for all their messages, and the content of messages which are less than 4 minutes old.');
+        casper.test.assertEval(function () {
+            return document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked;
+        }, 'Allow message editing Setting activated');
+        casper.test.assertEval(function () {
+            return $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val() === '4';
+        }, 'Message content edit limit still 4');
+    });
+});
+
+casper.then(function () {
+    // allow arbitrary message editing
+    casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+        casper.evaluate(function () {
+            $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val('0');
+        });
+        casper.click('form.admin-realm-form input.btn');
+    });
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can now edit the content and topics of all their past messages!');
+        casper.test.assertEval(function () {
+            return document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked;
+        }, 'Allow message editing Setting still activated');
+        casper.test.assertEval(function () {
+            return $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val() === '0';
+        }, 'Message content edit limit is 0');
+    });
+});
+
+casper.then(function () {
+    // disallow message editing, with illegal edit limit value. should be fixed by admin.js
+    casper.waitForSelector('input[type="checkbox"][id="id_realm_allow_message_editing"]', function () {
+        casper.evaluate(function () {
+            $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val('moo');
+        });
+        casper.click('input[type="checkbox"][id="id_realm_allow_message_editing"]');
+        casper.click('form.admin-realm-form input.btn');
+    });
+});
+
+casper.then(function () {
+    casper.waitUntilVisible('#admin-realm-message-editing-status', function () {
+        casper.test.assertSelectorHasText('#admin-realm-message-editing-status', 'Users can no longer edit their past messages!');
+        casper.test.assertEval(function () {
+            return !(document.querySelector('input[type="checkbox"][id="id_realm_allow_message_editing"]').checked);
+        }, 'Allow message editing Setting de-activated');
+        casper.test.assertEval(function () {
+            return $('input[type="text"][id="id_realm_message_content_edit_limit_minutes"]').val() === '10';
+        }, 'Message content edit limit has been reset to its default');
+    });
+});
+
+casper.then(function () {
+    casper.test.info("Changing realm default language");
+    casper.evaluate(function () {
+        $('#id_realm_default_language').val('de').change();
+    });
+    casper.click('form.admin-realm-form input.btn');
+});
+
+casper.waitUntilVisible('#admin-realm-default-language-status', function () {
+    casper.test.assertSelectorHasText('#admin-realm-default-language-status', 'Default language changed!');
+});
+
+common.then_log_out();
+
+casper.run(function () {
+    casper.test.done();
+});

frontend_tests/casper_tests/11-admin.js

@@ -1,192 +0,0 @@
-var common = require('../casper_lib/common.js').common;
-var test_credentials = require('../casper_lib/test_credentials.js').test_credentials;
-
-common.start_and_log_in();
-
-casper.then(function () {
-    casper.test.info('Administration page');
-    casper.click('a[href^="#administration"]');
-    casper.test.assertUrlMatch(/^http:\/\/[^\/]+\/#administration/, 'URL suggests we are on administration page');
-    casper.test.assertExists('#administration.tab-pane.active', 'Administration page is active');
-});
-
-// Test only admins may create streams Setting
-casper.waitForSelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]', function () {
-    casper.click('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]');
-    casper.click('form.admin-realm-form input.btn');
-
-    // Test setting was activated
-    casper.waitUntilVisible('#admin-realm-create-stream-by-admins-only-status', function () {
-        casper.test.assertSelectorHasText('#admin-realm-create-stream-by-admins-only-status', 'Only Admins may now create new streams!');
-        casper.test.assertEval(function () {
-            return document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked;
-        }, 'Only admins may create streams Setting activated');
-    });
-});
-
-casper.then(function () {
-    // Leave the page and return
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#subscriptions"]');
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#administration"]');
-
-    casper.waitForSelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]', function () {
-        // Test Setting was saved
-        casper.test.assertEval(function () {
-            return document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked;
-        }, 'Only admins may create streams Setting saved');
-
-        // Deactivate setting
-        casper.click('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]');
-        casper.click('form.admin-realm-form input.btn');
-        casper.waitUntilVisible('#admin-realm-create-stream-by-admins-only-status', function () {
-            casper.test.assertSelectorHasText('#admin-realm-create-stream-by-admins-only-status', 'Any user may now create new streams!');
-            casper.test.assertEval(function () {
-                return !(document.querySelector('input[type="checkbox"][id="id_realm_create_stream_by_admins_only"]').checked);
-            }, 'Only admins may create streams Setting deactivated');
-        });
-    });
-});
-
-// Test user deactivation and reactivation
-casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
-    casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
-    casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
-    casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
-    casper.click('#do_deactivate_user_button');
-});
-
-casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"].deactivated_user', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
-    casper.click('.user_row[id="user_cordelia@zulip.com"] .reactivate');
-});
-
-casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]:not(.deactivated_user)', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
-});
-
-// Test Deactivated users section of admin page
-casper.waitForSelector('.user_row[id="user_cordelia@zulip.com"]', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
-    casper.click('.user_row[id="user_cordelia@zulip.com"] .deactivate');
-    casper.test.assertTextExists('Deactivate cordelia@zulip.com', 'Deactivate modal has right user');
-    casper.test.assertTextExists('Deactivate now', 'Deactivate now button available');
-    casper.click('#do_deactivate_user_button');
-});
-
-casper.then(function () {
-    // Leave the page and return
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#subscriptions"]');
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#administration"]');
-
-    casper.test.assertSelectorHasText("#administration a[aria-controls='deactivated-users']", "Deactivated Users");
-    casper.click("#administration a[aria-controls='deactivated-users']");
-
-
-    casper.waitForSelector('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] .reactivate', function () {
-        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Reactivate');
-        casper.click('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] .reactivate');
-    });
-
-    casper.waitForSelector('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"] button:not(.reactivate)', function () {
-        casper.test.assertSelectorHasText('#admin_deactivated_users_table .user_row[id="user_cordelia@zulip.com"]', 'Deactivate');
-    });
-
-    casper.test.assertSelectorHasText("#administration a[aria-controls='organization']", "Organization");
-    casper.click("#administration a[aria-controls='organization']");
-});
-
-// Test bot deactivation and reactivation
-casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
-    casper.click('.user_row[id="user_new-user-bot@zulip.com"] .deactivate');
-});
-
-casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"].deactivated_user', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Reactivate');
-    casper.click('.user_row[id="user_new-user-bot@zulip.com"] .reactivate');
-});
-casper.waitForSelector('.user_row[id="user_new-user-bot@zulip.com"]:not(.deactivated_user)', function () {
-    casper.test.assertSelectorHasText('.user_row[id="user_new-user-bot@zulip.com"]', 'Deactivate');
-});
-
-// Test custom realm emoji
-casper.waitForSelector('.admin-emoji-form', function () {
-    casper.fill('form.admin-emoji-form', {
-        'name': 'MouseFace',
-        'url': 'http://localhost:9991/static/images/integrations/logos/jenkins.png'
-    });
-    casper.click('form.admin-emoji-form input.btn');
-});
-
-casper.waitUntilVisible('div#admin-emoji-status', function () {
-    casper.test.assertSelectorHasText('div#admin-emoji-status', 'Custom emoji added!');
-});
-
-casper.waitForSelector('.emoji_row', function () {
-    casper.test.assertSelectorHasText('.emoji_row .emoji_name', 'MouseFace');
-    casper.test.assertExists('.emoji_row img[src="http://localhost:9991/static/images/integrations/logos/jenkins.png"]');
-    casper.click('.emoji_row button.delete');
-});
-
-casper.waitWhileSelector('.emoji_row', function () {
-    casper.test.assertDoesntExist('.emoji_row');
-});
-
-function get_suggestions(str) {
-    casper.then(function () {
-        casper.evaluate(function (str) {
-            $('.create_default_stream')
-            .focus()
-            .val(str)
-            .trigger($.Event('keyup', { which: 0 }));
-        }, str);
-    });
-}
-
-function select_from_suggestions(item) {
-    casper.then(function () {
-        casper.evaluate(function (item) {
-            var tah = $('.create_default_stream').data().typeahead;
-            tah.mouseenter({
-                currentTarget: $('.typeahead:visible li:contains("'+item+'")')[0]
-            });
-            tah.select();
-        }, {item: item});
-    });
-}
-
-// Test default stream creation and addition
-casper.then(function () {
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#subscriptions"]');
-    casper.click('#settings-dropdown');
-    casper.click('a[href^="#administration"]');
-    var stream_name = "Scotland";
-    // It matches with all the stream names which has 'O' as a substring (Rome, Scotland, Verona etc).
-    // I used 'O' to make sure that it works even if there are multiple suggestions.
-    // Capital 'O' is used instead of small 'o' to make sure that the suggestions are not case sensitive.
-    get_suggestions("O");
-    select_from_suggestions(stream_name);
-    casper.waitForSelector('.default_stream_row[id='+stream_name+']', function () {
-        casper.test.assertSelectorHasText('.default_stream_row[id='+stream_name+'] .default_stream_name', stream_name);
-    });
-    casper.waitForSelector('.default_stream_row[id='+stream_name+']', function () {
-        casper.test.assertSelectorHasText('.default_stream_row[id='+stream_name+'] .default_stream_name', stream_name);
-        casper.click('.default_stream_row[id='+stream_name+'] button.remove-default-stream');
-    });
-    casper.waitWhileSelector('.default_stream_row[id='+stream_name+']', function () {
-        casper.test.assertDoesntExist('.default_stream_row[id='+stream_name+']');
-    });
-});
-// TODO: Test stream deletion
-
-common.then_log_out();
-
-casper.run(function () {
-    casper.test.done();
-});

frontend_tests/casperjs/bin/casperjs

@@ -8,7 +8,7 @@ import sys
 
 def test_cmd(cmd):
     try:
-        return subprocess.check_output([__file__] + cmd.split(' '))
+        return subprocess.check_output([__file__] + cmd.split(' '), universal_newlines=True)
     except subprocess.CalledProcessError as err:
         sys.stderr.write('FAIL: %s\n' % ' '.join(err.cmd))
         sys.stderr.write('      %s\n' % err.output)

frontend_tests/node_tests/activity.js

@@ -1,11 +1,4 @@
-set_global('$', function () {
-    return {
-        on: function () {
-            return;
-        }
-    };
-});
-$.fn = {};
+global.stub_out_jquery();
 
 add_dependencies({
     util: 'js/util.js',
@@ -18,8 +11,7 @@ set_global('document', {
     }
 });
 
-var people = require("js/people.js");
-people.test_set_people_dict({
+global.people.test_set_people_dict({
     'alice@zulip.com': {
         full_name: 'Alice Smith'
     },

frontend_tests/node_tests/dispatch.js

@@ -0,0 +1,725 @@
+var assert = require('assert');
+var _ = global._;
+
+var noop = function () {};
+
+// The next section of cruft will go away when we can pull out
+// dispatcher from server_events.
+(function work_around_server_events_loading_issues() {
+    add_dependencies({
+        util: 'js/util.js'
+    });
+    set_global('document', {});
+    set_global('window', {
+        addEventListener: noop
+    });
+    global.stub_out_jquery();
+}());
+
+// These dependencies are closer to the dispatcher, and they
+// apply to all tests.
+set_global('tutorial', {
+    is_running: function () {
+        return false;
+    }
+});
+set_global('home_msg_list', {
+    select_id: noop,
+    selected_id: function () {return 1;}
+});
+set_global('echo', {
+    process_from_server: function (messages) {
+        return messages;
+    },
+    set_realm_filters: noop
+});
+
+// page_params is highly coupled to dispatching now
+set_global('page_params', {test_suite: false});
+var page_params = global.page_params;
+
+// alert_words is coupled to dispatching in the sense
+// that we write directly to alert_words.words
+add_dependencies({alert_words: 'js/alert_words.js'});
+
+// we also directly write to pointer
+set_global('pointer', {});
+
+var server_events = require('js/server_events.js');
+
+// This also goes away if we can isolate the dispatcher.  We
+// have to call it after doing the require on server_events.js,
+// so that it can set a private variable for us that bypasses
+// code that queue up events and early-exits.
+server_events.home_view_loaded();
+
+// This jQuery shim can go away when we remove $.each from
+// server_events.js.  (It's a simple change that just
+// requires some manual testing.)
+$.each = function (data, f) {
+    _.each(data, function (value, key) {
+        f(key, value);
+    });
+};
+
+// Set up our dispatch function to point to _get_events_success
+// now.
+function dispatch(ev) {
+    server_events._get_events_success([ev]);
+}
+
+
+// TODO: These events are not guaranteed to be perfectly
+//       representative of what the server sends.  For
+//       now we just want very basic test coverage.  We
+//       have more mature tests for events on the backend
+//       side in test_events.py, and we may be able to
+//       re-work both sides (js/python) so that we work off
+//       a shared fixture.
+var event_fixtures = {
+    alert_words: {
+        type: 'alert_words',
+        alert_words: ['fire', 'lunch']
+    },
+
+    default_streams: {
+        type: 'default_streams',
+        default_streams: [
+            {
+                name: 'devel',
+                description: 'devel',
+                invite_only: false,
+                stream_id: 1
+            },
+            {
+                name: 'test',
+                description: 'test',
+                invite_only: true,
+                stream_id: 1
+            }
+        ]
+    },
+
+    message: {
+        type: 'message',
+        message: {
+            content: 'hello'
+        },
+        flags: []
+    },
+
+    muted_topics: {
+        type: 'muted_topics',
+        muted_topics: [['devel', 'js'], ['lunch', 'burritos']]
+    },
+
+    pointer: {
+        type: 'pointer',
+        pointer: 999
+    },
+
+    presence: {
+        type: 'presence',
+        email: 'alice@example.com',
+        presence: {
+            client_name: 'electron',
+            is_mirror_dummy: false
+            // etc.
+        },
+        server_timestamp: 999999
+    },
+
+    // Please keep this next section un-nested, as we want this to partly
+    // be simple documentation on the formats of individual events.
+    realm__update__create_stream_by_admins_only: {
+        type: 'realm',
+        op: 'update',
+        property: 'create_stream_by_admins_only',
+        value: false
+    },
+
+    realm__update__invite_by_admins_only: {
+        type: 'realm',
+        op: 'update',
+        property: 'invite_by_admins_only',
+        value: false
+    },
+
+    realm__update__invite_required: {
+        type: 'realm',
+        op: 'update',
+        property: 'invite_required',
+        value: false
+    },
+
+    realm__update__name: {
+        type: 'realm',
+        op: 'update',
+        property: 'name',
+        value: 'new_realm_name'
+    },
+
+    realm__update__restricted_to_domain: {
+        type: 'realm',
+        op: 'update',
+        property: 'restricted_to_domain',
+        value: false
+    },
+
+    realm__update_dict__default: {
+        type: 'realm',
+        op: 'update_dict',
+        property: 'default',
+        data: {
+            'allow_message_editing': true,
+            'message_content_edit_limit_seconds': 5
+        }
+    },
+
+    realm_bot__add: {
+        type: 'realm_bot',
+        op: 'add',
+        bot: {
+            email: 'the-bot@example.com',
+            full_name: 'The Bot'
+            // etc.
+        }
+    },
+
+    realm_bot__remove: {
+        type: 'realm_bot',
+        op: 'remove',
+        bot: {
+            email: 'the-bot@example.com',
+            full_name: 'The Bot'
+        }
+    },
+
+    realm_bot__update: {
+        type: 'realm_bot',
+        op: 'update',
+        bot: {
+            email: 'the-bot@example.com',
+            full_name: 'The Bot Has A New Name'
+        }
+    },
+
+    realm_emoji: {
+        type: 'realm_emoji',
+        realm_emoji: {
+            'airplane': {
+                display_url: 'some_url'
+            }
+        }
+    },
+
+    realm_filters: {
+        type: 'realm_filters',
+        realm_filters: [
+            ['#[123]', 'ticket %(id)s']
+        ]
+    },
+
+    realm_user__add: {
+        type: 'realm_user',
+        op: 'add',
+        person: {
+            email: 'alice@example.com',
+            full_name: 'Alice User'
+            // etc.
+        }
+    },
+
+    realm_user__remove: {
+        type: 'realm_user',
+        op: 'remove',
+        person: {
+            email: 'alice@example.com',
+            full_name: 'Alice User'
+            // etc.
+        }
+    },
+
+    realm_user__update: {
+        type: 'realm_user',
+        op: 'update',
+        person: {
+            email: 'alice@example.com',
+            full_name: 'Alice NewName'
+            // etc.
+        }
+    },
+
+    referral: {
+        type: 'referral',
+        referrals: {
+            granted: 10,
+            used: 5
+        }
+    },
+
+    restart: {
+        type: 'restart',
+        immediate: true
+    },
+
+    stream: {
+        type: 'stream',
+        op: 'update',
+        name: 'devel',
+        property: 'color',
+        value: 'blue'
+    },
+
+    subscription__add: {
+        type: 'subscription',
+        op: 'add',
+        subscriptions: [
+            {
+                name: 'devel',
+                stream_id: 42
+                // etc.
+            }
+        ]
+    },
+
+    subscription__remove: {
+        type: 'subscription',
+        op: 'remove',
+        subscriptions: [
+            {
+                stream_id: 42
+            }
+        ]
+    },
+
+    subscription__peer_add: {
+        type: 'subscription',
+        op: 'peer_add',
+        user_email: 'bob@example.com',
+        subscriptions: [
+            {
+                name: 'devel',
+                stream_id: 42
+                // etc.
+            }
+        ]
+    },
+
+    subscription__peer_remove: {
+        type: 'subscription',
+        op: 'peer_remove',
+        user_email: 'bob@example.com',
+        subscriptions: [
+            {
+                stream_id: 42
+                // etc.
+            }
+        ]
+    },
+
+    subscription__update: {
+        type: 'subscription',
+        op: 'update',
+        name: 'devel',
+        property: 'color',
+        value: 'black'
+    },
+
+    update_display_settings__default_language: {
+        type: 'update_display_settings',
+        setting_name: 'default_language',
+        default_language: 'fr'
+    },
+
+    update_display_settings__left_side_userlist: {
+        type: 'update_display_settings',
+        setting_name: 'left_side_userlist',
+        left_side_userlist: true
+    },
+
+    update_display_settings__twenty_four_hour_time: {
+        type: 'update_display_settings',
+        setting_name: 'twenty_four_hour_time',
+        twenty_four_hour_time: true
+    },
+
+    update_global_notifications: {
+        type: 'update_global_notifications',
+        notification_name: 'enable_stream_sounds',
+        setting: true
+    },
+
+    update_message_flags__read: {
+        type: 'update_message_flags',
+        operation: 'add',
+        flag: 'read',
+        messages: [5, 999]
+    },
+
+    update_message_flags__starred: {
+        type: 'update_message_flags',
+        operation: 'add',
+        flag: 'starred',
+        messages: [7, 99]
+    }
+};
+
+function assert_same(actual, expected) {
+    // This helper prevents us from getting false positives
+    // where actual and expected are both undefined.
+    assert(expected);
+    assert.deepEqual(actual, expected);
+}
+
+// TODO: move this into library
+function capture_args(res, arg_names) {
+    // This function returns a function that, when
+    // arg_names are ['foo', 'bar'] sets res.foo
+    // to the first arg passed in and res.bar to
+    // the second args passed in.  (It's basically
+    // a mock.)
+
+    _.each(res, function (value, key) {
+        delete res[key];
+    });
+
+    return function () {
+        var my_arguments = _.clone(arguments);
+        _.each(arg_names, function (name, i) {
+            res[name] = my_arguments[i];
+        });
+    };
+}
+
+
+// This test suite is different than most, because
+// most modules we test are dependent on a few
+// set of modules, and it's useful for tests to
+// all share the same stubs.  For a dispatcher,
+// we want a higher level of isolation between
+// our tests, so we wrap them with a run() method.
+
+var run = (function () {
+    var wrapper = function (f) {
+        // We only ever mock one function at a time,
+        // so we can have a little helper.
+        var args = {}; // for stubs to capture args
+        function capture(names) {
+            return capture_args(args, names);
+        }
+
+        var clobber_callbacks = [];
+
+        var override = function (module, func_name, f) {
+            var impl = {};
+            impl[func_name] = f;
+            set_global(module, impl);
+
+            clobber_callbacks.push(function () {
+                // If you get a failure from this, you probably just
+                // need to have your test do its own overrides and
+                // not cherry-pick off of the prior test's setup.
+                set_global(module, 'UNCLEAN MODULE FROM PRIOR TEST');
+            });
+        };
+
+        f(override, capture, args);
+
+        _.each(clobber_callbacks, function (f) {
+            f();
+        });
+    };
+
+    return wrapper;
+}());
+
+
+run(function (override, capture, args) {
+    // alert_words
+    var event = event_fixtures.alert_words;
+    dispatch(event);
+    assert_same(global.alert_words.words, ['fire', 'lunch']);
+
+});
+
+run(function (override, capture, args) {
+    // default_streams
+    var event = event_fixtures.default_streams;
+    override('admin', 'update_default_streams_table', noop);
+    dispatch(event);
+    assert_same(page_params.realm_default_streams, event.default_streams);
+
+});
+
+run(function (override, capture, args) {
+    // message
+    var event = event_fixtures.message;
+    override('message_store', 'insert_new_messages', capture(['messages']));
+    server_events._get_events_success([event]);
+    dispatch(event);
+    assert_same(args.messages[0].content, event.message.content);
+
+});
+
+run(function (override, capture, args) {
+    // muted_topics
+    var event = event_fixtures.muted_topics;
+    override('muting_ui', 'handle_updates', capture(['muted_topics']));
+    dispatch(event);
+    assert_same(args.muted_topics, event.muted_topics);
+
+});
+
+run(function (override, capture, args) {
+    // pointer
+    var event = event_fixtures.pointer;
+    global.pointer.furthest_read = 0;
+    global.pointer.server_furthest_read = 0;
+    dispatch(event);
+    assert_same(global.pointer.furthest_read, event.pointer);
+    assert_same(global.pointer.server_furthest_read, event.pointer);
+
+});
+
+run(function (override, capture, args) {
+    // presence
+    var event = event_fixtures.presence;
+    override('activity', 'set_user_statuses', capture(['users', 'server_time']));
+    dispatch(event);
+    assert_same(args.users, {'alice@example.com': event.presence});
+    assert_same(args.server_time, event.server_timestamp);
+
+});
+
+run(function (override, capture, args) {
+    // realm
+    function test_realm_boolean(event, parameter_name) {
+        page_params[parameter_name] = true;
+        event = _.clone(event);
+        event.value = false;
+        dispatch(event);
+        assert.equal(page_params[parameter_name], false);
+        event = _.clone(event);
+        event.value = true;
+        dispatch(event);
+        assert.equal(page_params[parameter_name], true);
+    }
+
+    var event = event_fixtures.realm__update__create_stream_by_admins_only;
+    test_realm_boolean(event, 'realm_create_stream_by_admins_only');
+
+    event = event_fixtures.realm__update__invite_by_admins_only;
+    test_realm_boolean(event, 'realm_invite_by_admins_only');
+
+    event = event_fixtures.realm__update__invite_required;
+    test_realm_boolean(event, 'realm_invite_required');
+
+    event = event_fixtures.realm__update__name;
+    override('notifications', 'redraw_title', noop);
+    dispatch(event);
+    assert_same(page_params.realm_name, 'new_realm_name');
+
+    event = event_fixtures.realm__update__restricted_to_domain;
+    test_realm_boolean(event, 'realm_restricted_to_domain');
+
+    event = event_fixtures.realm__update_dict__default;
+    page_params.realm_allow_message_editing = false;
+    page_params.realm_message_content_edit_limit_seconds = 0;
+    dispatch(event);
+    assert_same(page_params.realm_allow_message_editing, true);
+    assert_same(page_params.realm_message_content_edit_limit_seconds, 5);
+
+});
+
+run(function (override, capture, args) {
+    // realm_bot
+    var event = event_fixtures.realm_bot__add;
+    override('bot_data', 'add', capture(['bot']));
+    dispatch(event);
+    assert_same(args.bot, event.bot);
+
+    event = event_fixtures.realm_bot__remove;
+    override('bot_data', 'remove', capture(['email']));
+    dispatch(event);
+    assert_same(args.email, event.bot.email);
+
+    event = event_fixtures.realm_bot__update;
+    override('bot_data', 'update', capture(['email', 'bot']));
+    dispatch(event);
+    assert_same(args.email, event.bot.email);
+    assert_same(args.bot, event.bot);
+
+});
+
+run(function (override, capture, args) {
+    // realm_emoji
+    var event = event_fixtures.realm_emoji;
+    override('emoji', 'update_emojis', capture(['realm_emoji']));
+    override('admin', 'populate_emoji', noop);
+    dispatch(event);
+    assert_same(args.realm_emoji, event.realm_emoji);
+
+});
+
+run(function (override, capture, args) {
+    // realm_filters
+    var event = event_fixtures.realm_filters;
+    page_params.realm_filters = [];
+    dispatch(event);
+    assert_same(page_params.realm_filters, event.realm_filters);
+
+});
+
+run(function (override, capture, args) {
+    // realm_user
+    var event = event_fixtures.realm_user__add;
+    override('people', 'add_in_realm', capture(['person']));
+    dispatch(event);
+    assert_same(args.person, event.person);
+
+    event = event_fixtures.realm_user__remove;
+    override('people', 'remove', capture(['person']));
+    dispatch(event);
+    assert_same(args.person, event.person);
+
+    event = event_fixtures.realm_user__update;
+    override('people', 'update', capture(['person']));
+    dispatch(event);
+    assert_same(args.person, event.person);
+
+});
+
+run(function (override, capture, args) {
+    // referral
+    var event = event_fixtures.referral;
+    override('referral', 'update_state', capture(['granted', 'used']));
+    dispatch(event);
+    assert_same(args.granted, event.referrals.granted);
+    assert_same(args.used, event.referrals.used);
+
+});
+
+run(function (override, capture, args) {
+    // restart
+    var event = event_fixtures.restart;
+    override('reload', 'initiate', capture(['options']));
+    dispatch(event);
+    assert.equal(args.options.save_pointer, true);
+    assert.equal(args.options.immediate, true);
+});
+
+run(function (override, capture, args) {
+    // stream
+    var event = event_fixtures.stream;
+
+    override(
+        'subs',
+        'update_subscription_properties',
+        capture(['name', 'property', 'value'])
+    );
+    override('admin', 'update_default_streams_table', noop);
+    dispatch(event);
+    assert_same(args.name, event.name);
+    assert_same(args.property, event.property);
+    assert_same(args.value, event.value);
+
+});
+
+run(function (override, capture, args) {
+    // subscription
+    var event = event_fixtures.subscription__add;
+    override('subs', 'mark_subscribed', capture(['name', 'sub']));
+    dispatch(event);
+    assert_same(args.name, 'devel');
+    assert_same(args.sub, event.subscriptions[0]);
+
+    event = event_fixtures.subscription__peer_add;
+    override('stream_data', 'add_subscriber', capture(['sub', 'email']));
+    dispatch(event);
+    assert_same(args.sub, event.subscriptions[0]);
+    assert_same(args.email, event.user_email);
+
+    event = event_fixtures.subscription__peer_remove;
+    override('stream_data', 'remove_subscriber', capture(['sub', 'email']));
+    dispatch(event);
+    assert_same(args.sub, event.subscriptions[0]);
+    assert_same(args.email, event.user_email);
+
+    event = event_fixtures.subscription__remove;
+    var stream_id_looked_up;
+    var sub_stub = 'stub';
+    override('stream_data', 'get_sub_by_id', function (stream_id) {
+        stream_id_looked_up = stream_id;
+        return sub_stub;
+    });
+    override('subs', 'mark_sub_unsubscribed', capture(['sub']));
+    dispatch(event);
+    assert_same(stream_id_looked_up, event.subscriptions[0].stream_id);
+    assert_same(args.sub, sub_stub);
+
+    event = event_fixtures.subscription__update;
+    override(
+        'subs',
+        'update_subscription_properties',
+        capture(['name', 'property', 'value'])
+    );
+    dispatch(event);
+    assert_same(args.name, event.name);
+    assert_same(args.property, event.property);
+    assert_same(args.value, event.value);
+
+});
+
+run(function (override, capture, args) {
+    // update_display_settings
+    var event = event_fixtures.update_display_settings__default_language;
+    page_params.default_language = 'en';
+    dispatch(event);
+    assert_same(page_params.default_language, 'fr');
+
+    event = event_fixtures.update_display_settings__left_side_userlist;
+    page_params.left_side_userlist = false;
+    dispatch(event);
+    assert_same(page_params.left_side_userlist, true);
+
+    event = event_fixtures.update_display_settings__twenty_four_hour_time;
+    page_params.twenty_four_hour_time = false;
+    dispatch(event);
+    assert_same(page_params.twenty_four_hour_time, true);
+
+});
+
+run(function (override, capture, args) {
+    // update_global_notifications
+    var event = event_fixtures.update_global_notifications;
+    override(
+        'notifications',
+        'handle_global_notification_updates',
+        capture(['name', 'setting'])
+    );
+    dispatch(event);
+    assert_same(args.name, event.notification_name);
+    assert_same(args.setting, event.setting);
+
+});
+
+run(function (override, capture, args) {
+    // update_message_flags__read
+    var event = event_fixtures.update_message_flags__read;
+    override('message_store', 'get', capture(['message_id']));
+    override('unread', 'mark_messages_as_read', noop);
+    dispatch(event);
+    assert_same(args.message_id, 999);
+});
+
+run(function (override, capture, args) {
+    // update_message_flags__starred
+    var event = event_fixtures.update_message_flags__starred;
+    override('ui', 'update_starred', capture(['message_id', 'new_value']));
+    dispatch(event);
+    assert_same(args.message_id, 99);
+    assert_same(args.new_value, true); // for 'add'
+});
+
+