Release Zulip Server 2.1.1.

This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.

It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.

Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions.  Thanks to @alfonsrv for the suggestion.

docs/overview/changelog.md

@@ -7,6 +7,15 @@ All notable changes to the Zulip server are documented in this file.
 This section lists notable unreleased changes; it is generally updated
 in bursts.
 
+### 2.1.1 -- 2019-12-13
+
+- Fixed upgrading to 2.1.x with the LDAP integration enabled in a
+  configuration where `AUTH_LDAP_REVERSE_EMAIL_SEARCH` is newly
+  required, but is not set yet.
+- Reimplemented --postgres-missing-dictionaries installer option,
+  used with our new support for a DBaaS managed database.
+- Improved documentation for `AUTH_LDAP_REVERSE_EMAIL_SEARCH`.
+
 ### 2.1.0 -- 2019-12-12
 
 **Highlights:**

docs/production/authentication-methods.md

@@ -176,29 +176,33 @@ In either configuration, you will need to do the following:
    the form it needs for authentication.  There are three supported
    ways to set up the username and/or email mapping:
 
-   (A) Using email addresses as usernames, if LDAP has each user's
-      email address.  To do this, just set `AUTH_LDAP_USER_SEARCH` to
-      query by email address.
+   (A) Using email addresses as Zulip usernames, if LDAP has each
+      user's email address:
+      * Make `AUTH_LDAP_USER_SEARCH` a query by email address.
+      * Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to the same query with
+        `%(email)s` rather than `%(user)s` as the search parameter.
+      * Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
+        attribute for the user's LDAP username in the search result
+        for `AUTH_LDAP_REVERSE_EMAIL_SEARCH`.
 
    (B) Using LDAP usernames as Zulip usernames, with email addresses
-      formed consistently like `sam` -> `sam@example.com`.  To do
-      this, set `AUTH_LDAP_USER_SEARCH` to query by LDAP username, and
-      `LDAP_APPEND_DOMAIN = "example.com"`.
+      formed consistently like `sam` -> `sam@example.com`:
+      * Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
+      * Set `LDAP_APPEND_DOMAIN = "example.com"`.
 
    (C) Using LDAP usernames as Zulip usernames, with email addresses
-      taken from some other attribute in LDAP (for example, `email`).
-      To do this, set `AUTH_LDAP_USER_SEARCH` to query by LDAP
-      username, and `LDAP_EMAIL_ATTR = "email"`.
-
-1. In configurations (A) and (C), you need to tell Zulip how to look
-   up a user's LDAP data given their user's email address:
-
-   * Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find an
-   LDAP user given their email address.  Generally, this will be
-   `AUTH_LDAP_USER_SEARCH` in configuration (A) or a search by
-   `LDAP_EMAIL_ATTR` in configuration (C).
-   * Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP attribute
-   for the user's LDAP username in that search result.
+      taken from some other attribute in LDAP (for example, `mail`):
+      * Set `AUTH_LDAP_USER_SEARCH` to query by LDAP username
+      * Set `LDAP_EMAIL_ATTR = "mail"`.
+      * Set `AUTH_LDAP_REVERSE_EMAIL_SEARCH` to a query that will find
+        an LDAP user given their email address (i.e. a search by
+        `LDAP_EMAIL_ATTR`).  For example:
+        ```
+        AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
+                                                    ldap.SCOPE_SUBTREE, "(mail=%(email)s)")
+        ```
+      * Set `AUTH_LDAP_USERNAME_ATTR` to the name of the LDAP
+        attribute for the user's LDAP username in that search result.
 
 You can quickly test whether your configuration works by running:
 
@@ -210,24 +214,29 @@ from the root of your Zulip installation.  If your configuration is
 working, that will output the full name for your user (and that user's
 email address, if it isn't the same as the "Zulip username").
 
-**Active Directory**: For Active Directory, one typically sets
-  `AUTH_LDAP_USER_SEARCH` to one of:
+**Active Directory**: Most Active Directory installations will use one
+of the following configurations:
 
 * To access by Active Directory username:
     ```
     AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
                                        ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
+    AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
+                                       ldap.SCOPE_SUBTREE, "(mail=%(email)s)")
     ```
+
 * To access by Active Directory email address:
     ```
     AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
                                        ldap.SCOPE_SUBTREE, "(mail=%(user)s)")
+    AUTH_LDAP_REVERSE_EMAIL_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
+                                                ldap.SCOPE_SUBTREE, "(mail=%(email)s)")
     ```
 
 **If you are using LDAP for authentication**: you will need to enable
 the `zproject.backends.ZulipLDAPAuthBackend` auth backend, in
-`AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`.  After doing
-so (and as always [restarting the Zulip server](settings.md) to ensure
+`AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`.  After doing so
+(and as always [restarting the Zulip server](settings.md) to ensure
 your settings changes take effect), you should be able to log into
 Zulip by entering your email address and LDAP password on the Zulip
 login form.

docs/production/deployment.md

@@ -134,14 +134,11 @@ follows:
 postgres_password = abcd1234
 ```
 
-Now complete the installation by running the following command to ask
-the Zulip installer to initialize the postgres database. (Note: The
-options are different from before).
+Now complete the installation by running the following commands.
 
 ```
-./zulip-server-*/scripts/setup/install --certbot \
-    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
-    --remote-postgres --postgres-missing-dictionaries
+# Ask Zulip installer to initialize the postgres database.
+su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
 
 # And then generate a realm creation link:
 su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'

scripts/lib/install

@@ -280,7 +280,7 @@ if [ "$DEPLOYMENT_TYPE" = "dockervoyager" ]; then
 fi
 
 if [ -n "$POSTGRES_MISSING_DICTIONARIES" ]; then
-    export POSTGRES_MISSING_DICTIONARIES="true"
+    crudini --set /etc/zulip/zulip.conf postgresql missing_dictionaries true
 fi
 
 if [ -n "$REMOTE_POSTGRES" ]; then

version.py

@@ -1,6 +1,6 @@
 import os
 
-ZULIP_VERSION = "2.1.0"
+ZULIP_VERSION = "2.1.1"
 # Add information on number of commits and commit hash to version, if available
 zulip_git_version_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'zulip-git-version')
 if os.path.exists(zulip_git_version_file):
@@ -10,7 +10,7 @@ if os.path.exists(zulip_git_version_file):
             ZULIP_VERSION = version
 
 LATEST_MAJOR_VERSION = "2.1"
-LATEST_RELEASE_VERSION = "2.1.0"
+LATEST_RELEASE_VERSION = "2.1.1"
 LATEST_RELEASE_ANNOUNCEMENT = "https://blog.zulip.org/2019/12/13/zulip-2-1-released/"
 
 # Bump the minor PROVISION_VERSION to indicate that folks should provision

zerver/migrations/0001_initial.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Generated by Django 1.11.2 on 2017-06-22 10:22
-import os
 
 import bitfield.models
 import django.contrib.auth.models
@@ -37,12 +36,11 @@ class Migration(migrations.Migration):
 
     initial = True
 
-    zulip_postgres_dictionaries_unavailable = os.getenv("POSTGRES_MISSING_DICTIONARIES", False)
     dependencies = [
         ('auth', '0001_initial'),
     ]
 
-    if zulip_postgres_dictionaries_unavailable:
+    if settings.POSTGRES_MISSING_DICTIONARIES:
         fts_sql = """
 CREATE TEXT SEARCH CONFIGURATION zulip.english_us_search (COPY=pg_catalog.english);
 """

zerver/migrations/0209_user_profile_no_empty_password.py

@@ -3,7 +3,6 @@
 from __future__ import unicode_literals
 
 from django.conf import settings
-from django.contrib.auth import get_backends
 from django.db import migrations
 from django.db.backends.postgresql_psycopg2.schema import DatabaseSchemaEditor
 from django.db.migrations.state import StateApps
@@ -13,7 +12,6 @@ from django.utils.timezone import now as timezone_now
 from zerver.lib.cache import cache_delete, user_profile_by_api_key_cache_key
 from zerver.lib.queue import queue_json_publish
 from zerver.lib.utils import generate_api_key
-from zproject.backends import EmailAuthBackend
 
 from typing import Any, Set, Union
 
@@ -159,8 +157,7 @@ def ensure_no_empty_passwords(apps: StateApps, schema_editor: DatabaseSchemaEdit
     # If Zulip's built-in password authentication is not enabled on
     # the server level, then we plan to skip resetting any users' API
     # keys, since the bug requires EmailAuthBackend.
-    email_auth_enabled = any(isinstance(backend, EmailAuthBackend)
-                             for backend in get_backends())
+    email_auth_enabled = 'zproject.backends.EmailAuthBackend' in settings.AUTHENTICATION_BACKENDS
 
     # A quick note: This query could in theory exclude users with
     # is_active=False, is_bot=True, or realm__deactivated=True here to

zproject/settings.py

@@ -293,6 +293,8 @@ elif REMOTE_POSTGRES_HOST != '':
     else:
         DATABASES['default']['OPTIONS']['sslmode'] = 'verify-full'
 
+POSTGRES_MISSING_DICTIONARIES = bool(get_config('postgresql', 'missing_dictionaries', None))
+
 ########################################################################
 # RABBITMQ CONFIGURATION
 ########################################################################