models: Add is_private flag to UserMessage and add index for it.

The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.

This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.

For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.

History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org.  A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).

docs/overview/changelog.md

@@ -15,6 +15,17 @@ in bursts.
   access full history, even before they joined the stream.
 - Added support for announcement-only streams.
 
+**Upgrade notes:**
+
+* Zulip 1.9 contains a significant database migration that can take
+  several minutes to run.  The upgrade process automatically minimizes
+  disruption by running this migration first, before beginning the
+  user-facing downtime.  However, if you'd like to watch the downtime
+  phase of the upgrade closely, we recommend
+  [running them first manually](../production/expensive-migrations.html)
+  and as well as the usual trick of
+  [doing an apt upgrade first](../production/maintain-secure-upgrade.html#applying-system-updates).
+
 **Full feature changelog:**
 - Added an organization setting for message deletion time limits.
 - Added an organization setting to control who can edit topics.

docs/production/expensive-migrations.md

@@ -4,8 +4,8 @@
 
 # Running expensive migrations early
 
-Zulip 1.7 contains some significant database migrations that can take
-several minutes to run.
+Zulip 1.7 and 1.9 each contain some significant database migrations
+that can take several minutes to run.
 
 The upgrade process automatically minimizes disruption by running
 these first, before beginning the user-facing downtime.  However, if
@@ -19,6 +19,14 @@ can run them manually before starting the upgrade:
   Postgres database.
 3. In the postgres shell, run the following commands:
 
+        CREATE INDEX CONCURRENTLY
+        zerver_usermessage_is_private_message_id
+        ON zerver_usermessage (user_profile_id, message_id)
+        WHERE (flags & 2048) != 0;
+
+(This first migration, `zerver_usermessage_is_private_message_id`, is
+the only one new in Zulip 1.9).
+
         CREATE INDEX CONCURRENTLY
         zerver_usermessage_mentioned_message_id
         ON zerver_usermessage (user_profile_id, message_id)
@@ -44,13 +52,13 @@ can run them manually before starting the upgrade:
         ON zerver_usermessage (user_profile_id, message_id)
         WHERE (flags & 1) = 0;
 
-4. These will take some time to run, during which the server will
-  continue to serve user traffic as usual with no disruption.  Once
-  they finish, you can proceed with installing Zulip 1.7.
+These will take some time to run, during which the server will
+continue to serve user traffic as usual with no disruption.  Once they
+finish, you can proceed with installing Zulip 1.7.
 
 To help you estimate how long these will take on your server: count
 the number of UserMessage rows, with `select COUNT(*) from zerver_usermessage;`
 at the `./manage.py dbshell` prompt.  At the time these migrations
-were run on chat.zulip.org, it had 75M UserMessage rows; the first 4
+were run on chat.zulip.org, it had 75M UserMessage rows; the first 5
 indexes took about 1 minute each to create, and the final,
 "unread_message" index took more like 10 minutes.