wal-g: Provide a to-local-disk backup option.

docs/production/export-and-import.md

@@ -508,44 +508,22 @@ for more details on supported options.
 
 ## Database-only backup tools
 
-The [Zulip-specific backup tool documented above](#backups) is perfect
+The [Zulip-specific backup tool documented above](#backups) is perfect for an
-for an all-in-one backup solution, and can be used for nightly
+all-in-one backup solution, and can be used for nightly backups. For
-backups. For administrators wanting continuous point-in-time backups,
+administrators wanting continuous point-in-time backups, Zulip has built-in
-Zulip has built-in support for taking daily backup snapshots along
+support for taking daily backup snapshots along with [streaming write-ahead log
-with [streaming write-ahead log (WAL)][wal] backups using
+(WAL)][wal] backups using [wal-g](https://github.com/wal-g/wal-g). By default,
-[wal-g](https://github.com/wal-g/wal-g) and storing them in Amazon S3.
+these backups are stored for 30 days.
-By default, these backups are stored for 30 days.
 
 Note these database backups, by themselves, do not constitute a full
 backup of the Zulip system! [See above](#backup-details) for other
 pieces which are necessary to back up a Zulip system.
 
-To enable continuous point-in-time backups:
+Daily full-database backups will be taken at 0200 UTC, and every [WAL][wal]
-
+archive file will be backed up as it is saved by PostgreSQL; these are written
-1. Edit `/etc/zulip/zulip-secrets.conf` on the
+every 16KiB of the WAL. This means that if there are periods of slow activity,
-   PostgreSQL server to add:
+it may be minutes before the backup is saved into S3 -- see
-
+[`archive_timeout`][archive-timeout] for how to set an upper bound on this.
-   ```ini
-   s3_region = # region to write to S3; defaults to EC2 host's region
-   s3_backups_key = # aws public key; optional, if access not through role
-   s3_backups_secret_key =  # aws secret key; optional, if access not through role
-   s3_backups_bucket = # name of S3 backup bucket
-   ```
-
-1. Run `/home/zulip/deployments/current/scripts/zulip-puppet-apply`.
-
-Daily full-database backups will be taken at 0200 UTC, and every WAL
-archive file will be written to S3 as it is saved by PostgreSQL; these
-are written every 16KiB of the WAL. This means that if there are
-periods of slow activity, it may be minutes before the backup is saved
-into S3 -- see [`archive_timeout`][archive-timeout] for how to set an
-upper bound on this. On an active Zulip server, this also means the
-Zulip server will be regularly sending PutObject requests to S3,
-possibly thousands of times per day.
-
-You may also want to adjust the
-[concurrency](system-configuration.md#backups-disk-concurrency) or [S3 storage
-class](system-configuration.md#backups-storage-class).
 
 If you need always-current backup availability, Zulip also has
 [built-in database replication support](deployment.md#postgresql-warm-standby).
@@ -554,6 +532,58 @@ You can (and should) monitor that backups are running regularly via
 the Nagios plugin installed into
 `/usr/lib/nagios/plugins/zulip_postgresql_backups/check_postgresql_backup`.
 
+### Streaming backups to S3
+
+This provides a durable and reliable off-host database backup, and we suggest
+this configuration for resilience to disk failures. Because backups are written
+to S3 as the WAL logs are written, this means that an active Zulip server will
+be regularly sending PutObject requests to S3, possibly thousands of times per
+day.
+
+1. Edit `/etc/zulip/zulip-secrets.conf` on the PostgreSQL server to add:
+
+   ```ini
+   s3_region = # region to write to S3; defaults to EC2 host's region
+   s3_backups_key = # aws public key; optional, if access not through role
+   s3_backups_secret_key =  # aws secret key; optional, if access not through role
+   s3_backups_bucket = # name of S3 backup bucket
+   ```
+
+1. Run:
+
+   ```shell
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply
+   ```
+
+You may also want to adjust the
+[concurrency](system-configuration.md#backups-disk-concurrency) or [S3 storage
+class](system-configuration.md#backups-storage-class).
+
+### Streaming backups to local disk
+
+As an alternative to storing backups to S3, you can also store backups to a
+local disk. This option is not recommended for disaster recovery purposes,
+since unless the directory is on a different disk from the database itself,
+_backups will likely also be lost if the database is lost._ This setting can be
+useful if the path is on a NAS mountpoint, or if some other process copies this
+data off the disk; or if backups are purely for point-in-time historical
+analysis of recent application-level data changes.
+
+1. Edit `/etc/zulip/zulip.conf` on the PostgreSQL server, and add to the existing
+   `[postgresql]` section:
+
+   ```ini
+    # Adjust this path to your desired storage location; this should be on a
+    # different disk than /var/lib/postgresql/ which stores the database.
+    backups_directory = /srv/zulip-db-backups
+   ```
+
+1. Run:
+
+   ```shell
+   /home/zulip/deployments/current/scripts/zulip-puppet-apply
+   ```
+
 [wal]: https://www.postgresql.org/docs/current/wal-intro.html
 [archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
 [mobile-push]: ../production/mobile-push-notifications.md

docs/production/system-configuration.md

@@ -257,6 +257,18 @@ may wish to increase this if you are taking backups on a replica, so can afford
 to affect other disk I/O, and have an SSD which is good at parallel random
 reads.
 
+#### `backups_directory`
+
+If S3 secrets are not configured, perform daily database backups to this path on
+disk instead. It should be owned by the `postgres` user.
+
+This option is not recommended for disaster recovery purposes, since unless the
+directory is on a different disk from the database itself, _backups will likely
+also be lost if the database is lost._ This setting can be useful if the path is
+on a NAS mountpoint, or if some other process copies this data off the disk; or
+if backups are purely for point-in-time historical analysis of recent
+application-level data changes.
+
 #### `backups_storage_class`
 
 What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when

puppet/zulip/files/postgresql/env-wal-g

@@ -4,27 +4,36 @@ if [ -z "$ZULIP_SECRETS_CONF" ]; then
 fi
 
 export PGHOST=/var/run/postgresql/
-AWS_REGION=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_region 2>/dev/null)
-if [ "$AWS_REGION" = "" ]; then
-    # Fall back to the current region, if possible
-    AZ=$(ec2metadata --availability-zone || true)
-    if [ -n "$AZ" ] && [ "$AZ" != "unavailable" ]; then
-        AWS_REGION=$(echo "$AZ" | sed 's/.$//')
-    fi
-fi
-export AWS_REGION
-AWS_ACCESS_KEY_ID=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_key 2>/dev/null)
-export AWS_ACCESS_KEY_ID
-AWS_SECRET_ACCESS_KEY=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_secret_key 2>/dev/null)
-export AWS_SECRET_ACCESS_KEY
-if ! s3_backups_bucket=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_bucket 2>&1); then
-    echo "Could not determine which s3 bucket to use:" "$s3_backups_bucket"
-    exit 1
-fi
-export WALG_S3_PREFIX="s3://$s3_backups_bucket"
 
-if storage_class=$(crudini --get /etc/zulip/zulip.conf postgresql backups_storage_class 2>&1); then
+s3_backups_bucket=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_bucket 2>/dev/null)
-    export WALG_S3_STORAGE_CLASS="$storage_class"
+
+if [ "$s3_backups_bucket" != "" ]; then
+    AWS_REGION=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_region 2>/dev/null)
+    if [ "$AWS_REGION" = "" ]; then
+        # Fall back to the current region, if possible
+        AZ=$(ec2metadata --availability-zone || true)
+        if [ -n "$AZ" ] && [ "$AZ" != "unavailable" ]; then
+            AWS_REGION=$(echo "$AZ" | sed 's/.$//')
+        fi
+    fi
+    export AWS_REGION
+    AWS_ACCESS_KEY_ID=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_key 2>/dev/null)
+    export AWS_ACCESS_KEY_ID
+    AWS_SECRET_ACCESS_KEY=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_secret_key 2>/dev/null)
+    export AWS_SECRET_ACCESS_KEY
+    export WALG_S3_PREFIX="s3://$s3_backups_bucket"
+
+    if storage_class=$(crudini --get /etc/zulip/zulip.conf postgresql backups_storage_class 2>&1); then
+        export WALG_S3_STORAGE_CLASS="$storage_class"
+    fi
+else
+    WALG_FILE_PREFIX=$(crudini --get /etc/zulip/zulip.conf postgresql backups_directory 2>/dev/null)
+    if [ "$WALG_FILE_PREFIX" != "" ]; then
+        export WALG_FILE_PREFIX
+    else
+        echo "Could not determine where to back up data to!"
+        exit 1
+    fi
 fi
 
 exec /usr/local/bin/wal-g "$@"

puppet/zulip/manifests/postgresql_backups.pp

@@ -50,6 +50,16 @@ class zulip::postgresql_backups {
     ],
   }
 
+  $postgresql_backup_directory = zulipconf('postgresql', 'backups_directory', '')
+  if $postgresql_backup_directory != '' {
+    file { $postgresql_backup_directory:
+      ensure => directory,
+      owner  => 'postgres',
+      group  => 'postgres',
+      mode   => '0600',
+    }
+  }
+
   file { "${zulip::common::nagios_plugins_dir}/zulip_postgresql_backups":
     require => Package[$zulip::common::nagios_plugins],
     recurse => true,

puppet/zulip/manifests/postgresql_base.pp

@@ -100,8 +100,9 @@ class zulip::postgresql_base {
     }
   }
 
-  $s3_backups_bucket = zulipsecret('secrets', 's3_backups_bucket', '')
+  $backups_s3_bucket = zulipsecret('secrets', 's3_backups_bucket', '')
-  if $s3_backups_bucket != '' {
+  $backups_directory = zulipconf('postgresql', 'backups_directory', '')
+  if $backups_s3_bucket != '' or $backups_directory != '' {
     include zulip::postgresql_backups
   }
 }