wal-g: Add support for incremental backups.

This only defaults to on for local-disk backups, since they are more
disk-size-sensitive, and local accesses are quite cheap compared to
loading multiple incremental backups from S3.

docs/production/export-and-import.md

@@ -556,8 +556,9 @@ day.
    ```
 
 You may also want to adjust the
-[concurrency](system-configuration.md#backups_disk_concurrency) or [S3 storage
-class](system-configuration.md#backups_storage_class).
+[concurrency](system-configuration.md#backups_disk_concurrency), [S3 storage
+class](system-configuration.md#backups_storage_class), or [incremental
+backups][incremental] configuration.
 
 ### Streaming backups to local disk
 
@@ -584,6 +585,10 @@ analysis of recent application-level data changes.
    /home/zulip/deployments/current/scripts/zulip-puppet-apply
    ```
 
+You may also want to adjust the [incremental backups][incremental]
+configuration.
+
 [wal]: https://www.postgresql.org/docs/current/wal-intro.html
 [archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
 [mobile-push]: ../production/mobile-push-notifications.md
+[incremental]: system-configuration.md#backups_incremental

docs/production/system-configuration.md

@@ -269,6 +269,11 @@ on a NAS mountpoint, or if some other process copies this data off the disk; or
 if backups are purely for point-in-time historical analysis of recent
 application-level data changes.
 
+#### `backups_incremental`
+
+The number of delta (incremental) database backups to take between full backups.
+Defaults to 0 for S3 backups, and 6 for local-disk backups.
+
 #### `backups_storage_class`
 
 What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when

puppet/zulip/files/postgresql/env-wal-g

@@ -5,6 +5,9 @@ fi
 
 export PGHOST=/var/run/postgresql/
 
+WALG_DELTA_MAX_STEPS=$(crudini --get /etc/zulip/zulip.conf postgresql backups_incremental 2>/dev/null)
+export WALG_DELTA_MAX_STEPS
+
 s3_backups_bucket=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_bucket 2>/dev/null)
 
 if [ "$s3_backups_bucket" != "" ]; then
@@ -30,6 +33,11 @@ else
     WALG_FILE_PREFIX=$(crudini --get /etc/zulip/zulip.conf postgresql backups_directory 2>/dev/null)
     if [ "$WALG_FILE_PREFIX" != "" ]; then
         export WALG_FILE_PREFIX
+        if [ "$WALG_DELTA_MAX_STEPS" = "" ]; then
+            # Default to only taking a full backup every week
+            export WALG_DELTA_MAX_STEPS=6
+        fi
+
     else
         echo "Could not determine where to back up data to!"
         exit 1