uploads: Skip the outgoing proxy if S3_KEY is unset.

When the credentials are provided by dint of being run on an EC2
instance with an assigned Role, we must be able to fetch the instance
metadata from IMDS -- which is precisely the type of internal-IP
request that Smokescreen denies.

While botocore supports a `proxies` argument to the `Config` object,
this is not actually respected when making the IMDS queries; only the
environment variables are read from.  See
https://github.com/boto/botocore/issues/2644

As such, implement S3_SKIP_PROXY by monkey-patching the
`botocore.utils.should_bypass_proxies` function, to allow requests to
IMDS to be made without Smokescreen impeding them.

Fixes #20715.

docs/production/deployment.md

@@ -274,6 +274,19 @@ class, and configure the `[http_proxy]` block as above.
 [smokescreen-acls]: https://github.com/stripe/smokescreen#acls
 [ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
 
+### S3 file storage requests and outgoing proxies
+
+By default, the [S3 file storage backend][s3] bypasses the Smokescreen
+proxy, because when running on EC2 it may require metadata from the
+IMDS metadata endpoint, which resides on the internal IP address
+169.254.169.254 and would thus be blocked by Smokescreen.
+
+If your S3-compatible storage backend requires use of Smokescreen or
+some other proxy, you can override this default by setting
+`S3_SKIP_PROXY = False` in `/etc/zulip/settings.py`.
+
+[s3]: upload-backends.md#s3-backend-configuration
+
 ## Putting the Zulip application behind a reverse proxy
 
 Zulip is designed to support being run behind a reverse proxy server.

docs/production/upload-backends.md

@@ -28,6 +28,8 @@ backend. To enable this backend, you need to do the following:
 
 1. Set `s3_key` and `s3_secret_key` in /etc/zulip/zulip-secrets.conf
    to be the S3 access and secret keys for the IAM account.
+   Alternately, if your Zulip server runs on an EC2 instance, set the
+   IAM role for the EC2 instance to the role.
 
 1. Set the `S3_AUTH_UPLOADS_BUCKET` and `S3_AVATAR_BUCKET` settings in
    `/etc/zulip/settings.py` to be the names of the S3 buckets you

zerver/lib/upload.py

@@ -85,6 +85,14 @@ INLINE_MIME_TYPES = [
 # through a sanitization function.
 
 
+# https://github.com/boto/botocore/issues/2644 means that the IMDS
+# request _always_ pulls from the environment.  Monkey-patch the
+# `should_bypass_proxies` function if we need to skip them, based
+# on S3_SKIP_PROXY.
+if settings.S3_SKIP_PROXY is True:  # nocoverage
+    botocore.utils.should_bypass_proxies = lambda url: True
+
+
 class RealmUploadQuotaError(JsonableError):
     code = ErrorCode.REALM_UPLOAD_QUOTA
 

zproject/default_settings.py

@@ -137,6 +137,7 @@ S3_AVATAR_BUCKET = ""
 S3_AUTH_UPLOADS_BUCKET = ""
 S3_REGION: Optional[str] = None
 S3_ENDPOINT_URL: Optional[str] = None
+S3_SKIP_PROXY = True
 LOCAL_UPLOADS_DIR: Optional[str] = None
 MAX_FILE_UPLOAD_SIZE = 25
 

zproject/prod_settings_template.py

@@ -721,6 +721,7 @@ LOCAL_UPLOADS_DIR = "/home/zulip/uploads"
 # S3_AVATAR_BUCKET = ""
 # S3_REGION = None
 # S3_ENDPOINT_URL = None
+# S3_SKIP_PROXY = True
 
 ## Maximum allowed size of uploaded files, in megabytes.  This value is
 ## capped at 80MB in the nginx configuration, because the file upload