saml: Support syncing custom profile fields with SAML attributes.

Fixes #17277.

The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.

We detail that limitation in the documentation.

docs/production/authentication-methods.md

@@ -202,8 +202,6 @@ corresponding LDAP attribute is `linkedinProfile` then you just need
 to add `'custom_profile_field__linkedin_profile': 'linkedinProfile'`
 to the `AUTH_LDAP_USER_ATTR_MAP`.
 
-[custom-profile-fields]: https://zulip.com/help/add-custom-profile-fields
-
 #### Automatically deactivating users with Active Directory
 
 Starting with Zulip 2.0, Zulip supports synchronizing the
@@ -375,7 +373,7 @@ it as follows:
       if `SOCIAL_AUTH_SUBDOMAIN="auth"` and `EXTERNAL_HOST=zulip.example.com`,
       this should be `https://auth.zulip.example.com/complete/saml/`.
 
-2. Tell Zulip how to connect to your SAML provider(s) by filling
+1. Tell Zulip how to connect to your SAML provider(s) by filling
    out the section of `/etc/zulip/settings.py` on your Zulip server
    with the heading "SAML Authentication".
    * You will need to update `SOCIAL_AUTH_SAML_ORG_INFO` with your
@@ -403,7 +401,7 @@ it as follows:
      5. The `display_name` and `display_icon` fields are used to
         display the login/registration buttons for the IdP.
 
-3. Install the certificate(s) required for SAML authentication.  You
+1. Install the certificate(s) required for SAML authentication.  You
     will definitely need the public certificate of your IdP.  Some IdP
     providers also support the Zulip server (Service Provider) having
     a certificate used for encryption and signing.  We detail these
@@ -430,7 +428,7 @@ it as follows:
     chmod 640 /etc/zulip/saml/zulip-private-key.key
     ```
 
-4. (Optional) If you configured the optional public and private server
+1. (Optional) If you configured the optional public and private server
    certificates above, you can enable the additional setting
    `"authnRequestsSigned": True` in `SOCIAL_AUTH_SAML_SECURITY_CONFIG`
    to have the SAMLRequests the server will be issuing to the IdP
@@ -439,15 +437,34 @@ it as follows:
    assertions in the SAMLResponses the IdP will send about
    authenticated users.
 
-5. Enable the `zproject.backends.SAMLAuthBackend` auth backend, in
+1. Enable the `zproject.backends.SAMLAuthBackend` auth backend, in
 `AUTHENTICATION_BACKENDS` in `/etc/zulip/settings.py`.
 
-6. [Restart the Zulip server](../production/settings.md) to ensure
+1. (Optional) New in Zulip 5.0: Zulip can synchronize [custom profile
+   fields][custom-profile-fields] from the SAML provider. Just
+   configure the `SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT`; the
+   [LDAP](#synchronizing-custom-profile-fields) documentation for
+   synchronizing custom profile fields will be helpful. Servers
+   installed before Zulip 5.0 may want to [update inline comment
+   documentation][update-inline-comments] so they can take advantage
+   of the latest inline SAML documentation in
+   `/etc/zulip/settings.py`.
+
+   Note that in contrast with LDAP, Zulip can only query the SAML
+   database for a user's settings when the user authenticates to Zulip
+   using SAML, so custom profile fields are only synchronized when the
+   user logs in.
+
+   Note also that the SAML feature currently only synchronizes custom
+   profile fields during login, not during account creation; we
+   consider this [a bug](https://github.com/zulip/zulip/issues/18746).
+
+1. [Restart the Zulip server](../production/settings.md) to ensure
 your settings changes take effect.  The Zulip login page should now
 have a button for SAML authentication that you can use to log in or
 create an account (including when creating a new organization).
 
-7. If the configuration was successful, the server's metadata can be
+1. If the configuration was successful, the server's metadata can be
 found at `https://yourzulipdomain.example.com/saml/metadata.xml`. You
 can use this for verifying your configuration or provide it to your
 IdP.
@@ -705,3 +722,6 @@ helpful developer documentation on
 The `DevAuthBackend` method is used only in development, to allow
 passwordless login as any user in a development environment.  It's
 mentioned on this page only for completeness.
+
+[custom-profile-fields]: https://zulip.com/help/add-custom-profile-fields
+[update-inline-comments]: ../production/upgrade-or-modify.html#updating-settings-py-inline-documentation

zerver/lib/types.py

@@ -65,5 +65,6 @@ class SAMLIdPConfigDict(TypedDict, total=False):
     display_name: str
     display_icon: str
     limit_to_subdomains: List[str]
+    extra_attrs: List[str]
     x509cert: str
     x509cert_path: str

zerver/tests/test_auth_backends.py

@@ -2454,6 +2454,101 @@ class SAMLAuthBackendTest(SocialAuthBase):
             ],
         )
 
+    def test_social_auth_custom_profile_field_sync(self) -> None:
+        birthday_field = CustomProfileField.objects.get(
+            realm=self.user_profile.realm, name="Birthday"
+        )
+        old_birthday_field_value = CustomProfileFieldValue.objects.get(
+            user_profile=self.user_profile, field=birthday_field
+        ).value
+
+        idps_dict = copy.deepcopy(settings.SOCIAL_AUTH_SAML_ENABLED_IDPS)
+        idps_dict["test_idp"]["extra_attrs"] = ["mobilePhone"]
+
+        sync_custom_attrs_dict = {
+            "zulip": {
+                "saml": {
+                    "phone_number": "mobilePhone",
+                }
+            }
+        }
+
+        with self.settings(
+            SOCIAL_AUTH_SAML_ENABLED_IDPS=idps_dict,
+            SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT=sync_custom_attrs_dict,
+        ):
+            account_data_dict = self.get_account_data_dict(email=self.email, name=self.name)
+            result = self.social_auth_test(
+                account_data_dict,
+                subdomain="zulip",
+                extra_attributes=dict(mobilePhone=["123412341234"], birthday=["2021-01-01"]),
+            )
+        data = load_subdomain_token(result)
+        self.assertEqual(data["email"], self.email)
+        self.assertEqual(data["full_name"], self.name)
+        self.assertEqual(data["subdomain"], "zulip")
+        self.assertEqual(result.status_code, 302)
+
+        phone_field = CustomProfileField.objects.get(
+            realm=self.user_profile.realm, name="Phone number"
+        )
+        phone_field_value = CustomProfileFieldValue.objects.get(
+            user_profile=self.user_profile, field=phone_field
+        ).value
+        self.assertEqual(phone_field_value, "123412341234")
+
+        # Verify the Birthday field doesn't get synced - because it isn't configured for syncing.
+        new_birthday_field_value = CustomProfileFieldValue.objects.get(
+            user_profile=self.user_profile, field=birthday_field
+        ).value
+        self.assertEqual(new_birthday_field_value, old_birthday_field_value)
+
+    def test_social_auth_custom_profile_field_sync_custom_field_not_existing(self) -> None:
+        sync_custom_attrs_dict = {
+            "zulip": {
+                "saml": {
+                    "title": "title",
+                    "phone_number": "mobilePhone",
+                }
+            }
+        }
+        self.assertFalse(
+            CustomProfileField.objects.filter(
+                realm=self.user_profile.realm, name__iexact="title"
+            ).exists()
+        )
+
+        idps_dict = copy.deepcopy(settings.SOCIAL_AUTH_SAML_ENABLED_IDPS)
+        idps_dict["test_idp"]["extra_attrs"] = ["mobilePhone", "title"]
+
+        with self.settings(
+            SOCIAL_AUTH_SAML_ENABLED_IDPS=idps_dict,
+            SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT=sync_custom_attrs_dict,
+        ):
+            account_data_dict = self.get_account_data_dict(email=self.email, name=self.name)
+            with self.assertLogs(self.logger_string, level="WARNING") as m:
+                result = self.social_auth_test(
+                    account_data_dict,
+                    subdomain="zulip",
+                    extra_attributes=dict(mobilePhone=["123412341234"], birthday=["2021-01-01"]),
+                )
+        data = load_subdomain_token(result)
+        self.assertEqual(data["email"], self.email)
+        self.assertEqual(data["full_name"], self.name)
+        self.assertEqual(data["subdomain"], "zulip")
+        self.assertEqual(result.status_code, 302)
+
+        self.assertEqual(
+            m.output,
+            [
+                self.logger_output(
+                    "Exception while syncing custom profile fields for "
+                    + f"user {self.user_profile.id}: Custom profile field with name title not found.",
+                    "warning",
+                )
+            ],
+        )
+
 
 class AppleAuthMixin:
     BACKEND_CLASS = AppleAuthBackend

zproject/backends.py

@@ -1439,6 +1439,8 @@ def social_associate_user_helper(
         # strip removes the unnecessary ' '
         return_data["full_name"] = f"{first_name or ''} {last_name or ''}".strip()
 
+    return_data["extra_attrs"] = kwargs["details"].get("extra_attrs", {})
+
     return user_profile
 
 
@@ -1547,6 +1549,28 @@ def social_auth_finish(
     else:
         is_signup = False
 
+    extra_attrs = return_data.get("extra_attrs", {})
+    attrs_by_backend = settings.SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT.get(realm.subdomain, {})
+    if user_profile is not None and extra_attrs and attrs_by_backend:
+        # This is only supported for SAML right now, though the design
+        # is meant to be easy to extend this to other backends if desired.
+        # Unlike with LDAP, here we can only do syncing during the authentication
+        # flow, as that's when the data is provided and we don't have a way to query
+        # for it otherwise.
+        assert backend.name == "saml"
+        custom_profile_field_name_to_attr_name = attrs_by_backend.get(backend.name, {})
+        custom_profile_field_name_to_value = {}
+        for field_name, attr_name in custom_profile_field_name_to_attr_name.items():
+            custom_profile_field_name_to_value[field_name] = extra_attrs.get(attr_name)
+        try:
+            sync_user_profile_custom_fields(user_profile, custom_profile_field_name_to_value)
+        except SyncUserException as e:
+            backend.logger.warning(
+                "Exception while syncing custom profile fields for user %s: %s",
+                user_profile.id,
+                str(e),
+            )
+
     # At this point, we have now confirmed that the user has
     # demonstrated control over the target email address.
     #
@@ -1966,6 +1990,24 @@ class AppleAuthBackend(SocialAuthMixin, AppleIdAuth):
             return None
 
 
+class ZulipSAMLIdentityProvider(SAMLIdentityProvider):
+    def get_user_details(self, attributes: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Overriden to support plumbing of additional Attributes
+        from the SAMLResponse.
+        """
+        result = super().get_user_details(attributes)
+
+        extra_attr_names = self.conf.get("extra_attrs", [])
+        result["extra_attrs"] = {}
+        for extra_attr_name in extra_attr_names:
+            result["extra_attrs"][extra_attr_name] = self.get_attr(
+                attributes=attributes, conf_key=None, default_attribute=extra_attr_name
+            )
+
+        return result
+
+
 @external_auth_method
 class SAMLAuthBackend(SocialAuthMixin, SAMLAuth):
     auth_backend_name = "SAML"
@@ -2002,6 +2044,12 @@ class SAMLAuthBackend(SocialAuthMixin, SAMLAuth):
                     del settings.SOCIAL_AUTH_SAML_ENABLED_IDPS[idp_name]
         super().__init__(*args, **kwargs)
 
+    def get_idp(self, idp_name: str) -> ZulipSAMLIdentityProvider:
+        """Given the name of an IdP, get a SAMLIdentityProvider instance.
+        Forked to use our subclass of SAMLIdentityProvider for more flexibility."""
+        idp_config = self.setting("ENABLED_IDPS")[idp_name]
+        return ZulipSAMLIdentityProvider(idp_name, **idp_config)
+
     def auth_url(self) -> str:
         """Get the URL to which we must redirect in order to
         authenticate the user. Overriding the original SAMLAuth.auth_url.

zproject/default_settings.py

@@ -103,6 +103,8 @@ SOCIAL_AUTH_APPLE_EMAIL_AS_USERNAME = True
 SOCIAL_AUTH_OIDC_ENABLED_IDPS: Dict[str, Dict[str, Optional[str]]] = {}
 SOCIAL_AUTH_OIDC_FULL_NAME_VALIDATED = False
 
+SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT: Dict[str, Dict[str, Dict[str, str]]] = {}
+
 # Other auth
 SSO_APPEND_DOMAIN: Optional[str] = None
 

zproject/prod_settings_template.py

@@ -413,6 +413,10 @@ SOCIAL_AUTH_SAML_ENABLED_IDPS: Dict[str, Any] = {
         "attr_last_name": "last_name",
         "attr_username": "email",
         "attr_email": "email",
+        ## List of additional attributes to fetch from the SAMLResponse.
+        ## These attributes will be available for synchronizing custom profile fields.
+        ## in SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT.
+        # "extra_attrs": ["title", "mobilePhone"],
         ##
         ## The "x509cert" attribute is automatically read from
         ## /etc/zulip/saml/idps/{idp_name}.crt; don't specify it here.
@@ -455,6 +459,16 @@ SOCIAL_AUTH_SAML_SUPPORT_CONTACT = {
     "emailAddress": ZULIP_ADMINISTRATOR,
 }
 
+# SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT = {
+#    "example_org": {
+#        "saml": {
+#            # Format: "<custom profile field name>": "<attribute name from extra_attrs above>"
+#            "title": "title",
+#            "phone_number": "mobilePhone",
+#        }
+#    }
+# }
+
 ########
 ## Apple authentication ("Sign in with Apple").
 ##