integrations: Move configs in config_options to url_options.

Currently we have 2 implementations of `config_options`:

- It's used for generating optional webhook URL parameters. These
settings also come with custom UI in the "Generate integration URL"
modal.

- In `/bots` API, it's used as schema for the bots `BotConfigData`. Each
type of bots have different ways of defining their `BotConfigData`
fields. Currently, only embedded bots use `BotConfigData`, and only the
incoming webhooks use `config_options` to configure a bot's
`BotConfigData`; thus, the `config_options` remain unused.

To avoid confusion as to which implementation of `config_options` is
used by an integration, this separates the first use case -- to generate
optional webhook URL -- to a new field called `url_options`. Thus, the
`config_options` field is reserved only for the second use case.

api_docs/incoming-webhooks-walkthrough.md

@@ -210,26 +210,54 @@ tools which you can use to test your webhook - 2 command line tools and a GUI.
 
 ### Webhooks requiring custom configuration
 
-In rare cases, it's necessary for an incoming webhook to require
+In cases where an incoming webhook integration supports optional URL parameters,
-additional user configuration beyond what is specified in the post
+one can use the `url_options` feature. It's a field in the `WebhookIntegration`
-URL.  The typical use case for this is APIs like the Stripe API that
+class that is used when [generating a URL for an integration](/help/generate-integration-url)
-require clients to do a callback to get details beyond an opaque
+in the web app, which encodes the user input for each URL parameter in the
-object ID that one would want to include in a Zulip notification.
+incoming webhook's URL.
 
-These configuration options are declared as follows:
+These URL options are declared as follows:
 
 ```python
-    WebhookIntegration('helloworld', ['misc'], display_name='Hello World',
+    WebhookIntegration(
-                       config_options=[('HelloWorld API key', 'hw_api_key', check_string)])
+        'helloworld',
+        ...
+        url_options=[
+          WebhookUrlOption(
+            name='ignore_private_repositories',
+            label='Exclude notifications from private repositories',
+            validator=check_string
+          ),
+        ],
+    )
 ```
 
-`config_options` is a list describing the parameters the user should
+`url_options` is a list describing the parameters the web app UI should offer when
-configure:
+generating the incoming webhook URL:
-    1. A user-facing string describing the field to display to users.
-    2. The field name you'll use to access this from your `view.py` function.
-    3. A Validator, used to verify the input is valid.
 
-Common validators are available in `zerver/lib/validators.py`.
+  - `name`: The parameter name that is used to encode the user input in the
+    integration's webhook URL.
+  - `label`: A short descriptive label for this URL parameter in the web app UI.
+  - `validator`: A validator function, which is used to determine the input type
+    for this option in the UI, and to indicate how to validate the input.
+    Currently, the web app UI only supports these validators:
+      - `check_bool` for checkbox/select input.
+      - `check_string` for text input.
+
+!!! warn ""
+
+    **Note**: To add support for other validators, you can update
+    `web/src/integration_url_modal.ts`. Common validators are available in
+    `zerver/lib/validator.py`.
+
+In rare cases, it may be necessary for an incoming webhook to require
+additional user configuration beyond what is specified in the POST
+URL. A typical use case for this would be APIs that require clients
+to do a callback to get details beyond an opaque object ID that one
+would want to include in a Zulip notification message.
+
+The `config_options` field in the `WebhookIntegration` class is reserved
+for this use case.
 
 ## Step 4: Manually testing the webhook
 

api_docs/unmerged.d/ZF-f9d19d.md

@@ -0,0 +1,7 @@
+* [`POST /register`](/api/register-queue): Added a `url_options` object
+  to the `realm_incoming_webhook_bots` object for incoming webhook
+  integration URL parameter options. Previously, these optional URL
+  parameters were included in the `config_options` field (see feature
+  level 318 entry). The `config_options` object is now reserved for
+  configuration data that can be set when creating an bot user for a
+  specific incoming webhook integration.

web/src/integration_url_modal.ts

@@ -19,19 +19,19 @@ import {realm} from "./state_data.ts";
 import * as stream_data from "./stream_data.ts";
 import * as util from "./util.ts";
 
-type ConfigOption = {
+type UrlOption = {
     key: string;
     label: string;
     validator: string;
 };
 
-const config_option_schema = z.object({
+const url_option_schema = z.object({
     key: z.string(),
     label: z.string(),
     validator: z.string(),
 });
 
-const config_options_schema = z.array(config_option_schema);
+const url_options_schema = z.array(url_option_schema);
 
 export function show_generate_integration_url_modal(api_key: string): void {
     const default_url_message = $t_html({defaultMessage: "Integration URL will appear here."});
@@ -107,8 +107,8 @@ export function show_generate_integration_url_modal(api_key: string): void {
             update_url();
         }
 
-        function render_config(config: ConfigOption[]): void {
+        function render_url_options(config: UrlOption[]): void {
-            const validated_config = config_options_schema.parse(config);
+            const validated_config = url_options_schema.parse(config);
             $config_container.empty();
 
             for (const option of validated_config) {
@@ -203,7 +203,7 @@ export function show_generate_integration_url_modal(api_key: string): void {
                 (bot) => bot.name === selected_integration,
             );
             const all_event_types = selected_integration_data?.all_event_types;
-            const config = selected_integration_data?.config_options;
+            const url_options = selected_integration_data?.url_options;
 
             if (all_event_types !== null) {
                 $("#integration-events-parameter").removeClass("hide");
@@ -234,8 +234,8 @@ export function show_generate_integration_url_modal(api_key: string): void {
 
             const selected_events = set_events_param(params);
 
-            if (config) {
+            if (url_options) {
-                for (const option of config) {
+                for (const option of url_options) {
                     let $input_element;
                     if (
                         option.key === "branches" &&
@@ -316,8 +316,8 @@ export function show_generate_integration_url_modal(api_key: string): void {
                 (bot) => bot.name === selected_integration,
             );
 
-            if (selected_integration_data?.config_options) {
+            if (selected_integration_data?.url_options) {
-                render_config(selected_integration_data.config_options);
+                render_url_options(selected_integration_data.url_options);
             }
 
             dropdown.hide();

web/src/state_data.ts

@@ -386,6 +386,15 @@ export const realm_schema = z.object({
                     }),
                 )
                 .optional(),
+            url_options: z
+                .array(
+                    z.object({
+                        key: z.string(),
+                        label: z.string(),
+                        validator: z.string(),
+                    }),
+                )
+                .optional(),
         }),
     ),
     realm_inline_image_preview: z.boolean(),

zerver/lib/events.py

@@ -750,6 +750,16 @@ def fetch_initial_state_data(
                 ]
                 if integration.config_options
                 else [],
+                "url_options": [
+                    {
+                        "key": c.name,
+                        "label": c.label,
+                        "validator": c.validator.__name__,
+                    }
+                    for c in integration.url_options
+                ]
+                if integration.url_options
+                else [],
             }
             for integration in WEBHOOK_INTEGRATIONS
             if integration.legacy is False

zerver/lib/integrations.py

@@ -13,7 +13,7 @@ from django_stubs_ext import StrPromise
 
 from zerver.lib.storage import static_path
 from zerver.lib.validator import check_bool, check_string
-from zerver.lib.webhooks.common import WebhookConfigOption
+from zerver.lib.webhooks.common import WebhookConfigOption, WebhookUrlOption
 
 """This module declares all of the (documented) integrations available
 in the Zulip server.  The Integration class is used as part of
@@ -79,12 +79,14 @@ class Integration:
         stream_name: str | None = None,
         legacy: bool = False,
         config_options: Sequence[WebhookConfigOption] = [],
+        url_options: Sequence[WebhookUrlOption] = [],
     ) -> None:
         self.name = name
         self.client_name = client_name if client_name is not None else name
         self.secondary_line_text = secondary_line_text
         self.legacy = legacy
         self.doc = doc
+        self.url_options = url_options
 
         # Note: Currently only incoming webhook type bots use this list for
         # defining how the bot's BotConfigData should be. Embedded bots follow
@@ -247,6 +249,7 @@ class WebhookIntegration(Integration):
         stream_name: str | None = None,
         legacy: bool = False,
         config_options: Sequence[WebhookConfigOption] = [],
+        url_options: Sequence[WebhookUrlOption] = [],
         dir_name: str | None = None,
     ) -> None:
         if client_name is None:
@@ -261,6 +264,7 @@ class WebhookIntegration(Integration):
             stream_name=stream_name,
             legacy=legacy,
             config_options=config_options,
+            url_options=url_options,
         )
 
         if function is None:
@@ -410,9 +414,7 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         "azuredevops",
         ["version-control"],
         display_name="AzureDevOps",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration("beanstalk", ["version-control"], stream_name="commits"),
     WebhookIntegration("basecamp", ["project-management"]),
@@ -423,9 +425,7 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         logo="images/integrations/logos/bitbucket.svg",
         display_name="Bitbucket Server",
         stream_name="bitbucket",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration(
         "bitbucket2",
@@ -433,9 +433,7 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         logo="images/integrations/logos/bitbucket.svg",
         display_name="Bitbucket",
         stream_name="bitbucket",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration(
         "bitbucket",
@@ -464,9 +462,7 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         "gitea",
         ["version-control"],
         stream_name="commits",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration(
         "github",
@@ -474,11 +470,11 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         display_name="GitHub",
         function="zerver.webhooks.github.view.api_github_webhook",
         stream_name="github",
-        config_options=[
+        url_options=[
-            WebhookConfigOption(name="branches", description="", validator=check_string),
+            WebhookUrlOption(name="branches", label="", validator=check_string),
-            WebhookConfigOption(
+            WebhookUrlOption(
                 name="ignore_private_repositories",
-                description="Exclude notifications from private repositories",
+                label="Exclude notifications from private repositories",
                 validator=check_bool,
             ),
         ],
@@ -497,18 +493,14 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         "gitlab",
         ["version-control"],
         display_name="GitLab",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration("gocd", ["continuous-integration"], display_name="GoCD"),
     WebhookIntegration(
         "gogs",
         ["version-control"],
         stream_name="commits",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration("gosquared", ["marketing"], display_name="GoSquared"),
     WebhookIntegration("grafana", ["monitoring"]),
@@ -543,10 +535,10 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
     WebhookIntegration(
         "opsgenie",
         ["meta-integration", "monitoring"],
-        config_options=[
+        url_options=[
-            WebhookConfigOption(
+            WebhookUrlOption(
                 name="eu_region",
-                description="Use Opsgenie's European service region",
+                label="Use Opsgenie's European service region",
                 validator=check_bool,
             )
         ],
@@ -563,9 +555,7 @@ WEBHOOK_INTEGRATIONS: list[WebhookIntegration] = [
         "rhodecode",
         ["version-control"],
         display_name="RhodeCode",
-        config_options=[
+        url_options=[WebhookUrlOption(name="branches", label="", validator=check_string)],
-            WebhookConfigOption(name="branches", description="", validator=check_string)
-        ],
     ),
     WebhookIntegration("rundeck", ["deployment"]),
     WebhookIntegration("semaphore", ["continuous-integration", "deployment"]),

zerver/lib/webhooks/common.py

@@ -61,6 +61,13 @@ class WebhookConfigOption:
     validator: Callable[[str, str], str | bool | None]
 
 
+@dataclass
+class WebhookUrlOption:
+    name: str
+    label: str
+    validator: Callable[[str, str], str | bool | None]
+
+
 def get_setup_webhook_message(integration: str, user_name: str | None = None) -> str:
     content = SETUP_MESSAGE_TEMPLATE.format(integration=integration)
     if user_name:

zerver/openapi/zulip.yaml

@@ -16351,6 +16351,8 @@ paths:
                                 **Changes**: New in Zulip 8.0 (feature level 207).
                             config_options:
                               $ref: "#/components/schemas/WebhookConfigOption"
+                            url_options:
+                              $ref: "#/components/schemas/WebhookUrlOption"
                       recent_private_conversations:
                         description: |
                           Present if `recent_private_conversations` is present in `fetch_event_types`.
@@ -24709,21 +24711,20 @@ components:
     WebhookConfigOption:
       type: array
       description: |
-        An array of configuration options where each option is an
+        An array of configuration options that can be set when creating
-        object containing a unique identifier, a human-readable name,
+        a bot user for this incoming webhook integration.
-        and a validation function name hinting how to verify the
-        correct input format.
 
-        This is an unstable API expected to be used only by the Zulip web
+        This is an unstable API. Please discuss in chat.zulip.org before
-        apps. Please discuss in chat.zulip.org before using it.
+        using it.
 
-        **Changes**: In Zulip 10.0 (feature level 318), changed `config`
+        **Changes**: As of Zulip 11.0 (feature level ZF-f9d19d), this
-        to `config_options`. The `config_options` field defines which options
+        object is reserved for integration-specific configuration options
-        should be offered when creating URLs for this integration. Previously,
+        that can be set when creating a bot user. Previously, this object
-        the `config` field was a key-value pair describing integration-specific
+        also included optional webhook URL parameters, which are now
-        configuration data that needs to be included when creating an incoming
+        specified in the `url_options` object.
-        webhook bot. The feature was never operational because there is currently
+
-        no way to associate an incoming webhook bot with an integration.
+        Before Zulip 10.0 (feature level 318), this field was named `config`,
+        and was reserved for configuration data key-value pairs.
       items:
         type: object
         additionalProperties: false
@@ -24731,7 +24732,7 @@ components:
           key:
             type: string
             description: |
-              A key for the configuration option to use in generated URLs.
+              A key for the configuration option.
           label:
             type: string
             description: |
@@ -24740,8 +24741,38 @@ components:
             type: string
             description: |
               The name of the validator function for the configuration
-              option. Currently generated values are `check_bool` and
+              option.
-              `check_string`.
+    WebhookUrlOption:
+      type: array
+      description: |
+        An array of optional URL parameter options for the incoming webhook
+        integration. In the web app, these are used when
+        [generating a URL for an integration](/help/generate-integration-url).
+
+        This is an unstable API expected to be used only by the Zulip web
+        app. Please discuss in chat.zulip.org before using it.
+
+        **Changes**: New in Zulip 11.0 (feature level ZF-f9d19d). Previously,
+        these optional URL parameter options were included in the
+        `config_options` object.
+      items:
+        type: object
+        additionalProperties: false
+        properties:
+          key:
+            type: string
+            description: |
+              The parameter variable to encode the users input for this
+              option in the integrations webhook URL.
+          label:
+            type: string
+            description: |
+              A human-readable label of the url option.
+          validator:
+            type: string
+            description: |
+              The name of the validator function for the configuration
+              option.
     CustomProfileField:
       type: object
       additionalProperties: false