From f064cb0bf787dc212265e6ca5e974c0884f0d1cd Mon Sep 17 00:00:00 2001 From: Jose Mallma Date: Thu, 4 Dec 2025 18:21:07 +0100 Subject: [PATCH 1/4] [ECT-4454][prometheus] Add documentation for V2 webhook --- prometheus/README.md | 79 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 3 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 1bec439fb4b77..1c993cf79c0b5 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -56,7 +56,7 @@ Note: Bucket data for a given `` Prometheus histogram met ### Events -Prometheus Alertmanager alerts are automatically sent to your Datadog event stream following the webhook configuration. +Prometheus Alertmanager alerts are automatically sent to your Datadog event stream following the webhook configuration. See the [Prometheus Alertmanager](#prometheus-alertmanager) section for setup instructions. ### Service Checks @@ -66,8 +66,75 @@ The Prometheus check does not include any service checks. Send Prometheus Alertmanager alerts in the event stream. Natively, Alertmanager sends all alerts simultaneously to the configured webhook. To see alerts in Datadog, you must configure your instance of Alertmanager to send alerts one at a time. You can add a group-by parameter under `route` to have alerts grouped by the actual name of the alert rule. ### Setup + + + + 1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: + +{{< highlight yaml "hl_lines=3-5" >}} +receivers: +- name: datadog + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus +route: + group_by: ['alertname'] + group_wait: 10s + group_interval: 5m + receiver: datadog + repeat_interval: 3h +{{< /highlight >}} + +
+The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts sharing the same values for the specified labels will be grouped into a single notification. For more details on routing configuration, see the Prometheus Alertmanager documentation. +
+ +**Note**: This endpoint accepts only one event in the payload at a time. + +2. (Optional) Use matchers to redirect specific alerts to different receivers. Alertmanager supports matchers to route notifications based on any label. For more information on matcher syntax, see the [Alertmanager matcher documentation][12]. + +The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with the [Datadog On-Call][11] product and redirect pages to different teams: + +{{< highlight yaml "hl_lines=5 9 18-19" >}} +receivers: +- name: datadog-ops + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=ops +- name: datadog-db + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=database + +route: + group_by: ['alertname'] + group_wait: 10s + group_interval: 5m + receiver: datadog-ops + repeat_interval: 3h + routes: + - matchers: + - team="database" + receiver: datadog-db +{{< /highlight >}} + +
+Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure pages are marked as resolved. Note that resolved notifications may take up to the next group_interval to be sent. +
+ +3. Restart the Prometheus and Alertmanager services. + +```shell +sudo systemctl restart prometheus.service alertmanager.service ``` + + + + +1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: + +{{< highlight yaml "hl_lines=3-5" >}} receivers: - name: datadog webhook_configs: @@ -79,15 +146,19 @@ route: group_interval: 5m receiver: datadog repeat_interval: 3h -``` +{{< /highlight >}} **Note**: This endpoint accepts only one event in the payload at a time. 2. Restart the Prometheus and Alertmanager services. -``` + +```shell sudo systemctl restart prometheus.service alertmanager.service ``` + + + ## Troubleshooting Need help? Contact [Datadog support][7]. @@ -108,3 +179,5 @@ Need help? Contact [Datadog support][7]. [8]: https://www.datadoghq.com/blog/monitor-prometheus-metrics [9]: https://docs.datadoghq.com/agent/prometheus/ [10]: https://docs.datadoghq.com/developers/prometheus/ +[11]: https://docs.datadoghq.com/service_management/on-call/ +[12]: https://prometheus.io/docs/alerting/latest/configuration/#matcher From 620432615da19bf1e2a849b5b3bae5f236f7dc0f Mon Sep 17 00:00:00 2001 From: Jose Mallma Date: Fri, 5 Dec 2025 15:00:57 +0100 Subject: [PATCH 2/4] remove highlighted code --- prometheus/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 1c993cf79c0b5..c5d1daf1d4d99 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -72,7 +72,7 @@ Send Prometheus Alertmanager alerts in the event stream. Natively, Alertmanager 1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: -{{< highlight yaml "hl_lines=3-5" >}} +```yaml receivers: - name: datadog webhook_configs: @@ -84,7 +84,7 @@ route: group_interval: 5m receiver: datadog repeat_interval: 3h -{{< /highlight >}} +```
The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts sharing the same values for the specified labels will be grouped into a single notification. For more details on routing configuration, see the Prometheus Alertmanager documentation. @@ -96,7 +96,7 @@ The group_by parameter determines how alerts are grouped together w The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with the [Datadog On-Call][11] product and redirect pages to different teams: -{{< highlight yaml "hl_lines=5 9 18-19" >}} +```yaml receivers: - name: datadog-ops webhook_configs: @@ -117,7 +117,7 @@ route: - matchers: - team="database" receiver: datadog-db -{{< /highlight >}} +```
Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure pages are marked as resolved. Note that resolved notifications may take up to the next group_interval to be sent. @@ -134,7 +134,7 @@ sudo systemctl restart prometheus.service alertmanager.service 1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: -{{< highlight yaml "hl_lines=3-5" >}} +```yaml receivers: - name: datadog webhook_configs: @@ -146,7 +146,7 @@ route: group_interval: 5m receiver: datadog repeat_interval: 3h -{{< /highlight >}} +``` **Note**: This endpoint accepts only one event in the payload at a time. From ac17389956d2e7a3721a7c247787c3ef0448ffda Mon Sep 17 00:00:00 2001 From: Jose Mallma <163123546+josemallma-dd@users.noreply.github.com> Date: Wed, 10 Dec 2025 12:47:32 +0100 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com> --- prometheus/README.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index c5d1daf1d4d99..728ed5169a715 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -70,7 +70,7 @@ Send Prometheus Alertmanager alerts in the event stream. Natively, Alertmanager -1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: +1. Edit the `alertmanager.yml` configuration file to include the following: ```yaml receivers: @@ -87,14 +87,15 @@ route: ```
-The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts sharing the same values for the specified labels will be grouped into a single notification. For more details on routing configuration, see the Prometheus Alertmanager documentation. +
    +
  • The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts with matching values for the specified labels are combined into a single notification. For details on routing configuration, see the Prometheus Alertmanager documentation.
    • +
  • This endpoint accepts only one event in the payload at a time.
    • +
-**Note**: This endpoint accepts only one event in the payload at a time. +2. (Optional) Use matchers to redirect specific alerts to different receivers. Matchers allow routing based on any alert label. For syntax details, see the [Alertmanager matcher documentation][12]. -2. (Optional) Use matchers to redirect specific alerts to different receivers. Alertmanager supports matchers to route notifications based on any label. For more information on matcher syntax, see the [Alertmanager matcher documentation][12]. - -The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with the [Datadog On-Call][11] product and redirect pages to different teams: +The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with [Datadog On-Call][11] and redirect pages to different teams: ```yaml receivers: @@ -120,7 +121,7 @@ route: ```
-Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure pages are marked as resolved. Note that resolved notifications may take up to the next group_interval to be sent. +Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure that pages are marked as resolved. Note that resolved notifications may be delayed until the next group_interval.
3. Restart the Prometheus and Alertmanager services. @@ -132,7 +133,7 @@ sudo systemctl restart prometheus.service alertmanager.service -1. Edit the Alertmanager configuration file, `alertmanager.yml`, to include the following: +1. Edit the `alertmanager.yml` configuration file to include the following: ```yaml receivers: @@ -148,7 +149,9 @@ route: repeat_interval: 3h ``` -**Note**: This endpoint accepts only one event in the payload at a time. +
+This endpoint accepts only one event in the payload at a time. +
2. Restart the Prometheus and Alertmanager services. From 9e2a5c78f35bb5f0ee9165423cda35e7f44f039c Mon Sep 17 00:00:00 2001 From: Jose Mallma Date: Wed, 10 Dec 2025 12:59:12 +0100 Subject: [PATCH 4/4] add correct indentation --- prometheus/README.md | 142 +++++++++++++++++++++---------------------- 1 file changed, 71 insertions(+), 71 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 728ed5169a715..6b97a8aabc00d 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -72,92 +72,92 @@ Send Prometheus Alertmanager alerts in the event stream. Natively, Alertmanager 1. Edit the `alertmanager.yml` configuration file to include the following: -```yaml -receivers: -- name: datadog - webhook_configs: - - send_resolved: true - url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus -route: - group_by: ['alertname'] - group_wait: 10s - group_interval: 5m - receiver: datadog - repeat_interval: 3h -``` - -
-
    -
  • The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts with matching values for the specified labels are combined into a single notification. For details on routing configuration, see the Prometheus Alertmanager documentation.
    • -
  • This endpoint accepts only one event in the payload at a time.
    • -
-
+ ```yaml + receivers: + - name: datadog + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus + route: + group_by: ['alertname'] + group_wait: 10s + group_interval: 5m + receiver: datadog + repeat_interval: 3h + ``` + +
+
    +
  • The group_by parameter determines how alerts are grouped together when sent to Datadog. Alerts with matching values for the specified labels are combined into a single notification. For details on routing configuration, see the Prometheus Alertmanager documentation.
    • +
  • This endpoint accepts only one event in the payload at a time.
    • +
+
2. (Optional) Use matchers to redirect specific alerts to different receivers. Matchers allow routing based on any alert label. For syntax details, see the [Alertmanager matcher documentation][12]. -The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with [Datadog On-Call][11] and redirect pages to different teams: - -```yaml -receivers: -- name: datadog-ops - webhook_configs: - - send_resolved: true - url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=ops -- name: datadog-db - webhook_configs: - - send_resolved: true - url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=database - -route: - group_by: ['alertname'] - group_wait: 10s - group_interval: 5m - receiver: datadog-ops - repeat_interval: 3h - routes: - - matchers: - - team="database" - receiver: datadog-db -``` - -
-Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure that pages are marked as resolved. Note that resolved notifications may be delayed until the next group_interval. -
+ The V2 webhook supports additional query parameters. For example, use the `oncall_team` parameter to integrate with [Datadog On-Call][11] and redirect pages to different teams: + + ```yaml + receivers: + - name: datadog-ops + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=ops + - name: datadog-db + webhook_configs: + - send_resolved: true + url: https://event-management-intake.datadoghq.com/api/v2/events/webhook?dd-api-key=&integration_id=prometheus&oncall_team=database + + route: + group_by: ['alertname'] + group_wait: 10s + group_interval: 5m + receiver: datadog-ops + repeat_interval: 3h + routes: + - matchers: + - team="database" + receiver: datadog-db + ``` + +
+ Setting send_resolved: true (the default value) enables Alertmanager to send notifications when alerts are resolved in Prometheus. This is particularly important when using the oncall_team parameter to ensure that pages are marked as resolved. Note that resolved notifications may be delayed until the next group_interval. +
3. Restart the Prometheus and Alertmanager services. -```shell -sudo systemctl restart prometheus.service alertmanager.service -``` + ```shell + sudo systemctl restart prometheus.service alertmanager.service + ``` 1. Edit the `alertmanager.yml` configuration file to include the following: -```yaml -receivers: -- name: datadog - webhook_configs: - - send_resolved: true - url: https://app.datadoghq.com/intake/webhook/prometheus?api_key= -route: - group_by: ['alertname'] - group_wait: 10s - group_interval: 5m - receiver: datadog - repeat_interval: 3h -``` - -
-This endpoint accepts only one event in the payload at a time. -
+ ```yaml + receivers: + - name: datadog + webhook_configs: + - send_resolved: true + url: https://app.datadoghq.com/intake/webhook/prometheus?api_key= + route: + group_by: ['alertname'] + group_wait: 10s + group_interval: 5m + receiver: datadog + repeat_interval: 3h + ``` + +
+ This endpoint accepts only one event in the payload at a time. +
2. Restart the Prometheus and Alertmanager services. -```shell -sudo systemctl restart prometheus.service alertmanager.service -``` + ```shell + sudo systemctl restart prometheus.service alertmanager.service + ```