This module is part of a larger suite of modules that provide alerts in Datadog. Other modules can be found on the Terraform Registry
We have two base modules we use to standardise development of our Monitor Modules:
- generic monitor Used in 90% of our alerts
- service check monitor
Modules are generated with this tool: https://github.com/kabisa/datadog-terraform-generator
Monitors:
| Monitor name | Default enabled | Priority | Query |
|---|---|---|---|
| CPU Load | True | 3 | avg(${var.cpu_usage_evaluation_period}):avg:apache.performance.cpu_load{${local.cpu_usage_filter}} by {service,host} > ${var.cpu_usage_critical} |
| Traffic | True | 3 | avg(last_15m):avg:apache.net.bytes_per_s{tag:xxx} by {service,host} > 100000 |
| Vhost Request Rate | True | 3 | avg(last_30m):per_minute(avg:apache.net.hits{tag:xxx,env:${var.env}} by {service,host}) > |
| Vhost Workers | True | 2 | max(last_15m):min:apache.performance.idle_workers{tag:xxx} by {service,host} < 1 |
pre-commit was used to do Terraform linting and validating.
Steps:
- Install pre-commit. E.g.
brew install pre-commit. - Run
pre-commit installin this repo. (Every time you clone a repo with pre-commit enabled you will need to run the pre-commit install command) - That’s it! Now every time you commit a code change (
.tffile), the hooks in thehooks:config.pre-commit-config.yamlwill execute.
Query:
avg(${var.cpu_usage_evaluation_period}):avg:apache.performance.cpu_load{${local.cpu_usage_filter}} by {service,host} > ${var.cpu_usage_critical}| variable | default | required | description |
|---|---|---|---|
| cpu_usage_enabled | True | No | |
| cpu_usage_warning | 75 | No | |
| cpu_usage_critical | 80 | No | |
| cpu_usage_evaluation_period | last_15m | No | |
| cpu_usage_note | "" | No | |
| cpu_usage_docs | "" | No | |
| cpu_usage_filter_override | "" | No | |
| cpu_usage_alerting_enabled | True | No | |
| cpu_usage_no_data_timeframe | None | No | |
| cpu_usage_notify_no_data | False | No | |
| cpu_usage_ok_threshold | None | No | |
| cpu_usage_name_prefix | None | No | |
| cpu_usage_name_suffix | None | No | |
| cpu_usage_priority | 3 | No | Number from 1 (high) to 5 (low). |
Query:
avg(last_15m):avg:apache.net.bytes_per_s{tag:xxx} by {service,host} > 100000| variable | default | required | description |
|---|---|---|---|
| traffic_enabled | True | No | |
| traffic_warning | 75000 | No | |
| traffic_critical | 100000 | No | |
| traffic_evaluation_period | last_15m | No | |
| traffic_note | "" | No | |
| traffic_docs | "" | No | |
| traffic_filter_override | "" | No | |
| traffic_alerting_enabled | True | No | |
| traffic_no_data_timeframe | None | No | |
| traffic_notify_no_data | False | No | |
| traffic_ok_threshold | None | No | |
| traffic_name_prefix | None | No | |
| traffic_name_suffix | None | No | |
| traffic_priority | 3 | No | Number from 1 (high) to 5 (low). |
This monitor raises alerts when it cannot successfully connect to an apache vhost.
| variable | default | required | description |
|---|---|---|---|
| vhost_availability_enabled | True | No | |
| vhost_availability_freshness_cycles | 3 | No | |
| vhost_availability_warning | 2 | No | |
| vhost_availability_critical | 3 | No | |
| vhost_availability_evaluation_period | last_5m | No | |
| vhost_availability_note | "" | No | |
| vhost_availability_docs | This monitor raises alerts when it cannot successfully connect to an apache vhost. | No | |
| vhost_availability_include_tags | [] | No | |
| vhost_availability_exclude_tags | [] | No | |
| vhost_availability_filter_override | "" | No | |
| vhost_availability_alerting_enabled | True | No | |
| vhost_availability_no_data_timeframe | 0 | No | |
| vhost_availability_notify_no_data | False | No | |
| vhost_availability_ok_threshold | 1 | No | |
| vhost_availability_name_prefix | "" | No | |
| vhost_availability_name_suffix | "" | No | |
| vhost_availability_priority | 2 | No | Number from 1 (high) to 5 (low). |
| vhost_require_full_window | False | No |
Query:
avg(last_30m):per_minute(avg:apache.net.hits{tag:xxx,env:${var.env}} by {service,host}) > | variable | default | required | description |
|---|---|---|---|
| vhost_request_rate_enabled | True | No | |
| vhost_request_rate_warning | None | No | |
| vhost_request_rate_critical | Yes | ||
| vhost_request_rate_evaluation_period | last_30m | No | |
| vhost_request_rate_note | "" | No | |
| vhost_request_rate_docs | "" | No | |
| vhost_request_rate_filter_override | "" | No | |
| vhost_request_rate_alerting_enabled | True | No | |
| vhost_request_rate_no_data_timeframe | None | No | |
| vhost_request_rate_notify_no_data | False | No | |
| vhost_request_rate_ok_threshold | None | No | |
| vhost_request_rate_name_prefix | None | No | |
| vhost_request_rate_name_suffix | None | No | |
| vhost_request_rate_priority | 3 | No | Number from 1 (high) to 5 (low). |
| vhost_request_new_group_delay | 300 | No | Time (in seconds) to skip evaluations for new groups. |
Query:
max(last_15m):min:apache.performance.idle_workers{tag:xxx} by {service,host} < 1| variable | default | required | description |
|---|---|---|---|
| vhost_workers_enabled | True | No | |
| vhost_workers_warning | 5 | No | |
| vhost_workers_critical | 1 | No | |
| vhost_workers_evaluation_period | last_15m | No | |
| vhost_workers_note | "" | No | |
| vhost_workers_docs | "" | No | |
| vhost_workers_filter_override | "" | No | |
| vhost_workers_alerting_enabled | True | No | |
| vhost_workers_no_data_timeframe | None | No | |
| vhost_workers_notify_no_data | False | No | |
| vhost_workers_ok_threshold | None | No | |
| vhost_workers_name_prefix | None | No | |
| vhost_workers_name_suffix | None | No | |
| vhost_workers_priority | 2 | No | Number from 1 (high) to 5 (low). |
| variable | default | required | description |
|---|---|---|---|
| filter_str | Yes | ||
| env | Yes | ||
| service | Yes | ||
| service_display_name | None | No | |
| notification_channel | Yes | ||
| additional_tags | [] | No | |
| locked | False | No | |
| name_prefix | "" | No | |
| name_suffix | "" | No |
