Home

ClickhouseMemoryUsageCritical

Meaning

ClickHouse on the dev environment is using more than 95% of the available memory consistently for over 10 minutes. This means the server is close to running out of RAM.

Full context

This alert indicates that the ClickHouse instance in the development environment is consuming more than 95% of the available system memory for at least 10 minutes. Even though it's not production, this environment is still actively used for development, testing, or staging purposes. A memory spike like this might be caused by unoptimized queries, large result sets, data loading operations, or misconfigured settings. Persistent high memory usage can impact the stability of the dev environment and delay ongoing work.

Impact

• Slower query execution and degraded application performance
• Potential for service crashes or forced restarts
• Disruption to development and CI/CD pipelines
• Risk of masking issues that could replicate in production if not addressed early

Diagnosis

• Review recent queries or workloads that may be memory-intensive (e.g., large joins, aggregations)
• Check for data ingestion jobs or background merges running at the time of the alert
• Validate ClickHouse memory limits and configurations (e.g., max_memory_usage, max_bytes_before_external_group_by)
• Inspect logs for signs of memory allocation errors or throttling

Mitigation

• Optimize or throttle problematic queries; limit result set sizes
• Pause or reschedule heavy data load jobs during peak usage
• Adjust memory-related settings or scale resources if needed
• Restart the ClickHouse instance only if necessary and during a safe window
• Set up memory usage dashboards to monitor trends and adjust proactively