Skip to content

Prometheus: Refresh usage guide based on Docker Compose #401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Prometheus: Refresh usage guide based on Docker Compose #401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f43d09e
Prometheus: Refresh usage guide based on Docker Compose
amotl Oct 15, 2025
6310bb6
Prometheus: Implement suggestions by CodeRabbit
amotl Oct 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions docs/howto/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Instructions how to get tasks done with CrateDB.
:::{rubric} 2025
:::

- {ref}`prometheus-usage`
- {ref}`influxdb-usage`
- {ref}`amqp-usage`
- {ref}`mqtt-usage`
Expand Down Expand Up @@ -62,11 +63,6 @@ Instructions how to get tasks done with CrateDB.
- {ref}`terraform-usage`
- {ref}`trino-usage`

:::{rubric} 2022
:::

- {ref}`prometheus-usage`

Comment on lines -65 to -69
Copy link
Member Author

@amotl amotl Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refreshing this article did not take many efforts. It was merely just copy/paste from a series of others following the same scheme, then slotting in the Docker Compose file and configuration files from the original article. If you agree with those kinds of changes, we'd like to encourage you to find similar spots to modernize easily.


:::{note}
You can expect the more recent documents to be more up-to-date than previous
Expand Down
51 changes: 51 additions & 0 deletions docs/integrate/prometheus/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Service composition file for Docker Compose or Podman Compose
# https://cratedb.com/docs/guide/integrate/prometheus/

services:

cratedb:
image: "docker.io/crate:latest"
command: >
crate -Cdiscovery.type=single-node
ports:
- "4200:4200"
- "5432:5432"
volumes:
- cratedb-data:/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:4200/"]
interval: 5s
timeout: 30s
retries: 5
restart: unless-stopped

cratedb-ddl:
image: docker.io/crate/crate:latest
command: sh -c "crash --hosts http://cratedb:4200/ -c 'SELECT 1'; crash --hosts http://cratedb:4200/ < /var/ddl.sql"
volumes:
- ./ddl.sql:/var/ddl.sql
depends_on:
- cratedb

prometheus:
image: "docker.io/prom/prometheus:latest"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
restart: unless-stopped

cratedb-prometheus-adapter:
image: ghcr.io/crate/cratedb-prometheus-adapter:0.5.8
command: -config.file /etc/cratedb-prometheus-adapter.yaml
ports:
- "9268:9268/tcp"
volumes:
- ./cratedb-prometheus-adapter.yaml:/etc/cratedb-prometheus-adapter.yaml
depends_on:
- cratedb-ddl
- prometheus
restart: unless-stopped

volumes:
cratedb-data:
22 changes: 22 additions & 0 deletions docs/integrate/prometheus/cratedb-prometheus-adapter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# https://cratedb.com/docs/guide/integrate/prometheus/

cratedb_endpoints:
- host: "cratedb" # Host to connect to (default: "localhost").
port: 5432 # Port to connect to (default: 5432).
user: "crate" # Username to use (default: "crate")
password: "" # Password to use (default: "").
schema: "testdrive" # Schema to use (default: "").
max_connections: 0 # The maximum number of concurrent connections (default: runtime.NumCPU()).
# It will get forwarded to pgx's `pool_max_conns`, and determines
# the maximum number of connections in the connection pool for
# both connection pools (read and write).
read_pool_size_max: 0 # Configure the maximum pool size for read operations individually.
# (default: runtime.NumCPU())
write_pool_size_max: 0 # Configure the maximum pool size for write operations individually.
# (default: runtime.NumCPU())
connect_timeout: 10 # TCP connect timeout (seconds) (default: 10).
# It has the same meaning as libpq's `connect_timeout`.
read_timeout: 5 # Query context timeout for read queries (seconds) (default: 5).
write_timeout: 5 # Query context timeout for write queries (seconds) (default: 5).
enable_tls: false # Whether to connect using TLS (default: false).
allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).
11 changes: 11 additions & 0 deletions docs/integrate/prometheus/ddl.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
-- https://cratedb.com/docs/guide/integrate/prometheus/

CREATE TABLE IF NOT EXISTS "testdrive"."metrics" (
"timestamp" TIMESTAMP,
"labels_hash" STRING,
"labels" OBJECT(DYNAMIC),
"value" DOUBLE,
"valueRaw" LONG,
"day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
) PARTITIONED BY ("day__generated");
20 changes: 3 additions & 17 deletions docs/integrate/prometheus/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
```{div} .clearfix
```


## About
:::{rubric} About
:::

[Prometheus] is an open-source systems monitoring and alerting toolkit
for collecting metrics data from applications and infrastructures.
Expand Down Expand Up @@ -37,8 +37,6 @@ Prometheus's main features are:
::::


## Learn

:::{rubric} Remote Endpoints and Storage
:::

Expand All @@ -57,15 +55,10 @@ massively scale-out Prometheus.

::::{grid}

:::{grid-item-card} Blog: Getting started
:link: https://cratedb.com/blog/getting-started-prometheus-cratedb-long-term-storage
:link-type: url
Getting started with Prometheus and CrateDB for long-term storage.
:::

Comment on lines -60 to -65
Copy link
Member Author

@amotl amotl Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaelkremmel: The article Getting Started With Prometheus and CrateDB for Long-Term Storage still uses CrateDB 2.2 and already bears the admonition This article is more than 4 years old. That's fine, but let's stop referring to it hereby.

Now that the canonical documentation page will be modernized, can you prominently refer here at the top of the page instead?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for letting me know @amotl. I unpublished the blog post and created a redirect to the documentation. No need to keep old blog posts if they could confuse readers and also LLMs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, let's review and bring in the updated content (this patch) to make the modernization complete.

:::{grid-item-card} Store Prometheus long-term metrics into CrateDB
:link: prometheus-usage
:link-type: ref
:columns: 6
Set up CrateDB as a long-term metrics store for Prometheus using Docker Compose.
:::

Expand Down Expand Up @@ -99,13 +92,6 @@ usage guide.
::::


:::{rubric} Repositories
:::

[CrateDB] •
[Prometheus] •
[CrateDB Prometheus Adapter]


```{seealso}
[CrateDB and Prometheus]
Expand Down
26 changes: 26 additions & 0 deletions docs/integrate/prometheus/prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# https://cratedb.com/docs/guide/integrate/prometheus/

global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any time-series scraped from this config.
- job_name: 'prometheus'

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s

static_configs:
- targets: ['prometheus:9090']

remote_write:
- url: http://cratedb-prometheus-adapter:9268/write
remote_read:
- url: http://cratedb-prometheus-adapter:9268/read
167 changes: 26 additions & 141 deletions docs/integrate/prometheus/usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,149 +2,40 @@
# Store Prometheus long-term metrics into CrateDB

This usage guide shows how to use Docker Compose to run the services
CrateDB, Prometheus, and the CrateDB Prometheus Adapter.

## Motivation

[Prometheus] is a monitoring system that collects metrics from applications and
infrastructure. It focuses on ingesting large volumes of concise time‑series
events—timestamped points with key‑value labels.

This data helps you track a system’s state and trajectory. Many Prometheus users
want to retain it long term.

[CrateDB] helps here. With the [CrateDB Prometheus Adapter], you can store metrics
in CrateDB and use its fast ingestion and query performance to scale Prometheus.

## Set up Docker Compose

Run CrateDB, Prometheus, and the CrateDB Prometheus Adapter as [Docker containers].
Use [Docker Compose] to centralize container management so you can build and run
all containers with one command and define their connections in a YAML file.

Install Docker by following the [Docker installation guide].

Create a directory on your machine to host the configuration files.
Create three YAML files with your preferred editor and save them with the `.yml` extension.

### Create `docker-compose.yml`

Create `docker-compose.yml` to configure the three containers.
Define services for CrateDB, Prometheus, and the adapter. Mount `config.yml`
and `prometheus.yml` into the adapter and Prometheus containers, respectively.
You will create these files in the following steps.
```yaml
services:
cratedb:
image: "docker.io/crate:latest"
command: >
crate -Cdiscovery.type=single-node
ports:
- "4200:4200"
- "5432:5432"
volumes:
- cratedb-data:/data
restart: unless-stopped
prometheus:
image: "docker.io/prom/prometheus:latest"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
restart: unless-stopped
cratedb-prometheus-adapter:
image: "ghcr.io/crate/cratedb-prometheus-adapter:latest"
volumes:
- ./config.yml:/etc/cratedb-prometheus-adapter/config.yml
ports:
- "9268:9268"
depends_on:
- cratedb
- prometheus
restart: unless-stopped
volumes:
cratedb-data:
```

### Create `prometheus.yml`

Next, create `prometheus.yml` following the [Prometheus documentation].
This file defines the services that Prometheus scrapes.

To keep it simple, monitor Prometheus itself.

To forward samples to CrateDB, set `remote_write` and `remote_read` to the adapter URL (see the [CrateDB Prometheus Adapter setup](https://github.com/crate/cratedb-prometheus-adapter)).
CrateDB, Prometheus, and the [CrateDB Prometheus Adapter].

Because the adapter runs in Docker, use the adapter service name from `docker-compose.yml` (`cratedb-prometheus-adapter`) instead of `localhost` in the URLs.
## Prerequisites

Use the following `prometheus.yml`:
```yaml
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
Use Docker or Podman to run all components. This approach works consistently
across Linux, macOS, and Windows.

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
### Files

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any time-series scraped from this config.
- job_name: 'prometheus'
First, download and save all required files to your machine.
- {download}`compose.yaml`
- {download}`cratedb-prometheus-adapter.yaml`
- {download}`ddl.sql`
- {download}`prometheus.yml`

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
### Services

static_configs:
- targets: ['prometheus:9090']
Start services using Docker Compose or Podman Compose.
If you use Podman, replace `docker` with `podman` (or enable the podman‑docker
compatibility shim) and run `podman compose up`.

remote_write:
- url: http://cratedb-prometheus-adapter:9268/write
remote_read:
- url: http://cratedb-prometheus-adapter:9268/read
```
### Create `config.yml`

Finally, create `config.yml` following the [CrateDB Prometheus Adapter].
This file defines the CrateDB endpoints that the adapter writes to.

Set the host to `cratedb` (the service name in `docker-compose.yml`) instead of `localhost`. Keep the remaining variables at their defaults.
```yaml
cratedb_endpoints:
- host: "cratedb" # Host to connect to (default: "localhost")
port: 5432 # Port to connect to (default: 5432).
user: "crate" # Username to use (default: "crate")
password: "" # Password to use (default: "").
schema: "" # Schema to use (default: "").
connect_timeout: 10 # TCP connect timeout (seconds) (default: 10).
max_connections: 5 # The maximum number of concurrent connections (default: 5).
enable_tls: false # Whether to connect using TLS (default: false).
allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).
```shell
docker compose up
```
Place `docker-compose.yml`, `config.yml`, and `prometheus.yml` in the same directory on your machine.

## Start the services
## Explore data

Finally, navigate to your CrateDB–Prometheus directory and start Docker Compose:
CrateDB stores the metrics in the designated table, ready for inspection and analysis.
By default, the included `ddl.sql` creates the table `testdrive.metrics`.
```shell
cd /Users/Path/To/Directory/CrateDB-Prometheus
docker-compose up
docker compose exec cratedb crash -c "SELECT * FROM testdrive.metrics ORDER BY timestamp LIMIT 5;"
```

After Docker Compose starts, open the CrateDB Admin UI at `http://localhost:4200`
and create a `metrics` table to store metrics gathered by Prometheus.
```sql
CREATE TABLE "metrics" (
"timestamp" TIMESTAMP,
"labels_hash" STRING,
"labels" OBJECT(DYNAMIC),
"value" DOUBLE,
"valueRaw" LONG,
"day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
) PARTITIONED BY ("day__generated")
```
## Screenshots

Navigate to `http://localhost:9090` to open the Prometheus UI. Go to **Status** → **Targets**.

Expand All @@ -154,25 +45,19 @@ Confirm that Prometheus monitors itself.

![Prometheus target details for the self-scrape job](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/57ccb5374b0ab524466de08feefbafde559dac87.png)

Return to the CrateDB Admin UI and select the `metrics` table you created.
Return to the CrateDB Admin UI at `http://localhost:4200` and select the `testdrive.metrics` table.

After a few minutes, Prometheus will have gathered hundreds of thousands of data points.
After a few minutes, Prometheus will have gathered a few thousand data points.

![CrateDB Admin UI showing the populated metrics table](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/22e8c7d5a90ec9240a4cb4269774e143759aa92e.jpeg)

Use CrateDB’s query engine to analyze and visualize this data with tools
like {ref}`grafana`.

Explore these related resources:

* [Visualizing time‑series data with Grafana and CrateDB](https://cratedb.com/blog/visualizing-time-series-data-with-grafana-and-cratedb)
* [Monitoring CrateDB with Prometheus and Grafana](https://cratedb.com/blog/monitoring-cratedb-with-prometheus-and-grafana)
like {ref}`grafana`. See also
[Monitoring a self-managed CrateDB cluster with Prometheus and Grafana].


[CrateDB]: https://cratedb.com/database
[CrateDB Prometheus Adapter]: https://github.com/crate/cratedb-prometheus-adapter
[Docker Compose]: https://docs.docker.com/compose/
[Docker containers]: https://www.docker.com/resources/what-container
[Docker installation guide]: https://docs.docker.com/get-docker/
[Monitoring a self-managed CrateDB cluster with Prometheus and Grafana]: https://community.cratedb.com/t/monitoring-a-self-managed-cratedb-cluster-with-prometheus-and-grafana/1236
[Prometheus]: https://prometheus.io/docs/introduction/overview/
[Prometheus documentation]: https://prometheus.io/docs/prometheus/latest/getting_started/