Inference Watch

Live safety monitoring (Prometheus + Grafana) and durable cross-run comparison (Aim + Postgres) for any long-running training or inference loop, whether that's RL/LLM training, HFT strategy runs, or anything else that emits a stream of step-level metrics.

Requirements

• Docker and Docker Compose
• Python 3.10+

Install

Clone the repo and install the Python dependencies:

git clone https://github.com/jacksonwade/observability.git
cd observability
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Start the stack:

docker compose -f compose.infra.yaml up

This brings up Prometheus, Grafana, Alertmanager, a Pushgateway, Postgres, and a small static dashboard, each on the ports below.

Run the example training loop to check everything is wired up:

python train_example.py

Then, in a separate terminal

aim up 

Ports

Service	Default port	Env var to override
Prometheus	9090	PROMETHEUS_HOST_PORT
Pushgateway	9091	PUSHGATEWAY_HOST_PORT
Grafana	3000	GRAFANA_HOST_PORT
Alertmanager	9093	ALERTMANAGER_HOST_PORT
Static dashboard	8088	DASHBOARD_HOST_PORT
Postgres	5432	POSTGRES_HOST_PORT
Aim UI	43800	not a compose service, see below

Override any of these to run a second copy of the stack alongside the first without port collisions, for example:

PROMETHEUS_HOST_PORT=9190 GRAFANA_HOST_PORT=3100 DASHBOARD_HOST_PORT=8188 \
  docker compose -f compose.infra.yaml up

Once running:

• Grafana: http://localhost:3000
• Static dashboard: http://localhost:8088
• Aim UI: http://localhost:43800 (after running aim up, see below)

Durable storage (optional)

Postgres is bundled in compose.infra.yaml. To use it, set the OBSERVABILITY_DSN environment variable to your Postgres DSN before starting a training run:

export OBSERVABILITY_DSN="postgresql://observability:observability@localhost:5432/observability"

If you already have a Postgres instance running elsewhere, point OBSERVABILITY_DSN at it and remove the postgres service from compose.infra.yaml instead of running the bundled one.

Aim logging turns on automatically if the aim package is installed (included in requirements.txt), independent of the Postgres DSN. Aim is not a Docker Compose service, it's a Python package that writes run data to a local .aim/ repo. To browse it, run its own UI server from the repo directory:

aim up

This starts the Aim UI at http://localhost:43800. Pass --port to change it if 43800 is taken, and --repo if aim_repo was set to something other than the default when Monitor was created.

Comparing runs after the fact

The dashboard has a second page, Runs (http://localhost:8088/runs.html), that does the after-the-fact half of the job without leaving the web app: pick recorded runs on the left and read one chart per metric with one line per run. The x axis switches between step and relative time (every run starts at zero, so wall-clock efficiency compares directly), smoothing is an overlay that keeps the raw line visible underneath, and a shareable URL carries the whole view: selection, colors, axis, smoothing.

Counters like tokens_generated_total draw as rates with the cumulative total in the legend. When a long run has to be downsampled, every bucket keeps its min and max, so a one-step loss spike still shows, and the panel says how many points it drew out of how many rows.

The page reads from the same Postgres store the Monitor writes to (see "Durable storage" above), through a small read-only JSON API (monitoring/runs_api.py, the runs-api service in compose.infra.yaml) that nginx proxies under /api/runs. Runs recorded without OBSERVABILITY_DSN set never reach Postgres, so they will not appear here; the page explains as much when storage is off.

Basic usage

from observability.monitoring.metrics import Monitor

with Monitor(run_id, mode="scrape") as mon:
    for step in range(steps):
        mon.log(step, loss=loss, accuracy=acc, kl=kl, tokens=n_tokens,
                 checks={"compile": True, "lint": False})

mode="scrape" (default) starts a local Prometheus HTTP server on prom_port (default 8000) for Prometheus to pull from. mode="push" pushes to a Pushgateway instead, for training processes Prometheus can't reach directly (pushgateway="host:9091" is required in this mode).

Static dashboard (webdash)

webdash is a zero-build, zero-JS-dependency live view of your metrics, served by the dashboard service on http://localhost:8088. It's plain HTML/CSS/JS with no compile step, so editing it is just editing the files directly.

The only file you should need to touch is webdash/config.js. It lists the panels shown on the live dashboard, and each panel is just a PromQL query plus a title and unit. To add, remove, or change a panel, edit the panels array there. Markup lives in index.html, styling in styles.css, and the rendering logic in app.js; none of the three has a metric name baked into it, so none of them need to change for a config-only edit. The Runs page (runs.html, runs.js) needs no configuring at all: its panels come from whatever metric names the selected runs actually recorded.

Nginx (webdash/nginx.conf) serves the static files and proxies /api/ to Prometheus's internal address, so the browser only ever talks to the dashboard's own origin regardless of which host port Prometheus is published on. Restart the dashboard service after changing config.js or nginx.conf:

docker compose -f compose.infra.yaml restart dashboard

Uninstall / teardown

docker compose -f compose.infra.yaml down

Add -v to also remove the Prometheus, Grafana, and Postgres data volumes.