Prometheus and Grafana

Store measurement history and explore dashboards.

1. Prepare the stack

From the v1.3.3 checkout:

cd examples/monitoring

In this tag, the monitoring example still references agent 0.0.57. Before starting, change its image in compose.yaml to the documented version:

image: decryptus/monit-docker:1.3.3

2. Start and sign in

sh start.sh

The script prepares a Grafana password in .env and waits for services. Read this file locally without sharing it. Open Grafana on localhost:3000 with user admin, then Prometheus on localhost:9090. These addresses refer to the host machine; use an SSH tunnel from a remote workstation.

This stack contains its own agent, Prometheus and Grafana. It does not include the lightweight interface from the previous guide and sends no notifications by default. Avoid competing stacks on the same ports.

3. Check measurements

In Prometheus, check the agent target and try these expressions:

monit_docker_ready
monit_docker_container_memory_usage_percent
monit_docker_container_cpu_usage_percent

An UP target means HTTP scraping works; also check monit_docker_ready. Container series are hidden when collection becomes invalid or stale. Agent counters reset on restart.

Prometheus and Grafana screenshots

These three captures show the example dashboard in Grafana 12.2.0, with synthetic data for six fictional containers. They illustrate the rendered dashboard, not production measurements or benchmarks. Click to enlarge; use arrow keys to browse and Escape to close.

4. Preserve data

The example retains fifteen days of metrics in a named volume. docker compose down stops the stack while preserving its volumes. --volumes would delete data; do not use it for a routine stop.

Continue with Alertmanager and notifications to turn Prometheus rules into messages.

The HTTP API and metrics contract covers responses, errors, freshness and exports, with curl examples.

By default, /metrics is open. Set serve --metrics-token-file FILE and configure the scraper with the matching bearer token to require authentication. Use --status-token-file for status reads.