Notifications

Prometheus evaluates alerts; Alertmanager groups and sends them by email or Slack.

1. Understand the flow

The agent exposes metrics; it does not directly send email or Slack messages. First install the monitoring stack. Advanced repository examples also provide Redis and DWho APIs.

2. Prepare destinations

In examples/monitoring, use the repository’s provided Alertmanager configuration and replace example recipients. Prepare the secret files referenced by compose.notifications.yaml, including an empty file for an unused destination. Put them in the example’s private directory with its required permissions; do not commit them.

SMTP passwords are read through auth_password_file and Slack URLs through api_url_file. Alertmanager does not automatically substitute environment variables in its YAML. Use your provider’s SMTP settings and put the actual Slack webhook URL only in its secret file.

On a fresh installation, prepare both channels:

mkdir -m 700 notifications.local
cp alertmanager.email-slack.yml notifications.local/alertmanager.yml
touch notifications.local/smtp_password notifications.local/slack_webhook_url
chmod 644 notifications.local/*

Choose alertmanager.email.yml or alertmanager.slack.yml instead for one channel. Keep the directory mode 0700; the mode 0644 files remain inaccessible to other host users through this private directory.

In notifications.local/alertmanager.yml, fill in to, from, smarthost and auth_username. SMTP must support STARTTLS. Using your editor, put the password on one line in notifications.local/smtp_password and the webhook on one line in notifications.local/slack_webhook_url.

3. Validate before delivery

GRAFANA_ADMIN_PASSWORD=validation-only docker compose \
  -f compose.yaml -f compose.notifications.yaml \
  run --rm --no-deps --entrypoint /bin/amtool alertmanager \
  check-config /etc/alertmanager/alertmanager.yml

4. Enable delivery

sh start.sh --notifications

This enables real delivery to configured destinations. Open Alertmanager on localhost:9093 and inspect logs and Prometheus alerts. A condition must persist for its rule’s for duration before firing.

The provided routing uses a 30-second initial wait, a five-minute group interval and a four-hour repeat interval. Resolutions are sent when send_resolved is enabled.

5. Stop the stack

docker compose -f compose.yaml -f compose.notifications.yaml down