Alertmanager

The alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

Note

The alertmanager is one of the tools that grew out of the prometheus project. Without prometheus you’ll need a very specific use-case to make use of the alertmanager.

Note

For this guide you should be familiar with the basic concepts of

License

alertmanager is licensed under the Apache License 2.0.

All relevant legal information can be found here

https://github.com/prometheus/alertmanager/blob/master/LICENSE

Prerequisites

We need to prepare a couple of directories.

The first directory is for storing the notification log and the alert silences:

[isabell@stardust ~]$ mkdir -p ~/var/lib/alertmanager
[isabell@stardust ~]$

The second directory is for storing the configuration files:

[isabell@stardust ~]$ mkdir ~/etc/alertmanager
[isabell@stardust ~]$

Installation

Find the latest version of alertmanager for the operating system linux and the architecture amd64 from the download page, download and extract it and enter the extracted directory:

[isabell@stardust ~]$ wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz
[isabell@stardust ~]$ tar xvzf alertmanager-0.20.0.linux-amd64.tar.gz
[isabell@stardust ~]$ cd alertmanager-0.20.0.linux-amd64
[isabell@stardust alertmanager-0.20.0.linux-amd64]$

Move the binary to ~/bin and the configuration file to ~/etc/alertmanager.

[isabell@stardust alertmanager-0.20.0.linux-amd64]$ mv alertmanager ~/bin/
[isabell@stardust alertmanager-0.20.0.linux-amd64]$ mv alertmanager.yml ~/etc/alertmanager
[isabell@stardust alertmanager-0.20.0.linux-amd64]$

Configuration

Setup daemon

Create the file ~/etc/services.d/alertmanager.ini with the following content:

[program:alertmanager]
command=alertmanager
  --web.listen-address="127.0.0.1:9093"
  --config.file=%(ENV_HOME)s/etc/alertmanager/alertmanager.yml
  --storage.path=%(ENV_HOME)s/var/lib/alertmanager/
autostart=yes
autorestart=yes

What the arguments for alertmanager mean:

--web.listen-address: The IP address and port alertmanager listens on.

--config.file: The full path to the alertmanager configuration file.

--storage.path: The path where alertmanager stores the notification log and the alert silences.

Finishing installation

Start alertmanager

After creating the configuration, tell supervisord to refresh its configuration and start the service:

[isabell@stardust ~]$ supervisorctl reread
SERVICE: available
[isabell@stardust ~]$ supervisorctl update
SERVICE: added process group
[isabell@stardust ~]$ supervisorctl status
SERVICE                            RUNNING   pid 26020, uptime 0:03:14
[isabell@stardust ~]$

Now point your alert creating tool or your prometheus service to the configured alertmanager port and you should receive alerts sent out via the alertmanager.

Best practices

Security

We did not configure a web backend because the alertmanager web interface should not be reachable from the public internet.

Everyone with access to the web interface is able to create and silence alarms.

Accessing the webinterface

One option to access the web interface is via a SSH tunnel:

[isabell@localhost ~]$ ssh -L 8080:localhost:9093 isabell@stardust.uberspace.de
[isabell@stardust ~]$

Now you can access the web interface via http://localhost:8080 on your workstation.

Tested with alertmanager 0.20.0, Uberspace 7.6.1.2

Written by: Malte Krupa <http://nafn.de>