Integrate Alertmanager with Email, MS Teams, Slack

Monitoring and alerting are a very integral part of infrastructure to make sure that our server and application are always up and running. One fine day my manager told me that we need to integrate our alertmanager with different platforms, like Email, MS Teams, and Slack. For that, I need to read many articles for the solution and it was very hectic and frustrating. So I thought people may require the same thing for their environment.

Alertmanager: Alertmanager is an open-source tool used to trigger alerts when resource usage is over the specified threshold. It handles the alert send by the client applications. It sends the alerts over Email, MS-Teams, slack, and other platforms.

Alertmanager Architecture

How Alertmanager works ?

The architecture of alertmanager is very simple and straightforward. As we know Prometheus monitors our servers and applications. During the monitoring process, if a resource crosses the specified threshold then Prometheus triggers the alert to alertmanager and alertmanager sends the alert to the specified platform like Email, MS Teams, Slack as we can see in the above architectural diagram. Now I will explain, how we can configure alertmanager and integrate alertmanager with different applications.

Alertmanager Installation

1. Download Alertmanager binaries

mkdir alertmanager
cd alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.22.2/alertmanager-0.22.2.linux-amd64.tar.gz

2. Untar alertmanager tar file.

tar -xvf alertmanager-0.22.2.linux-amd64.tar.gz

3. Copy alertmanager binary to /usr/bin.

cd alertmanager-0.22.2.linux-amd64
sudo cp alertmanager /usr/bin

4. Now we need to create the required configuration file for alertmanger.

sudo mkdir /etc/alertmanager
sudo vim /etc/alertmanager/alertmanager.yml

#Config for Email
global:

# The root route on which each incoming alert enters.
route:
  # default route if none match
  receiver: alert-emailer
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1m
  group_by: ['alertname', 'priority']

receivers:
- name: alert-emailer
  email_configs:
  - to: 'receiver@example.com'
    send_resolved: true
    from: 'sender@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'sender@example.com'
    auth_password: 'IamPassword'
    require_tls: yes

#Slack Config
global:
resolve_timeout: 1m
slack_api_url: 'https://hooks.slack.com/services/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'

route:
receiver: 'slack-notifications'

receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#monitoring'
  send_resolved: true
  

5. One more directory we need to create, because alertmanager stores its data in this file.

sudo mkdir /var/lib/alertmanager

6. The final part is to make service file for alertmanager.

sudo vim /etc/systemd/system/alertmanager.service

[Unit]
Description=AlertManager Server Service
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager \
    --cluster.listen-address=localhost:9094
[Install]
WantedBy=multi-user.target    
    

Finally we are all set to run alertmanager service, just need to reload the daemon because we have added a new service file and start the prometheus service.

sudo systemctl daemon-reload
sudo systemctl start alertmanager.service
sudo systemctl status alertmanager.service

You can get the web UI of alertmanager http://<serverIP>:9093 .

Now we are all set to trigger the alert. So just set back and relax, alertmanager will notify you regarding alert on the configured platform. Thanks !!