Netdata alarm notifications¶
exec line in health configuration defines an external script that will be called once
the alarm is triggered. The default script is alarm-notify.sh.
You can change the default script globally by editing
alarm-notify.sh is capable of sending notifications:
- to multiple recipients
- using multiple notification methods
- filtering severity per recipient
It uses roles. For example
Each alarm is assigned to one or more roles, using the
to line of the alarm configuration.
alarm-notify.sh uses its own configuration file
the default is here
(to edit it on your system run
to find the destination address of the notification for each method.
Each role may have one or more destinations.
So, for example the
sysadmin role may send:
- emails to email@example.com and firstname.lastname@example.org
- pushover.net notifications to USERTOKENS
- pushbullet.com push notifications to email@example.com and firstname.lastname@example.org
- messages to slack.com channel
- messages to Discord channels
settings per notification method:
all notification methods except email, require some configuration (i.e. API keys, tokens, destination rooms, channels, etc).
- recipients per role per notification method
You can run the following command by hand, to test alarms configuration:
# become user netdata su -s /bin/bash netdata # enable debugging info on the console export NETDATA_ALARM_NOTIFY_DEBUG=1 # send test alarms to sysadmin /usr/libexec/netdata/plugins.d/alarm-notify.sh test # send test alarms to any role /usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under
/usr/lib/netdata). You can always find the location of the alarm-notify.sh script in
If you need to dig even deeper, you can trace the execution with
bash -x. Note that in test mode, alarm-notify.sh calls itself with many more arguments. So first do
bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test