Let Munin croak alarm

As of Munin 1.2 there is a generic interface for sending warnings and errors from Munin. If a Munin plugin discovers that a plugin has a data source breaching its defined limits, Munin is able to alert the administrator either through simple command line invocations or through a monitoring system like Nagios or Icinga.

Note that if the receiving system can cope with only a limited number of messages at the time, the configuration directive contact.contact.max_messages may be useful.

When sending alerts, you might find good use in the Munin alert variables.

Note

Alerts not working? For some versions 1.4 and less, note that having more than one contact defined can cause munin-limits to hang.

Sending alerts through Nagios

How to set up Nagios and Munin to communicate has been thoroughly described in Munin and Nagios.

Alerts send by local system tools

Email Alert

To send email alerts directly from Munin use a command such as this:

contact.email.command mail -s "Munin-notification for ${var:group} :: ${var:host}" your@email.address.here

For an example with explanation please look at Munin alert email notification

Syslog Alert

To send syslog message with priority use a command such as this:

contact.syslog.command logger -p user.crit -t "Munin-Alert"

Alerts to or through external scripts

To run a script (in this example, ‘script’) from Munin use a command such as this in your munin.conf.

Make sure that:

  1. There is NO space between the ‘>’ and the first ‘script’
  2. ‘script’ is listed twice and
  3. The munin user can find the script – by either using an absolute path or putting the script somewhere on the PATH – and has permission to execute the script.
contact.person.command >script script

This syntax also will work (this time, it doesn’t matter if there is a space between ‘|’ and the first ‘script’ ... otherwise, all the above recommendations apply):

contact.person.command | script script

Either of the above will pipe all of Munin’s warning/critical output to the specified script. Below is an example script to handle this input and write it to a file:

#!/usr/bin/env ruby

File.open('/tmp/munin_alerts.log', 'a') do |f| #append
   f.puts Time.now
   for line in $stdin
      f.puts line
   end
end

The alerts getting piped into your script will look something like this:

localhost :: localdomain :: Inode table usage
       CRITICALs: open inodes is 32046.00 (outside range [:6]).

Syntax of warning and critical

The plugin.warning and plugin.critical values supplied by a plugin can be overwritten by the Munin master configuration in munin.conf.

Note that the warning/critical exception is raised only if the value is outside the defined value. E.g. foo.warning 100:200 will raise a warning only if the value is outside the range of 100 to 200.

Reformatting the output message

You can redefine the format of the output message by setting Global Directive contact.<something>.text in munin.conf using Munin alert variables.

Something like:

contact.pipevia.command | /path/to/script /path/to/script \
   --cmdlineargs="${var:group} ${var:host} ${var:graph_category} '${var:graph_title}'"

contact.pipevia.always_send warning critical

contact.pipevia.text  <munin group="${var:group}" host="${var:host}"\
  graph_category="${var:graph_category}" graph_title="${var:graph_title}" >\
  ${loop< >:wfields <warning label="${var:label}" value="${var:value}"\
    w="${var:wrange}" c="${var:crange}" extra="${var:extinfo}" /> }\
  ${loop< >:cfields <critical label="${var:label}" value="${var:value}"\
    w="${var:wrange}" c="${var:crange}" extra="${var:extinfo}" /> }\
  ${loop< >:ufields <unknown label="${var:label}" value="${var:value}"\
    w="${var:wrange}" c="${var:crange}" extra="${var:extinfo}" /> }\
  </munin>

Calls the script with the command line arguments (as a python list):

['/path/to/script','/path/to/script','--cmdlineargs="example.com', 'test.example.com', 'disk', 'Disk usage in percent', '']

and the input sent to the script is (whitespace added to break long line):

'<munin group="example.com" host="test.example.com" graph_category="disk" graph_title="Disk usage in percent" >
  <critical label="/home" value="98.41" w=":92" c=":98" extra="" />
</munin> '

(need for the second /path/to/script may vary, but this document says it is required)

If something goes wrong:

  • check the log file for munin-limits.log.
  • remember this script will run as the same user as the cron job that starts munin-cron.

For more examples see section Example usage below.

Munin Alert Variables

When using Munin’s built-in alert mechanisms, lots of variables are available. Generally, all directives recognized in the configuration protocol and in munin.conf.conf are available as ${var:directive}. We list some frequently used in the following section.

How variables are expanded

The ${var:value} variables get the correct values from munin-limits prior to expansion of the variable.

Then, the ${var:*range} variables are set from {fieldname}.warning and {fieldname}.critical.

Based on those, {fieldname}.label occurences where warning or critical levels are breached or unknown are summarized into the ${var:*fields} variables.

Example usage

Note that the sample command lines are wrapped for readability.

Example 1, iterating through warnings and criticals

contact.mail.command mail -s "[${var:group};${var:host}] -> ${var:graph_title} ->
                             warnings: ${loop<,>:wfields  ${var:label}=${var:value}} /
                             criticals: ${loop<,>:cfields  ${var:label}=${var:value}}" me@example.com

This stanza results in an e-mail with a subject like this:

[example.com;foo] -> HDD temperature -> warnings: sde=29.00,sda=26.00,sdc=25.00,sdd=26.00,sdb=26.05 / criticals:

Note that there are no breaches of critical level temperatures, only of warning level temperatures.

Example 2, reading ${var:wfields}, ${var:cfields} and ${var:ufields} directly

contact.mail.command mail -s "[${var:group};${var:host}] -> ${var:graph_title} ->
                             warnings: ${var:wfields} /
                             criticals: ${var:cfields} /
                             unknowns: ${var:ufields}" me@example.com

The result of this is the following:

[example.com;foo] -> HDD temperature -> warnings: sde sda sdc sdd sdb / criticals: / unknowns:

Iteration using Text::Balanced

The Text::Balanced iteration syntax used in munin-limits is as follows (extra spaces added for readability):

${ loop < join character > : list of words ${var:label} = ${var:value} }

Given a space separated list of words “a b c”, and the join character ”,” (comma), the output from the above will equal

a.label = a.value,b.label = b.value,c.label = c.value

in which the label and value variables will be substituted by their Munin values.

Please consult the Text::Balanced documentation for more details.