.. _tutorial-alert: ===================== Let Munin croak alarm ===================== Munin offers a generic interface for sending warnings and errors. If a Munin plugin discovers that a plugin has a data source breaching its defined limits, Munin is able to alert the administrator either through simple command line invocations or through a monitoring system like Nagios or Icinga. Note that if the receiving system can cope with only a limited number of messages at the time, the configuration directive :ref:`contact.contact.max_messages ` may be useful. When sending alerts, you might find good use in the :ref:`Munin alert variables `. Sending alerts through Nagios ============================= How to set up Nagios and Munin to communicate has been thoroughly described in :ref:`Munin and Nagios `. Alerts send by local system tools ================================= Email Alert ----------- To send email alerts directly from Munin use a command such as this: :: contact.email.command mail -s "Munin ${var:worst}: ${var:group}::${var:host}::${var:plugin}" your@email.address.here Syslog Alert ------------ To send syslog message with priority use a command such as this: :: contact.syslog.command logger -p user.crit -t "Munin-Alert" Alerts to or through external scripts ------------------------------------- To run a script (in this example, 'script') from Munin use a command such as this in your munin.conf. :: contact.person.command script args... Make sure that: #. The munin user can find the script -- by either using an absolute path or putting the script somewhere on the PATH -- and has permission to execute the script. #. The command is properly quoted and escaped, if necessary. There is some nuance here; specifically, if your command contains any of the characters ``&;`'\"|*?~<>^()[]{}$``, the whole thing will be passed as a single argument to ``sh -c`` for execution, and you need to take into account how the shell will interpret any quotes or special characters. If the command does *not* contain any of those characters, Munin itself (specifically, the perl ``exec()`` function) will split it into words and execute it without passing it through the shell. This will pipe all of Munin's warning/critical output to the specified script. Below is an example script to handle this input and write it to a file: :: #!/usr/bin/env ruby File.open('/tmp/munin_alerts.log', 'a') do |f| #append f.puts Time.now for line in $stdin f.puts line end end The alerts getting piped into your script will look something like this (but see "Reformatting the output message" below if you want to customize this): :: localhost :: localdomain :: Inode table usage CRITICALs: open inodes is 32046.00 (outside range [:6]). ``munin-limits`` will pipe messages to your script until it has sent ``max-messages`` messages, then close the pipe and spawn a new copy of the script for the next batch of messages. Syntax of warning and critical ============================== The ``plugin.warning`` and ``plugin.critical`` values supplied by a plugin can be overwritten by the Munin master configuration in :ref:`munin.conf `. Note that the warning/critical exception is raised only if the value is outside the defined value. E.g. ``foo.warning 100:200`` will raise a warning only if the value is outside the range of 100 to 200. A single number is interpreted as the maximum allowed value (without a lower limit). Example limits configuration emitted by a plugin: :: foo.label some data foo.warning 0:100 foo.critical 150 Reformatting the output message =============================== You can redefine the format of the output message by setting *Global Directive* :ref:`contact.\.text ` in :ref:`munin.conf ` using :ref:`Munin alert variables `. Something like: :: contact.pipevia.command /path/to/script \ --cmdlineargs="${var:group} ${var:host} ${var:graph_category} '${var:graph_title}'" contact.pipevia.always_send warning critical contact.pipevia.text \ ${loop< >:wfields }\ ${loop< >:cfields }\ ${loop< >:ufields }\ Calls the script with the command line argument: :: --cmdlineargs="example.com test.example.com disk 'Disk usage in percent'" and the input sent to the script is (whitespace added to break long line): :: ' ' (need for the second ``/path/to/script`` may vary, but this document says it is required) If you need to insert tabs or newlines into your messages, you can use ``\t`` or ``\n``. This, for example, will output the label and value of each critical field, separated by a tab, one per line: :: ${loop:cfields ${var:label}\t${var:value}\n} Note that this is a special feature of alert messages, not something that will work elsewhere in Munin config files. If something goes wrong: - check the log file for ``munin-limits.log``. - remember this script will run as the same user as the cron job that starts :ref:`munin-cron `. For more examples see section :ref:`Example usage ` below. .. _alert_variables: Munin Alert Variables ===================== When using Munin's built-in alert mechanisms, lots of variables are available. Generally, all directives recognized in the :ref:`configuration protocol ` and in :ref:`munin.conf `.conf are available as ``${var:directive}``. We list some frequently used in the following section. .. _alert_variable_global: Group or host or plugin related variables ----------------------------------------- These are directly available. ============ :Variable: **group** :Syntax: ``${var:group}`` :Reference: Group name as declared in munin.conf. ============ :Variable: **host** :Syntax: ``${var:host}`` :Reference: Host name as declared in munin.conf. ============ :Variable: **graph_title** :Syntax: ``${var:graph_title}`` :Reference: Plugin's title as declared via config protocol or set in munin.conf. ============ :Variable: **plugin** :Syntax: ``${var:plugin}`` :Reference: Plugin's name as declared via config protocol or set in munin.conf. ============ :Variable: **graph_category** :Syntax: ``${var:graph_category}`` :Reference: Plugin's category as declared via config protocol or set in munin.conf. ============ :Variable: **worst** :Syntax: ``${var:worst}`` :Reference: The name of the worst status detected in this run of munin-limits. From best to worst, the statuses are OK, UNKNOWN, WARNING, and CRITICAL. ============ :Variable: **worstid** :Syntax: ``${var:worstid}`` :Reference: A numeric equivalent of **worst**; 0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 for UNKNOWN. Note that this is not the same as the order of severity. .. _alert_variable_data: Data source related variables ----------------------------- The below table lists some variables related to the data fields in a plugin. To extract these, they must be iterated over, even if there is only one field. Iteration follows the syntax defined in the Perl module `Text::Balanced `_ (sample below the table). ============ :Variable: **{fieldname}.label** :Syntax: ``${var:label}`` :Reference: Label of the data field as declared via plugin's config protocol or set in munin.conf. ============ :Variable: **{fieldname}.value** :Syntax: ``${var:value}`` :Reference: Value of the data field as delivered by data fetch ============ :Variable: **{fieldname}.extinfo** :Syntax: ``${var:extinfo}`` :Reference: Extended info of the field, if declared via plugin's config protocol or set in munin.conf. ============ :Variable: **{fieldname}.warning** :Syntax: ``${var:wrange}`` :Reference: Numeric range for warning alerts of the field, if declared via plugin's config protocol or set in munin.conf. ============ :Variable: **{fieldname}.critical** :Syntax: ``${var:crange}`` :Reference: Numeric range for critical alerts of the field, if declared via plugin's config protocol or set in munin.conf. ============ :Variable: **ofields** :Syntax: ``${var:ofields}`` :Reference: Space separated list of fieldnames with a value inside the warning range (i.e. "ok" values). ============ :Variable: **fofields** :Syntax: ``${var:fofields}`` :Reference: Space separated list of fieldnames with a value inside the warning range, that were *not* ok the last time munin-limits ran (i.e. "freshly ok" values -- on the next run after this they are merely "ok"). ============ :Variable: **wfields** :Syntax: ``${var:wfields}`` :Reference: Space separated list of fieldnames with a value outside the warning range as detected by munin-limits. ============ :Variable: **cfields** :Syntax: ``${var:cfields}`` :Reference: Space separated list of fieldnames with a value outside the critical range as detected by munin-limits. ============ :Variable: **ufields** :Syntax: ``${var:ufields}`` :Reference: Space separated list of fieldnames with an unknown value as detected by munin-limit. ============ :Variable: **numufields**, **numcfields**, **numwfields**, **numfofields**, **numofields** :Syntax: ``${var:numufields}``, etc :Reference: The number of fields that are unknown, critical, warning, freshly OK, and OK, respectively. How variables are expanded -------------------------- The ``${var:value}`` variables get the correct values from munin-limits prior to expansion of the variable. Then, the ``${var:*range}`` variables are set from {fieldname}.warning and {fieldname}.critical. Based on those, ``{fieldname}.label`` occurrences where warning or critical levels are breached or unknown are summarized into the ``${var:*fields}`` variables. .. _alert_variables_example_usage: Example usage ------------- Note that the sample command lines are wrapped for readability. **Example 1, iterating through warnings and criticals** :: contact.mail.command mail -s "[${var:group};${var:host}] -> ${var:graph_title} -> warnings: ${loop<,>:wfields ${var:label}=${var:value}} / criticals: ${loop<,>:cfields ${var:label}=${var:value}}" me@example.com This stanza results in an e-mail with a subject like this: :: [example.com;foo] -> HDD temperature -> warnings: sde=29.00,sda=26.00,sdc=25.00,sdd=26.00,sdb=26.05 / criticals: Note that there are no breaches of critical level temperatures, only of warning level temperatures. **Example 2, reading ${var:wfields}, ${var:cfields} and ${var:ufields} directly** :: contact.mail.command mail -s "[${var:group};${var:host}] -> ${var:graph_title} -> warnings: ${var:wfields} / criticals: ${var:cfields} / unknowns: ${var:ufields}" me@example.com The result of this is the following: :: [example.com;foo] -> HDD temperature -> warnings: sde sda sdc sdd sdb / criticals: / unknowns: Iteration using Text::Balanced ------------------------------ The Text::Balanced iteration syntax used in munin-limits is as follows (extra spaces added for readability): :: ${ loop < join character > : list of words ${var:label} = ${var:value} } Given a space separated list of words "a b c", and the join character "," (comma), the output from the above will equal :: a.label = a.value,b.label = b.value,c.label = c.value in which the label and value variables will be substituted by their Munin values. Please consult the `Text::Balanced `_ documentation for more details.