.. _tutorial-troubleshooting: ================ Troubleshooting ================ This page lists some general troubleshooting strategies and methods for Munin. Quick Checklists ================ Graphs are blank or missing ---------------------------- 1. Is :ref:`munin-node` running on the monitored host? .. code-block:: bash sudo systemctl status munin-node 2. Can you reach the node from the master? .. code-block:: bash nc -z 4949 3. Is the node's IP in the ``allow`` list in ``/etc/munin/munin-node.conf``? 4. Is the host defined in ``/etc/munin/munin.conf`` on the master? 5. Has ``munin-cron`` run at least once since you added the host? Wait 5 minutes or run manually: .. code-block:: bash sudo -u munin /usr/share/munin/munin-cron 6. Check ``/var/log/munin/munin-update.log`` for errors. 7. Clear your browser cache — stale cached images can look blank. A plugin produces no graph --------------------------- 1. Is the plugin executable? .. code-block:: bash ls -la /etc/munin/plugins/ 2. Has ``munin-node`` been restarted since the plugin was added? 3. Does the plugin work with ``munin-run``? .. code-block:: bash sudo munin-run sudo munin-run config 4. Check ``/var/log/munin/munin-update.log`` for errors about this plugin. 5. Is the plugin's ``graph_category`` valid? Invalid categories silently drop graphs. munin-update is slow --------------------- 1. Check ``max_processes`` in ``/etc/munin/munin.conf``. Default is 16. Lower it if the master is resource-constrained. 2. Check ``timeout`` in ``/etc/munin/munin.conf``. Default is 180 seconds. Unreachable nodes will block until the timeout expires. 3. Use ``update_priority`` to run slow nodes first. 4. Consider using :ref:`munin-async ` for nodes on slow or unreliable links. RRD files filled with zeros or NaN ------------------------------------ 1. The plugin may declare the wrong data type. ``GAUGE`` is the default; ``COUNTER`` and ``DERIVE`` compute rates. If you declared ``COUNTER`` but the value is a gauge, you will see zeros. 2. The plugin may output non-numeric characters. Check with: .. code-block:: bash sudo munin-run 3. For new plugins, wait 20 minutes. RRD needs several data points before it can display meaningful graphs. Node won't accept connections ------------------------------ 1. Check the ``allow`` / ``cidr_allow`` directives in ``/etc/munin/munin-node.conf``. The master's IP must match. 2. Check firewall rules on the node: port 4949 must be open. 3. Check if another process is already listening on port 4949. Detailed Diagnostics ==================== Check node agent ----------------- Is the :ref:`munin-node` process (daemon) running on the host you want to monitor? Did you restart the :ref:`munin-node` process after you made changes to its configuration? Check connectivity ================== The examples show a :ref:`munin-node` agent running on 127.0.0.1; replace it with your node address. .. note:: You can use `netcat `_ to port 4949. Using ``telnet`` was the previous recommended way as it was a fairly standard install. We don't recommend it anymore since ``netcat`` is now almost as ubiquitous as ``telnet`` and it offers a real native TCP connection, whereas ``telnet`` `does not `_. Note that using `socat` also works perfectly, but it is not as mainstream. Does the :ref:`munin-node` agent allow connections from your munin master? Here we try to connect manually to the :ref:`munin-node` that runs on the Munin master host. It can be reached via IP address ``127.0.0.1`` or hostname ``localhost`` and port ``4949``. Output of a ``netcat`` session should be something like this: :: # nc localhost 4949 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. # munin node at [your hostname] Does the above output give the same hostname that should be expected upon configuration in :ref:`munin.conf`? .. note:: If you have a fully qualified domain name (FQDN) in :ref:`munin-node.conf`, the host you're monitoring has to identify itself with FQDN as well. E.g. if the masters node tree has the following entry: :: [foo.example.com] address foo.example.com ...then a netcat session to the node should give you the following output: :: # munin node at foo.example.com .. note:: If the connection test fails, check the :ref:`allow directive ` in :ref:`munin-node.conf` and make sure any firewalls allow contact on destination port 4949. Check the Logs ============== Munin's log files (typically below ``/var/log/munin/``) are a good source of information while debugging problems. Log files of a :ref:`munin-node`: * ``munin-node.log`` and ``munin-node-configure.log``: configuration issues and connection messages Log files of a :ref:`munin master `: * ``munin-cgi-graph.log`` and ``munin-graph.log``: issues with generating graphs * ``munin-cgi-html.log`` and ``munin-html.log``: issues with generating html content * ``munin-update.log``: fetch configuration and values from a remote :ref:`munin-node` * ``munin-limits.log``: generated alarms due to specified :ref:`warning `/:ref:`critical ` thresholds .. _debugging-plugins: Debugging Plugins ======================= Which plugins are enabled on the node? -------------------------------------- Does :ref:`munin-node` recognize any plugins? Try issuing the command ``list`` (being connected to the agent) and a (long) list of plugins should show. :: # nc localhost 4949 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. # munin node at foo.example.com list open_inodes irqstats if_eth0 df uptime [...] .. note:: Some plugins require specific capabilities (most notably: :ref:`multigraph `). These plugins do not show up in the list, unless the client announces this capability. For example type ``cap multigraph`` before ``list`` in order to also find multigraph plugins in the list. Check a particular plugin ------------------------- **Check on agent host** .. note:: All the commands here need to be run as user ``root``. A common method of becoming ``root`` is via the ``sudo`` command, but refer to your local documentation for a more specific instruction. Restart :ref:`munin-node`, as it only reads the plugin list upon start. (Good to test a plugin with :ref:`munin-run`, without enabling it right away.) :: /etc/init.d/munin-node restart Call :ref:`munin-run` on the monitored host to see whether the plugin runs through. Try with and without the ``config`` plugin argument. Both runs should not emit any error message. .. note:: You can also use the ``--debug`` flag, as it shows if the configuration file is correctly parsed, mostly for UID & environment variables. Regular run: :: # munin-run df _dev_hda1.value 83 Config run: :: # munin-run df config graph_title Filesystem usage (in %) graph_args --upper-limit 100 -l 0 graph_vlabel % graph_category disk graph_info This graph shows disk usage on the machine. _dev_hda1.label / _dev_hda1.info / (ext3) -> /dev/hda1 _dev_hda1.warning 92 _dev_hda1.critical 98 **Check from Munin master** Does the plugin run through :ref:`munin-node`, with and without config? Regular run: :: # nc foo.example.com 4949 Trying foo.example.com... Connected to foo.example.com. Escape character is '^]'. # munin node at foo.example.com fetch df _dev_hda1.value 83 [...] . With config: :: # nc foo.example.com 4949 Trying foo.example.com... Connected to foo.example.com. Escape character is '^]'. # munin node at foo.example.com config df graph_title Filesystem usage (in %) graph_args --upper-limit 100 -l 0 graph_vlabel % graph_category disk graph_info This graph shows disk usage on the machine. _dev_hda1.label /boot _dev_hda1.info /boot (ext3) -> /dev/hda1 _dev_hda1.warning 92 _dev_hda1.critical 98 [...] . If the plugin works for ``munin-run`` but not through ``netcat``, you might have a ``$PATH`` problem. .. note:: Set {{{env.PATH}}} for the plugin in the plugin's environment file. Check Munin Master ================== Do the directories specified by ``dbdir``, ``htmldir``, ``logdir`` and ``rundir`` defined in :ref:`munin.conf` have the correct permissions? (If you first run munin as root, maybe they're not readable/writeable by the user that runs the cron job) Is :ref:`munin-cron` established as a cron controlled process, run as the Munin user? Does the output when running :ref:`munin-update` as the Munin user on the server node show any errors? Try running "``munin-cron --debug > /tmp/munin-cron.debug``" and check the output file ``/tmp/munin-cron.debug``. Check data collection --------------------- This step will tell you whether :ref:`munin-update` (the master) is able to communicate with :ref:`munin-node` (the agent). Run :ref:`munin-update` as user ``munin`` on the Munin master machine. :: # su -s /bin/bash munin $ /usr/share/munin/munin-update --debug --nofork --host foo.example.com --service df You should get a line like this: :: Aug 11 22:39:51 - [6846] Updating /var/lib/munin/example.com/foo.example.com-df-_dev_hda1-g.rrd with 57 After this, replace ``df`` with the service you want to check, such as ``hddtemp_smartctl``. If one of these steps does not work, something is probably wrong with the plugin or how :ref:`munin-node` talks to the plugin. #. Does the plugin run when executed directly? If it runs when executed as root and not through :ref:`munin-run` (as described above), the plugin has a permission problem. See this `article on environment files `_. #. Does the plugin output contain too few, too many and/or illegal characters? #. Does Munin (:ref:`munin-cron` and its children) write values into RRD files? Hint: ``rrdtool fetch [rrd file] AVERAGE`` #. Does the plugin use legal field names? See :ref:`Notes on Field names `. #. In case you `loan data `_ from other graphs, check that the `fieldname.type `_ is set properly. See `Munin file names `_ for a quick reference on what any error messages in the logs might indicate. Frequent Incidents ================== SELinux blocks Munin plugins ---------------------------- * See `the documentation start page `_ for links to SELinux rules for Munin. RRD files are filled with 0 --------------------------------------------------------------------- although munin-node seems to show sane values. * The plugin's output shows GAUGE values, but were declared as COUNTER or DERIVE in the plugin's config. .. note:: GAUGE is the default data type in Munin! Any other data type for a field must be explicitly declared. RRD files are filled with ``NaN`` --------------------------------------------------------------------------- although munin-node seems to show sane values. * Check that there are no invalid characters in the plugin's output. * For new plugins let munin gather data for about 20 minutes and things will unwrinkle munin-node won't give any data ---------------------------------------------------------- although it is configured properly. * Check that there is a ``.value`` directive for every of the plugin's field names (yes, I managed to forget that recently). munin-node only temporary returns valid data -------------------------------------------- * Check that no race conditions occur. A typical race condition is updating a file with crontab while the plugin is trying to read the file. The graphs are empty -------------------- * The plugin's output shows GAUGE values, but were declared as COUNTER or DERIVE in the plugin's config. (GAUGE is default data type in Munin) * The files to be updated by Munin are owned by root or another user account * The local user browser cache may be corrupt, especially if "most" graphs are displayed correctly and "some" graphs are blank. In Firefox (or your browser of choice) go to tools and clear recent history, then check to see if the graphs are now properly displayed. A plugin's graph is missing --------------------------- Check the following conditions if there is no graph produced for plugin: * the plugin file (or a symlink to it) is placed in the plugin directory (typically: ``/etc/munin/plugins``) * the executable permission of the plugin file is set * :ref:`munin-node` was restarted after the plugin was added * user/group is configured for the plugin (if necessary) * the plugin works as expected locally via :ref:`munin-run` * the :ref:`munin master ` supports all capabilities required by the plugin (e.g. type ``cap multigraph`` before ``list`` in an interactive ``nc``/``telnet`` session) * no related error messages for this plugin appear in ``/var/log/munin/munin-update.log`` (on the :ref:`munin master `) * an rrd file is created on the :ref:`munin master ` (e.g. below ``/var/lib/munin``) Other mumbo-jumbo ----------------- * Run the different stages in :ref:`munin-cron` manually, using ``--debug``, ``--nofork``, something like this: :: # su - munin -c "/usr/lib/munin/munin-update \ --debug --nofork \ --host foo.example.com \ --service df" See also ======== * `No Graph FAQ `_ * :ref:`Upgrade notes `