Checkmk
to checkmk.com

1. Why command line?

When a Checkmk-System has been installed, it can be 100 % configured and operated using the web interface. There are nonetheless situations in which it is useful to dive into the depths of the command line, for example:

  • when searching for the source of problems

  • when automating the administration of Checkmk

  • when programming and testing your own extensions

  • to be able to understand how Checkmk functions internally

  • if you simply enjoy working with the command line!

This article will present the most important commands, files and directories on Checkmk’s command line.

2. The site user

2.1. Login as site user

When administering Checkmk, with a few exceptions you need never work as the root-user. In this article we will generally assume that you are logged in as a site user. That is done with, e.g.:

root@linux# su - mysite

It is also possible to make a direct SSH-login to a site without a detour via root. Since the site user is a ‘completely normal’ Linux user, you must simply assign a password for this (which requires root-permissions, once only, for the configuration):

root@linux# passwd mysite
Enter new UNIX password: **********
Retype new UNIX password: **********
passwd: password updated successfully

Afterwards an SSH-login directly from another computer should be possible (Windows-users preferably use PuTTY for this). From Linux this login is simply performed in the command line using the ssh command:

user@otherhost> ssh mysite@myserver123
mysite@localhost's password: **********

At the first login a ‘warning’ regarding an unknown host key will probably be received. When you are certain that in this brief moment no attacker has taken over your operating system’s IP-address, you can simply verify it with yes.

You can also work with the command line on the Checkmk-appliance. How that is done is explained in its own article.

2.2. Profile and environment variables

So that as few problems as possible arise, particularly as a result of individual distributions or differing operating system configurations, the Checkmk-system ensures that the site user – and likewise all of the monitoring’s processes – always have a clearly defined environment. Along with the home directory and the permissions, the environment variables play an important role.

Among other things, when logging in as a site user the following variables will be set or modified. These variables are available for use in all processes running within the site. This also applies to scripts that are indirectly invoked by these processes (for example, a user’s own notification scripts).

OMD_SITE

The site’s name (mysite). In custom scripts this variable should always be used rather than a hard coded site name (e.g. with $OMD_SITE in the shell). With this the script can also be used unchanged in other sites.

OMD_ROOT

The path for the site directory (/omd/sites/mysite)

PATH

Directories in which executable programs will be searched for. For example, Checkmk keeps the site’s bin/ here. In the case of identical names, Checkmk programs have priority – this is important, e.g., for the mail command, a special version of which is provided with a Checkmk installation.

LD_LIBRARY_PATH

Directories in which additional binary libraries are searched for. Using this variable Checkmk ensures that libraries provided with Checkmk have priority over those installed in the normal operating system.

PERL5LIB

Search path for the Perl module. Here, too, module variants delivered by Checkmk have priority in case of doubt.

LANG

The language setting for command line commands. This setting is adopted from the Linux installation. This variable is automatically deleted in the site’s processes, and the setting reverts to the default English! This also affects other regional settings. Removing LANG is very important, since a number of standard Nagios plug-ins, for example, the German language setting, uses a comma for the decimal separator instead of a point. Your output can thus not be accurately processed.

With the env command you can output all of the environment variables – adding | sort to this command arranges the list a bit more clearly:

OMD[mysite]:~$ env | sort
HOME=/omd/sites/mysite
LANG=de_DE.UTF-8
LD_LIBRARY_PATH=/omd/sites/mysite/local/lib:/omd/sites/mysite/lib
LOGNAME=mysite
MAILRC=/omd/sites/mysite/etc/mail.rc
MAIL=/var/mail/mysite
MANPATH=/omd/sites/mysite/share/man:
MODULEBUILDRC=/omd/sites/mysite/.modulebuildrc
MP_STATE_DIRECTORY=/omd/sites/mysite/var/monitoring-plugins
NAGIOS_PLUGIN_STATE_DIRECTORY=/omd/sites/mysite/var/monitoring-plugins
OMD_ROOT=/omd/sites/mysite
OMD_SITE=mysite
PATH=/omd/sites/mysite/lib/perl5/bin:/omd/sites/mysite/local/bin:/omd/sites/mysite/bin:/omd/sites/mysite/local/lib/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
PERL5LIB=/omd/sites/mysite/local/lib/perl5/lib/perl5:/omd/sites/mysite/lib/perl5/lib/perl5:
PERL_MM_OPT=INSTALL_BASE=/omd/sites/mysite/local/lib/perl5/
PWD=/omd/sites/mysite
REQUESTS_CA_BUNDLE=/omd/sites/mysite/var/ssl/ca-certificates.crt
SHELL=/bin/bash
SHLVL=1
TERM=xterm
USER=mysite
_=/usr/bin/env

Under Linux the environment is an attribute of a process. Every process has its own variables, which it automatically passes on to sub-processes. These start initially with the same, inherited variables, but can also alter them.

With the env command you can always only view the current shell’s environment. If you suspect there is an error in a particular process’s environment, with a small trick you can nonetheless output a listing of its environment. For this you only need the process-ID (PID). You can identify this with, e.g., ps ax, pstree -p or top. With this you can then access the process’s environ file directly via the /proc file system. Here as an example is a suitable command for the PID 13222:

OMD[mysite]:~$ tr \\0 \\n < /proc/13222/environ | sort

If you require custom variables for your own scripts or other software to be run in the site, store them in the etc/environment file which has been specially created for this purpose. All variables defined here will be available everywhere within the site:

etc/environment
# Custom environment variables
#
# Here you can set environment variables. These will
# be set in interactive mode when logging in as site
# user and also when starting the OMD processes with
# omd start.
#
# This file has shell syntax, but without 'export'.
# Better use quotes if your values contain spaces.
#
# Example:
#
# FOO="bar"
# FOO2="With some spaces"
#
MY_SUPER_VAR=blabla123
MY_OTHER_VAR=10.88.112.17

2.3. Customizing the shell

If you wish to customize your shell (Prompt or other things), you can perform this as usual in the .bashrc file. Environment variables nonetheless belong to etc/environment, so that they are certain to be available to all processes.

There is also nothing to prevent you having your own .vimrc file if you like working with VIM.

3. The directory structure

3.1. The separation of software and data

The following graphic shows the most important directories in a Checkmk installation with a site named mysite and a <version> called for example 2.0.0p8.cee:

Illustration of the directory structure of a Checkmk site.

The basis for this structure is provided by the /omd directory. Without exception, all of the files for Checkmk are found here. /omd is in fact a symbolic link to /opt/omd, while the actual physical data is located in /opt – but all data paths in Checkmk always use /omd.

Important is the separation of data (highlighted yellow) and software (blue). The site’s data is found in /omd/sites, and the installed software in /omd/versions.

3.2. Site directory

Like every Linux user, the site user also has a home directory, which we refer to as the site directory. If your site is named mysite it will be found in /omd/sites/mysite. As usual in Linux the shell abbreviates the its own home directory with a tilde (~) (or swung dash). Since immediately following a login you will actually be in this directory, the tilde appears automatically in the input prompt:

OMD[mysite]:~$

Subdirectories of the site directory are shown relative to the tilde:

OMD[mysite]:~$ cd var/log
OMD[mysite]:~/var/log$

A number of subdirectories are located within the site directory, these can be listed with ll:

OMD[mysite]:~$ ll
total 16
lrwxrwxrwx  1 mysite mysite   11 Jan 24 11:56 bin -> version/bin/
drwxr-xr-x 19 mysite mysite 4096 Jan 24 11:56 etc/
lrwxrwxrwx  1 mysite mysite   15 Jan 24 11:56 include -> version/include/
lrwxrwxrwx  1 mysite mysite   11 Jan 24 11:56 lib -> version/lib/
drwxr-xr-x  5 mysite mysite 4096 Jan 24 11:56 local/
lrwxrwxrwx  1 mysite mysite   13 Jan 24 11:56 share -> version/share/
drwxr-xr-x  2 mysite mysite 4096 Jan 24 09:57 tmp/
drwxr-xr-x 12 mysite mysite 4096 Jan 24 11:56 var/
lrwxrwxrwx  1 mysite mysite   29 Jan 24 11:56 version -> ../../versions/2.0.0p8.cee/

As can be seen, the directories bin, include, lib, share and version are symbolic links. The rest are ‘normal’ directories. This mirrors the separation of software and and data as explained above. The software directory must be accessible as a subdirectory in the site, but it is physically located in /omd/versions, and can also possibly be used by other sites.

Software Data

Directories

bin, include, lib, share

etc, local, tmp, var

Owner

root

Site user (mysite)

Created by

Checkmk installation

Site creation, configuration, and monitoring

Physical location

/omd/versions/2.0.0p8.cee/

/omd/sites/mysite/

File type

Symbolic links

Normal directories

3.3. Software

The software directories, as usual under Linux, belong to root and thus may not be altered by a site user. The following subdirectories are present – those in the example are physically located within the /omd/versions/2.0.0p8.cee, and they are accessible via symbolic links from the site directory:

bin/

Directory for executable programs. Here the cmk command is found, for example.

lib/

C-directories, plug-ins for Apache and Python – and in the nagios/plugins subdirectory – standard monitoring plug-ins, which are mostly written in C or Perl.

share/

The main part of the installed software. Very many components are located in share/check_mk – among others, well over 2,000 check plug-ins.

include/

Contains Include-files for C-programs, which should be linked to libraries in lib/. This directory is not important and is only used if you wish to translate C-programs yourself.

The version/ symbolic link is a ‘intermediate stop’ and serves as a relay point for the version used by the site. During a software update this will be switched from the old to the new version. Nonetheless, please do not attempt to perform an update manually by altering the link, since an update requires a number of other further steps – which will fail.

3.4. Data

The actual data for a site is found in the remaining subdirectories in the site directory. Without exception, these belong to the site user. The site directory itself is also included. Checkmk stores nothing apart from the directories listed there. You can create your own files and directories without problem here, in which tests, downloaded data, copies of log files, etc. can be kept as desired.

The following directories have been predefined:

etc/

Configuration files. These can be edited either by hand or by using Checkmk’s Setup.

Note: The scripts in etc/init.d are actually also ‘configuration’ files, since they are found in etc/. This is based on the same pattern as found in every Linux system under /etc/init.d/. We do advise against changing this script however, since this can lead to conflicts during a software update. Changes to the scripts are not necessary.

var/

Runtime data. All data generated by the monitoring will be stored here. Depending on the number of hosts and services, an immense volume of data can be accumulated – of which the largest part is the performance data recorded in the RRDs.

tmp/

Volatile data. Checkmk and other components store temporary data (which does not need to be retained) here. A tmpfs is therefore mounted here. This is a file system which manages data in RAM, thus generating zero Disk-IO. Restarting the computer results in the loss of all data in tmp/! Stopping and starting the site does not delete the data. Data such as sockets, pipes and PID-files can be found in tmp/run – these are necessary for communication and managing the server processes. Do not use tmp/ for storing your own data, since this directory is mounted to the RAM where space is rather limited. Store your own data directly in the site directory, or in your own subdirectory within it.

local/

Own extensions. A ‘shadow’ hierarchy of the software directories bin/, lib/ and share/ can be found in local/. These are intended for your own changes or extensions to the software. Also applicable here: Under no circumstances store test files, log files, security copies or anything else that does not belong there, in local/. Checkmk could attempt to execute these files as a part of the software. Likewise, in a distributed monitoring the data will also be duplicated to all remote sites.

3.5. Modifying and extending Checkmk – the local files

As just shown in the above table, the local directory with its numerous subdirectories is intended for your own extensions. In a new site, all of the directories in local/ are initially empty.

With the practical tree command you can quickly get an overview of the structure of local. The -L 3 option restricts the depth to 3:

OMD[mysite]:~$ tree -L 3 local
local
├── bin
├── lib
│   ├── apache
│   ├── check_mk -> python3/cmk
│   ├── nagios
│   │   └── plugins
│   ├── python
│   └── python3
│       └── cmk
└── share
    ├── check_mk
    │   ├── agents
    │   ├── alert_handlers
    │   ├── checkman
    │   ├── checks
    │   ├── inventory
    │   ├── locale
    │   ├── mibs
    │   ├── notifications
    │   ├── pnp-rraconf
    │   ├── pnp-templates
    │   ├── reporting
    │   └── web
    ├── diskspace
    ├── doc
    │   └── check_mk
    ├── nagios
    │   └── htdocs
    ├── nagvis
    │   └── htdocs
    └── snmp
        └── mibs

All of the directories in the lowest level are actively integrated in the software. A file stored here will be treated in the same way as if it was in the directory with the same name within /omd/versions/…​ (or respectively, in the logical path from the site under bin, lib or share).

Example: In the site, executable programs will be searched for in bin and in local/bin.

Here it applies that in the case of identical names the file in local always has priority. This enables modification of the software without the need to change installation files in /omd/versions/. The procedure is simple:

  1. Copy the desired file to the appropriate directory in local.

  2. Modify this file.

  3. Restart the appropriate service so that the change can take effect.

Regarding point 3 above, if it is not known exactly which service to which the change applies, simply restart the complete site with omd restart.

3.6. Log files

In Checkmk – as already-described – the log files are stored in the directory var/. All log files of the relevant components can be found there:

OMD[mysite]:~$ ll -R var/log/
var/log/:
total 48
-rw-r--r-- 1 mysite mysite  759 Sep 21 16:54 alerts.log
drwxr-xr-x 2 mysite mysite 4096 Sep 21 16:52 apache/
-rw-r--r-- 1 mysite mysite 8603 Sep 21 16:54 cmc.log
-rw-r--r-- 1 mysite mysite 3175 Sep 21 11:38 dcd.log
-rw-rw---- 1 mysite mysite    0 Oct 27 11:05 diskspace.log
-rw-r--r-- 1 mysite mysite  313 Sep 21 16:54 liveproxyd.log
-rw-r--r-- 1 mysite mysite   62 Sep 21 16:54 liveproxyd.state
drwxr-xr-x 2 mysite mysite 4096 Sep 20 13:44 mkeventd/
-rw-r--r-- 1 mysite mysite  676 Sep 21 16:54 mkeventd.log
-rw-r--r-- 1 mysite mysite  310 Sep 21 16:54 mknotifyd.log
-rw-r--r-- 1 mysite mysite  327 Sep 21 16:54 notify.log
-rw-r--r-- 1 mysite mysite  458 Sep 21 16:54 rrdcached.log
-rw-r--r-- 1 mysite mysite    0 Sep 21 16:52 web.log

var/log/apache:
total 32
-rw-r--r-- 1 mysite mysite 26116 Sep 21 16:54 access_log
-rw-r--r-- 1 mysite mysite   841 Sep 21 16:54 error_log
-rw-r--r-- 1 mysite mysite     0 Sep 22 10:21 stats

var/log/mkeventd:
total 0

On the web interface you can easily configure the extent to which data should be written to the log files by searching in Setup > General > Global settings for all entries with logging:

List of global settings for logging.

Alternatively it is possible to also customize the log levels on the command line in configuration files. The files are each called global.mk, but are located in different directories. Specify the entries if they are not already present, which is the case, if a Factory setting has not yet been changed.

~/etc/check_mk/conf.d/wato/global.mk
notification_logging = 15
alert_logging = 10
cmc_log_levels = {
 'cmk.alert'        : 5,
 'cmk.carbon'       : 5,
 'cmk.core'         : 5,
 'cmk.downtime'     : 5,
 'cmk.helper'       : 5,
 'cmk.livestatus'   : 5,
 'cmk.notification' : 5,
 'cmk.rrd'          : 5,
 'cmk.smartping'    : 5,
}
cmc_log_rrdcreation = None

In this file the Monitoring Core, Notifications and Alert Handlers entries are set:

  • For notification_logging you can choose between the values 10 for Full dump of all variables and command, 15 for Normal logging and 20 for Minimal logging.

  • The alert_logging can be set to 10 for Full dump of all variables or 20 for Normal logging.

  • For cmc_log_levels the amount of logged data increases by increasing the number. Here there are eight gradations (0 to 7) ranging from 0 for Emergency to 7 which stands for Debug.

  • With the three values None, 'terse' and 'full' for cmc_log_rrdcreation you can decide whether the creation of RRDs should be logged and how.

~/etc/check_mk/multisite.d/wato/global.mk
log_levels = {
 'cmk.web'                : 50,
 'cmk.web.auth'           : 10,
 'cmk.web.automations'    : 15,
 'cmk.web.background-job' : 10,
 'cmk.web.bi.compilation' : 20,
 'cmk.web.ldap'           : 30,
}

In this file you can set the User Interface logging.

The amount of logged data increases inversely as the count decreases. The lowest log level is 50 (Critical) while the most data will be logged at 10 which corresponds to the highest (Debug).

~/etc/check_mk/liveproxyd.d/wato/global.mk
liveproxyd_log_levels = {'cmk.liveproxyd': 20}

This file is used for Livestatus Proxy logging. The possible values here correspond to those for User Interface logging.

Important: Log files can quickly become very large if a high level has been set. It is generally advisable to use such settings for a 'temporary' customization, as an aid in problem identification for example.

4. The cmk command

Along with the omd command, which serves for starting and stopping sites, for the basic configuration of components, and for starting a software update, cmk is the most important command. With this a configuration for a monitoring core can be created, checks executed manually, a service discovery performed, and much more.

4.1. The command options

The command cmk is actually an abbreviation for check_mk, introduced to make the command faster to type. The command includes a built-in, very detailed online help, that can be called up with --help option:

OMD[mysite]:~$ cmk -h
WAYS TO CALL:
 cmk  --automation [COMMAND...]          Internal helper to invoke Check_MK actions
 cmk  --backup BACKUPFILE.tar.gz         make backup of configuration and data
 cmk  --cap [pack|unpack|list FILE.cap]  Pack/unpack agent packages (Enterprise only)
 cmk  --check [HOST [IPADDRESS]]         Check all services on the given HOST
...

As you can see in the command above, we have called help with the option -h instead of --help. Because what is true for the command itself is also true for its options: The less there is to type, the faster it goes. Not for all options, but for those that are often needed, there is therefore a short form in addition to the long form. Even though the long form is more intuitive, especially for beginners (check_mk --list-hosts) than the short form (cmk -l), we will use the short form in the User guide. If in doubt, you can always consult the command help. A longer look in the command help is a good idea in any case, as we will not present all options in the User guide.

By entering an option, you start the command cmk in a certain mode. Here follows the overview of the options we will present in this chapter, but also in other parts of the manual:

Option Function

Monitoring core

cmk -R

Restarting the core

cmk -O

Loading a new configuration into the core

cmk -U

Creating a new configuration for the core

cmk -N

Output the Nagios configuration of the core

Checks

cmk myserver123

Executing checks on host myserver123

Services

cmk -I myserver123

Executing a service discovery

cmk --check-discovery myserver123

Runs the discovery check on the host, which checks for new and disappeared services and for new host labels. When a change occurs, the host is 'marked' by creating a file with the host name in var/check_mk/autodiscovery — but only if automatic service configuration update is enabled in Checkmk (in the Periodic service discovery rule set).

cmk --discover-marked-hosts

Performs a service discovery for all hosts for which a discovery check previously executed with cmk --check-discovery found changes — and marked the affected host. By default, this command is executed in a Checkmk site every 5 minutes via cronjob. You can display the cronjobs with crontab -l.

Agents

cmk -d myserver123

Retrieving agent output

cmk -A myserver123

Baking agents

Diagnostics

cmk -l

Listing hosts

cmk --list-tag mytag

Listing hosts with host tag

cmk -D myserver123

Displaying host configuration

Information

cmk -V

Displays the Checkmk version installed in the site.

cmk --paths

Checkmk path overview: What is in which directory?

cmk -L

Listing check plug-ins

cmk -m

Accessing the check plug-ins catalog

cmk -M df

Showing a check plug-in manual page (here of plug-in df)

Special topics

cmk --update-dns-cache

Deletes the DNS cache and re-creates it. For details on the DNS cache, see the article on hosts. By default, this command is executed in a Checkmk site once a day via cronjob.

cmk --cleanup-piggyback

Deletes all obsolete piggyback data in the tmp/check_mk/piggyback/ directory. By default, this command is executed in a Checkmk site once a day via cronjob.

cmk -P

Managing MKPs

cmk --convert-rrds

Converting RRDs

cmk --snmpwalk myswitch

Pulling an SNMP walk from host myswitch

cmk --snmptranslate

Translating an SNMP walk

cmk --create-diagnostics-dump

Creating support diagnostics dump

In some modes, further, specific options are available to you, e.g. you can limit the service discovery to certain checks, e.g. to the check df with the command cmk -I --detect-plugins=df myserver123.

A number of options always work — regardless of the mode with which the command is executed:

Option Function

cmk -v

Prompts cmk to produce a detailed dump of its current activity ('verbose')

cmk -vv

The same as the above, with even more details: ‘very verbose’

cmk --cache

The information is read from cache files, even if they are out of date. The agent is only contacted if no cache file exists. The cached agent data of the host can be found under tmp/check_mk/cache.

cmk --no-tcp

Works like --cache, however it will interrupt with if a cache file is absent or not current. Thus in any situation you can suppress an access to the agent.

cmk --no-cache

The information is always fetched up to date, i.e. no cache files are used.

cmk --usewalk

For SNMP hosts: instead of accessing the SNMP agent this uses a stored SNMP walk, that has been previously pulled with cmk --snmpwalk myserver123. These walks are stored in var/check_mk/snmpwalks.

cmk --debug

If an error occurs, this option ensures that it will no longer be intercepted, rather the original Python exception will be displayed in full. This can be important information for the developer, by showing the exact program location in which the error is located. It will also be very helpful with locating errors in self-written check plug-ins. If when invoking cmk an error is encountered which should be reported to support or feedback, repeat the request with the added --debug option, and attach the Python trace to your e-mail.

In the following section we will show how the commands can be used. The examples are mostly shown in an abbreviated form.

4.2. Commands for the monitoring core

The commercial editions utilizes the Checkmk Micro Core (CMC) as its monitoring core, the CRE Checkmk Raw Edition uses Nagios. An important task for the cmk is the generation of a configuration file that is readable for the core, and which contains all of the configured hosts, services, contacts, contact groups, time periods, etc. On the basis of this information the core knows which checks are to be executed and which objects it should provide using the GUI’s Livestatus.

For Nagios as well as for the CMC, it is fundamental that the number of hosts, services and other objects always remains static during the operation, and that this number can only be altered through the generation of a new configuration, followed by a reloading of the core. With Nagios a restart of the core is also needed. The CMC has a very efficient function for the reloading of its configuration during active processing.

The following table highlights important differences between the configurations of both cores:

Nagios CMC

Configuration file

etc/nagios/conf.d/check_mk_objects.cfg

var/check_mk/core/config

File type

Text file with define-commands

Compressed and optimized binary file

Activation

Core restart

Core command for reloading the configuration

Command

cmk -R

cmk -O

Regenerating the configuration is always necessary if the contents of the configuration file in etc/check_mk/conf.d, or automatically-detected services in var/check_mk/autochecks have been modified. The Setup keeps a record of such changes and highlights them in the GUI as to activating changes. Should you ‘bypass the Setup’ by modifying the configuration manually or with a script, you will also need to attend to the activation manually. The following commands serve this function:

Option Function

cmk -R

Generates a new configuration for the core and restarts the core (analogous to omd restart core). This is the method provided for Nagios.

cmk -O

Generates the configuration for the core and loads this without a restart of the active processing (analogous to omd reload core). This is the recommended variant with the CMC.

Attention: With Nagios as the core this option still functions, but it can lead to memory holes and other instabilities. Apart from that, this option does in any case not perform a genuine reload, rather it internally stops and restarts the process, as it were.

cmk -U

Generates the configuration for the core without activating it.

cmk -N

For diagnostic purposes, this outputs the configuration to be generated on the standard output, without altering the actual configuration file. Here you can enter a host’s name simply in order to view the host’s configuration (e.g. cmk -N myserver123).

To summarize: If you want to customize a Checkmk configuration and activate the changes, in Nagios you will subsequently require:

OMD[mysite]:~$ cmk -R

And with the CMC:

OMD[mysite]:~$ cmk -O

4.3. Executing checks

A second mode in Checkmk deals with the execution of a host’s Checkmk-based checks. With this you can allow all automatically detected, and also manually configured services, to be immediately checked, without needing to bother yourself with the monitoring core or the GUI.

To do this, enter cmk --check followed by the name of a host configured in the monitoring. Since the --check option is the default option of cmk, you can also omit it. In addition, you should always add the two options -n (do not send results to the core) and -v (output all results). More on this in the description of the options below.

OMD[mysite]:~$ cmk -nv myserver123
Checkmk version 2.0.0p8
CPU load             15 min load 0.22 at 8 Cores (0.03 per Core)
CPU utilization      Total CPU: 8.20%
Disk IO SUMMARY      Read: 14.0 kB/s, Write: 316 kB/s, Latency: 442 microseconds
Filesystem /         82.0% used (177.01 of 215.81 GB), (warn/crit at 80.00/90.00%),
Interface 2          [wlo1], (up), MAC: 5C:80:B6:3E:38:7F, Speed: unknown, In: 1.02 kB/s, Out: 902 B/s
Kernel Performance   Process Creations: 67.82/s, Context Switches: 4183.41/s, Major Page Faults: 1.71/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory               Total virtual memory: 37.07% - 6.08 GB of 16.41 GB
Mount options of /   Mount options exactly as expected
NTP Time             sys.peer - stratum 2, offset 16.62 ms, jitter 5.19 ms, last reac
Number of threads    Count: 1501 threads, Usage: 1.19%
TCP Connections      Established: 11
Temperature Zone 0   25.0 °C
Uptime               Up since Jul 29 2021 08:38:32, Uptime: 4 hours 43 minutes
[agent] Version: 2.0.0b5, OS: linux, execution time 0.9 sec | execution_time=0.850 user_time=0.050 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.800

Further tips:

  • Do not use this command in monitored production hosts which use log file monitoring. Log messages are only sent once by agents, and it can happen that a manual cmk -nv ‘catches’ these and that they will then be lost from the monitoring. In such a situation use the --no-tcp option.

  • If Nagios is being used for the core and -n is omitted, the effect will be an immediate actualization of the check results in the core and in the GUI.

  • The command is useful when developing your own check plug-ins, because it enables a quicker test than by using the GUI. If the check fails and returns an UNKNOWN, the --debug option can help to find the problem location in the code.

The following options influence the command:

Option Function

-v

Check results output: Without this option you will only see the output from the service Check_MK itself, and not the results from the other services.

-n

Dry run: Results are not passed to the core, the performance counter is not updated.

--detect-plugins=df,uptime

Restricts the execution to the check plug-ins df and uptime. In the case of SNMP hosts, only the data required for these will be retrieved. This option is practical if you develop your own check plug-ins and only want to test these.

4.4. Retrieving agent output

The command cmk -d retrieves and displays the output from a host’s Checkmk agent. With cmk -d agent data is retrieved in the same way as with the monitoring. It is neither validated nor processed. Thus the data showed exactly matches the data that is turned over to the Agent Controller (when TLS encryption is enabled) or to a tunneling program in case datasource programs are configured.

OMD[mysite]:~$ cmk -d myserver123
<<<check_mk>>>
Version: 2.1.0b5
AgentOS: linux
Hostname: myserver123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
OnlyFrom:
<<<df>>>
udev              devtmpfs     8155492         4   8155488       1% /dev
tmpfs             tmpfs        1634036      1208   1632828       1% /run
/dev/sda5         ext4       226298268 175047160  39732696      82% /
none              tmpfs              4         0         4       0% /sys/fs/cgroup

You can even run cmk -d using the name or IP address of a host that is not configured in the monitoring. In this case legacy settings for the host will be assumed (TCP connection to port 6556, no Agent Controller, no encryption, no datasource program).

4.5. Baking agents

In the commercial editions you can also bake the agents from the command line, as you would otherwise do via the web interface. This gives you the option, for example, of updating the agents regularly, for example via a cronjob.

To bake the agents, use the option -A followed by the name of a host (or several):

OMD[mysite]:~$ cmk -Av myserver123
myserver123...linux_deb:baking...linux_rpm:baking...(fast repackage)...solaris_pkg:baking...windows_msi:baking...linux_tgz:baking...(fast repackage)...solaris_tgz:baking...(fast repackage)...aix_tgz:baking...OK

The output shows that the agent packages available for the host myserver123 have been successfully baked. If you do not specify a host, the packages will be baked for all hosts.

The command only bakes when necessary. If the packages are still up to date, the output will look something like this:

OMD[mysite]:~$ cmk -Av myserver123
myserver123..linux_deb:uptodate...linux_rpm:uptodate...solaris_pkg:uptodate...windows_msi:uptodate...linux_tgz:uptodate...solaris_tgz:uptodate...aix_tgz:uptodate...OK

You can still force baking with the -f (force) option.

4.6. Listing hosts

The cmk -l command simply lists the names of all hosts configured in the Setup:

OMD[mysite]:~$ cmk -l
myserver123
myserver124
myserver125

Because the data is provided ‘naked’ and ‘unprocessed’, it is easy to use in scripts – for example a loop across all host names can be constructed:

OMD[mysite]:~$ for host in $(cmk -l) ; do echo "Host: $host" ; done
Host: myserver123
Host: myserver124
Host: myserver125

If, instead of echo you insert a command that performs something meaningful, this can be really useful.

The cmk --list-tag invocation likewise outputs host names, but also offers the possibility of filtering by host tags. Simply enter a host tag and you will receive all hosts having this tag. The following example lists all hosts that are monitored by SNMP:

OMD[mysite]:~$ cmk --list-tag snmp
myswitch01
myswitch02
myswitch03

Enter multiple tags and they will be linked with ‘and’. The below command delivers all hosts that are monitored by both SNMP and Checkmk agents. As no hosts fulfill this condition, the output remains empty:

OMD[mysite]:~$ cmk --list-tag snmp tcp

4.7. Displaying host configuration

For one or more specified host, cmk -D displays the configured services, host tags, labels and other attributes. Because the list of services is so extensive it can look somewhat confusing on the terminal. Send the output through less -S to avoid a break:

OMD[mysite]:~$ cmk -D myserver123 | less -S
myserver123
Addresses:              192.168.178.34
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:mysite], [tcp:tcp]
Labels:                 [cmk/check_mk_server:yes], [cmk/os_family:linux]
Host groups:            mylinuxservers
Contact groups:         all
Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
  TCP: 192.168.178.34:6556
  Process piggyback data from /omd/sites/mysite/tmp/check_mk/piggyback/mycmkserver
Services:
Type of agent:          TCP (port: 6556)
Is aggregated:          no
Services:
  checktype        item              params
  ---------------- ----------------- ------------
  cpu.loads        None              (5.0, 10.0)
  kernel.util      None              {}

4.8. Checkmk path overview

The cmk --paths command displays in which directories Checkmk expects which things. This list does not cover the complete Checkmk system, rather only those things that the command line tool cmk itself works with. Nonetheless it sometimes helps to locate things more quickly:

OMD[mysite]:~$ cmk --paths
Files copied or created during installation
  Main components of check_mk             : /omd/sites/mysite/share/check_mk/modules/
  Checks                                  : /omd/sites/mysite/share/check_mk/checks/
  Notification scripts                    : /omd/sites/mysite/share/check_mk/notifications/
  Inventory plugins                       : /omd/sites/mysite/share/check_mk/inventory/
  Agents for operating systems            : /omd/sites/mysite/share/check_mk/agents/
  Documentation files                     : /omd/sites/mysite/share/doc/check_mk/
  Check_MK's web pages                    : /omd/sites/mysite/share/check_mk/web/
  Check manpages (for check_mk -M)        : /omd/sites/mysite/share/check_mk/checkman/
  Binary plugins (architecture specific)  : /omd/sites/mysite/lib/
  Templates for PNP4Nagios                : /omd/sites/mysite/share/check_mk/pnp-templates/
  Startscript for Nagios daemon           : /omd/sites/mysite/etc/init.d/core
  Path to Nagios executable               : /omd/sites/mysite/bin/nagios

Configuration files edited by you
  Directory that contains main.mk         : /omd/sites/mysite/etc/check_mk/
  Directory containing further *.mk files : /omd/sites/mysite/etc/check_mk/conf.d/

4.9. Information on the check plug-ins

Checkmk provides a large number of ready to use plug-ins as standard. In every release a few new ones are added, and Version 2.0.0 already includes around 2,000 plug-ins. Three cmk command options give you access to information about these plug-ins.

cmk -L produces a table of all plug-ins with their name, type and a description. At the same time, any self-written plug-ins stored in local/ will also be listed.

The following are possible types:

agent

Evaluates the data from a Checkmk agent. This is (normally) retrieved via TCP Port 6556.

snmp

Serves the monitoring of devices via SNMP.

active

Calls a standard type of Nagios-compatible plug-in for the monitoring. Here Checkmk actually only adopts the configuration.

The list can of course be filtered simply with grep if something specific is being searched for:

OMD[mysite]:~$ cmk -L | grep f5
f5_bigip_apm                      snmp  F5 Big-IP: Number of Current SSL/VPN Connections
f5_bigip_chassis_temp             snmp  F5 Big-IP: Chassis Temperature
f5_bigip_cluster                  snmp  F5 Big-IP: Cluster State, Up to Firmware Version 10
f5_bigip_cluster_status           snmp  F5 Big-IP: Active/Active or Passive/Active Cluster Status (< V11.2)
f5_bigip_cluster_status_v11_2     snmp  F5 Big-IP: Active/Active or Passive/Active Cluster Status (> V11.2)
f5_bigip_cluster_v11              snmp  F5 Big-IP: Cluster State for Firmware Version >= 11
f5_bigip_conns                    snmp  F5 Big-IP: Number of Current Connections
f5_bigip_cpu_temp                 snmp  F5 Big-IP: CPU Temperature
f5_bigip_fans                     snmp  F5 Big-IP: System Fans
f5_bigip_interfaces               snmp  F5 Big-IP: Special Network Interfaces
f5_bigip_mem                      snmp  F5 Big-IP: Usage of Memory
f5_bigip_mem_tmm                  snmp  F5 Big-IP: Usage of TMM Memory
f5_bigip_pool                     snmp  F5 Big-IP: Load Balancing Pools
f5_bigip_psu                      snmp  F5 Big-IP: Power Supplies
f5_bigip_snat                     snmp  F5 Big-IP: Source NAT
f5_bigip_vcmpfailover             snmp  F5 Big-IP: Active/Active or Passive/Active vCMP Guest Failover Status
f5_bigip_vcmpguests               snmp  F5 Big-IP: Show Failover States of all vCMP Guests Running on a vCMP Host
f5_bigip_vserver                  snmp  F5 Big-IP: Virtual Servers

If you want more information on a certain plug-in, documentation as manual page (or man page) can be called up with cmk -M:

OMD[mysite]:~$ cmk -M f5_bigip_pool

This produces the following output:

Example of a check plug-in manual page.

Using cmk -m with no further options will access a complete catalog of all check plug-ins man pages.

OMD[mysite]:~$ cmk -m

You can navigate interactively in this catalog:

Main menu for selecting a manual page.
Submenu for selecting a manual page.

5. Configuration without Setup

The Setup menu is a great configuration tool. There are however good reasons to prefer a configuration with text files in the good, old Linux tradition. If you are of the same opinion there is some good news: Checkmk can be completely configured using text files. And since the Setup menu actions does no more than process (this same) text files, this is not even an either/or situation.

5.1. Where is the documentation?

If you are expecting a comprehensive compendium covering the exact structure of all of the configuration files used by Checkmk, we will unfortunately have to disappoint you here. The complexity and diversity contained in the configuration files is simply too much to describe completely in this User guide.

The following example shows an entire completed parameter set for the check plug-in which monitors file systems in Checkmk. Because of the many parameters, the screenshot is divided into two parts, and set in lower-case characters:

Complete parameter set of the check plug-in for monitoring file systems.

The corresponding passage in the configuration file looks like this (somewhat more nicely formatted):

{ 'inodes_levels'      : (10.0, 5.0),
  'levels'             : (80.0, 90.0),
  'levels_low'         : (50.0, 60.0),
  'magic'              : 0.8,
  'magic_normsize'     : 20,
  'show_inodes'        : 'onlow',
  'show_levels'        : 'onmagic',
  'show_reserved'      : True,
  'subtract_reserved'  : False,
  'trend_mb'           : (100, 200),
  'trend_perc'         : (5.0, 10.0),
  'trend_perfdata'     : True,
  'trend_range'        : 24,
  'trend_showtimeleft' : True,
  'trend_timeleft'     : (12, 6)},

As can be seen, there are more than 10 different parameters, each with its own individual logic. Some are configured using floating-point numbers, (0.8), some with integers (24), some with keywords ('onlow'), some with boolean values (True), and others using tuples to code various combinations of these ((5.0, 10.0)).

This is just one example from over 2,000 plug-ins. And there are of course other configurations possible as check parameters: One only needs to think of time periods, Event Console rules, user profiles, and many more.

Of course that doesn’t mean you cannot use text files as configuration! If you don’t yet know the exact syntax for your chosen configuration task, you only need the correct tool for it – and this tool we call Setup:

  1. Create a Checkmk test site.

  2. Use the Setup menu to configure the desired parameters.

  3. Locate the configuration file that has been changed as a result (more on this below).

  4. Carry over the exact syntax from the relevant section of this file in your production system.

You thus only need to know in which file Setup writes.

Note: When it comes to the names of directories, files or even file contents, you will often find the term wato. WATO is the abbreviation for Web Administration Tool: the Checkmk configuration tool up to and including version 1.6.0. The function of WATO has been taken over by the Setup menu, or Setup for short, from version 2.0.0 onwards. Although WATO has been (almost) completely replaced by Setup in the web interface, it lives on in the file system.

5.2. Which configuration file is used?

There is a practical command for finding out which file Setup has just changed: find. By invoking ‘find’ with the following parameters you can find all files (-type f) under etc/ which have been altered within the last minute (-mmin -1):

OMD[mysite]:~$ find etc/ -mmin -1 -type f
etc/check_mk/conf.d/wato/rules.mk

The basis of a configuration is always the etc/check_mk directory. Below this is a subdivision into various domains, which generally apply to a specific service. At the same time each has a directory with the suffix .d, under which all files with the suffix .mk will be read automatically in alphabetic order. In some there will also be a main file which is read first of all. This is intended only for manual alteration, and is never modified by Setup.

Domain Configuration directory Main file Changes activated

Monitoring

etc/check_mk/conf.d/

main.mk

cmk -O or cmk -R

GUI

etc/check_mk/multisite.d/

multisite.mk

automatically

Event Console

etc/check_mk/mkeventd.d/

mkeventd.mk

omd reload mkeventd

Notification spooler

etc/check_mk/mknotifyd.d/

automatically

5.3. Setup and configuration files

Below the .d/ configuration directories there is always the subdirectory wato, e.g. etc/check_mk/conf.d/wato. The Setup fundamentally only reads and writes here. However, the service responsible for the configuration directory also reads the other files in "its" .d directory, if you have stored some manually-created files there. This means:

  • If it is required that the manual configuration be visible and editable in the Setup, use the existing paths.

  • If it is required that the configuration simply functions, but is not visible in the Setup, then use your own files externally to wato/.

  • If it is required that the configuration be visible in the Setup, but not changeable, some of the files can be locked.

Locking files and folders

A common reason for manually creating configuration files without Setup is needing to import hosts to be monitored from a Configuration Management Database (CMDB). Here, in contrast to methods using the REST API, you create the folders for the hosts with a script directly in etc/check_mk/conf.d/wato and in each case the file hosts.mk for the hosts contained in the folder and possibly also the file .wato, which contains the folder attributes.

If this import is not just a one-off, rather it is to be repeated regularly because the CMDB is the leading system, it would be very impractical if your users make any changes to the files using the Setup, as these will be lost with the next import.

A hosts.mk-file can be locked by inserting a line for the lock attribute:

hosts.mk
# Created by WATO
# encoding: utf-8

_lock = True

When opening this folder in the Setup, the following message is displayed above the Hosts list:

Message that editing of hosts in the folder is locked.

All actions which would alter the hosts.mk file are thus locked in the GUI. This does not apply to the service discovery of course. A host’s configured services are stored in var/check_mk/autochecks/.

The folder attributes can also be locked. This is done by an entry in the folder’s .wato file: set the lock attribute to True in the file’s dictionary:

.wato
{'title': 'My folder',
 'attributes': {},
 'num_hosts': 1,
 'lock': True,
 'lock_subfolders': False,
 '__id': '7f2a8906d3c3448fac8a379e2d1cec0e'}

If you also set the lock_subfolders attribute to True, you prevent the creation and deletion of subfolders.

Locking of other files – such as rules.mk, for example – is not currently possible.

5.4. The files syntax

In purely formal terms, all of Checkmk’s configuration files are written in Python 3 syntax. There are two types of files:

  • Those which are executed like a script by Python. Among these is, e.g., hosts.mk.

  • Those which are read in as values by Python. Among these is, e.g., .wato.

The executable files can be recognized by their having variables which are substituted for assignments with values (=). The other files usually contain a Python dictionary which begin with an opening bracket {. Sometimes they are simple values.

If a non-ASCII character is required in a file (a German Umlaut (ä, ö, ü), for example), the following comment must be coded in the first or second line:

somefile.mk
# encoding: utf-8

A syntax error will otherwise occur when reading the file. For further tips on Python syntax we recommend visiting a specialist site, for example: The Python Language Reference.

5.5. Checking configuration files

If you edit configuration files in etc/check_mk/ manually, it is a good idea to have the configuration checked. You can do this with the tool cmk-update-config, which is actually only executed automatically after a version update with omd update:

OMD[mysite]:~$ cmk-update-config
...
Verifying Checkmk configuration...
 01/05 Cleanup precompiled host and folder files...
 02/05 Rulesets...
Exception while trying to load rulesets: Cannot read configuration file "/omd/sites/mysite/etc/check_mk/conf.d/wato/rules.mk":
 '[' was never closed (<string>, line 44)

You can abort the update process (A) and try to fix the incompatibilities or try to continue the update (c).
Abort update? [A/c]
...

For example, in the excerpt above you can see the reference to an unclosed bracket.

On this page