Install check_mk agent on Centos 7

This is an update to the other check_mk client instructions which is more general.There are 3 major steps: a. installing the required package repository, b.installing the client package and c. configuring systemd and the firewall to start and allow the check-mk server to access it.

1. For this you need to install the epel repository by installing the epel package available from EPEL. Then, you can search the repository to verify that it does have the check-mk client.
yum search check-mk-agent
2. The above command should show the current version of the agent available to be installed. To install it: yum install check-mk-agent
3. Change to the systemd service directory: cd /etc/systemd/system
4. Copy the check-mk agent socket file: cp /usr/lib/systemd/system/check_mk.socket .
5. Create a link for the check-mk agent service file: ln -s /usr/lib/systemd/system/check_mk@.service .
6. Enable the check-mk agent service: systemctl enable check_mk.socket . Then start it: systemctl start check_mk.socket.This can be verified with the command systemctl status check_mk.socket. Look for the line beginning with "Active" which should say "active (listening)".
7. Open the port in the firewall: firewall-cmd --zone=public --add-port=6556/tcp --permanent . The --permanent option will save the rule and make it permanent, even if the server is rebooted.
 After this, add the server at the OMD or check_mk server and you're done.
For Extra Credit: Be aware of where the locations for the plugins are by running rpm -ql check-mk-agent to see all the files installed and where the plugins are. Copy and configure those that you need to /usr/share/check-mk-agent/plugins to be activated and used by the agent.

How to quickly install check_mk agent plugins

Check_mk has many agent plugins that a lot of us are not aware of. Some of them would open up monitoring opportunities that were not even considered about. Best of all, some of these plugins use APIs readily available to talk to applications and services. Theses same APIs are used by other commercial programs so the information gleamed from them is commercial-grade.

The hardest part of installing the application is to copy the plugin. The best option is to use the plugin provided by the monitoring Check_mk system. This ensures that the plugin works with the check_mk agent installed on the host. But it may not be convenient to download the plugin to the PC and upload it back to another server. So this is a really quick way to do this.

1. First, ssh to the host and go the plugin directory, usually /usr/lib/check_mk_agent/plugins
2. Open the browser to the WATO-Configuration ==> Monitoring Agents window of your Check_mk / OMD installation. 
3. Right-click on the plugin you wish to upload to the host and copy the URL
4. Go to the ssh window and type wget.
5. Then paste the URL, which should look something like  http://192.168.1.100/prod_3f/check_mk/agents/plugins/apache_status. This is for the apache_status plugin.
6. Since Check_mk / OMD requires a login, edit the line to look like this  http://<Username>:<password>@192.168.1.100/prod_3f/check_mk/agents/plugins/apache_status and press Enter.
7. Change the permission of the plugin to be executable using chmod ugo+x apache_status 

That should do the trick. Go to Wato-Configuration ==> Hosts. Select the Host that the plugin was added to. Click on the Services button and activate the (new) missing services. Apply the configuration and the plugin should now be active. Check the services of that Host to see the new services.
Some plugins have specific requirements, like the credentials to access the service it monitors. The apache_status plugin needs a specific configuration to be in the apache configuration file. For example, the mk_mysql agent requires the credentials to be in the /etc/check_mk/mysql.cfg file. The file should look like this (replace with the correct information)

[client]
user=mysqluser
password=mysqluserpassword

Meet these requirements and  the agent can run and report on the service. Finally, you can go into WATO and check for the Host's Services menu item. The newly discovered services can be enabled here.

Check_mk Inventory not updating Solved

If you are just installing the check_mk inventory plugin, please understand the the plugin only run between a set interval times. This is because the inventory process is places a high load on the server and repeating it would impact the performance of the server. The default interval is 4 hours.
You can force an inventory check by doing the following

1. Find and delete the mk_inventory.last.* file. This is the file that stores the last time the inventory process was run
2. Go to the check_mk web interface and trigger / refresh the Check_MK HW/SW Inventory service.

Observe the Output of the check plugin entry in the service detail table for Check_MK HW/SW Inventory service. It should read a lot of entries.

Setting up the Check_mk Inventory plugin

Check_mk and by extension, OMD have a feature to audit the hardware and software inventory of a server that is being monitored. This can be used to find servers below of a specific specification (e.g RAM too low) or details of an installed software package. To do this, the Check_mk inventory plugin has to be installed. This is a two phase process. First we need to install the check_mk inventory plugin and then we need to setup Check_mk/OMD to start polling the server for inventory information.

Installing the check_mk inventory plugin

The plugin itself is very similar to how other plugins are installed.

1. Obtain the plugin. This can be found in OMD at Wato Configuration ==> Monitoring agent. Download the file.
2. Copy the file to the check_mk plugin directory. This can be at either in /usr/lib/check_mk_agent/plugins/ for RedHat/CentOS or in a directory called plugins located in the same directory as the check_mk_agent.exe file.
3. For Linux hosts, change the file permissions to allow the plugin to be executable.
4. For Windows hosts, the system account running the check_mk agent has the appropriate permissions to check the service the plugin is for.

Configuring Check_mk to poll for inventory information

Once the plugin is installed, check_mk must be told which servers to poll for the inventory information. Polling for this is done less often than polling for status. It is done via a ruleset.

1. In Wato Configuration, go to Hosts.
2. Click on the Rulesets button on the top.
3. Click on Hardware/Software Inventory button. (bottom left button)
4. Click on Do hardware/software Inventory
5. If there are no rules in the folder, click Create rule in folder. Otherwise click on the green pencil icon to edit the existing rule.
6. Check the box marked Specify explicit host names and enter the hostname of the host to be inventoried. 
7. If this is an existing rule, understand the current settings and modify accordingly. If you are not sure, go back, create a new rule and repeat rule 6. 

 It will take some time for the hardware and software to be inventoried. Go to View ==> All Hosts. The hosts that are being inventoried have an icon showing a blue window with a CD in front of it.

Click on this icon to show the inventory. The inventory is displayed as a tree, so you need to expand it to show more items. If you are seeing only the hardware inventory showing only RAM information, the plugin is not working or hasn't been run yet. Click the refresh icon in the All Hosts screen to trigger this.

There are more inventory related operation in the Main Menu. Once the inventory is populated, these functions will be useful to discover or find hardware or software installed in the hosts that are being monitored.
If a few hours has passed and the inventory is still not updating, try this to immediately force an inventory check.

Installing Check_MK client on a Linux server

Installing a Check_Mk client is not hard but can be quite lengthy. This means there are a lot of things to do and a lot of chances to make a mistake. Hopefully the steps below will help you set up a client without too much fuss. I've broken it into 4 major steps.
First, some pre-requisites: Xinetd has to be installed on the server to be monitored. Second, the Check_mk or OMD server has to be installed or at the very least, you know what the IP address of the server will be and have access to the Check_mk client files.
The 4 Steps are :

1. Install the Check_MK client program / agent to the server to be monitored
1. Find where check_mk client package is stored on the monitoring server. Usually at /opt/omd/versions/1.30/share/check_mk/agents. If you stored the client files on a different server, use that instead. There is also a URL available at the Wato configuration interface under Monitoring Agents.
2. Find the client that matches the OS of the server to be monitored / target server. If for a Linux server, find the matching package. Eg. for CentOS the agent is check-mk-agent-1.2.6p12-1.noarch.rpm. There is a MSI Windows installer file for Windows server.
3. Copy the file to server to be monitored. For example, for a CentOS server: scp /opt/omd/versions/1.20/share/check_mk/agents/check-mk-agent-1.2.6p12-1.noarch.rpm user@targetserver:/tmp
4.  Install the agent:  yum install check-mk-agent-1.2.6p12-1.noarch.rpm
5. Check whether the client is working by running the file you just copied from the prompt (check_mk_agent). 
6. For extra points, copy the waitmax program in the same directory to the same location.This program is sometimes used by the check_mk_agent program. If you installed from a package, this is automatically done.
2. Configure xinetd to respond to polls from monitoring server 
1. Copy the xinet.conf file from the monitoring server to the target server's /etc/xinetd.d directory, renaming the file into check_mk in the process. This can be achieved with a command like this:   scp /opt/omd/versions/1.20/share/check_mk/agents/xinetd.conf user@targetserver:/etc/xinetd.d/check_mk
2. Configure the xinetd check_mk file.
1. Uncomment the #only_from  line and add the IP address of the monitoring server.
2.  Extra points: Add a line to configure the agent to only listen to only one interface. This is useful is the server has multiple interfaces. The line to listen only on 192.168.1.1 is : bind = 192.168.1.1
3. Restart the xinetd on the target server using the comment service xinetd restart.
3. Create Firewall rules to allow the monitoring server access to Check_MK client (port 6556). This depends on the firewall being used. Below is an example how to do it on a CentOS/RedHat server running iptables.
1.  Edit the file /etc/sysconfig/iptables
2. Find other lines that look similar to the following line and add this : -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 6556 -j ACCEPT
3.  Make sure that the line you added is between the lines marked *filter and COMMIT.
4. Apply the new setting by running the command: service iptables restart.
4. Add entry on the monitoring server. Use WATO to add the new host.

If your security processes don't allow direct copying to the any of the directories mentioned above, copy the files to a directory where you have the rights to, first. Then logon onto the target server, escalate your privileges with the sudo command and copy the files to the correct location.

Solved! New Check_mk user unable to access Network Topology

After creating a new user on OMD, that user should get access to the features that his rights/role allows him to. This applies across all of the OMD apps, Nagios, Check_mk, DokuWiki and NagVis. 

However, users created by Wato, the tool to configure Nagios, can't get access to the Network Topology item on the side menu. A message saying that the user is not allowed comes up, even though the user's role is an administrator. Stangely enough, even if you clone the user 'omd' to create a new user, the new user still won't be able to use the Network Topology.

It seems this is due to a bug in NagVis or WATO. It could also be a mismatch in role mapping that OMD uses to create users across of it's apps. But there is a workaround.

Basically the user in Nagvis needs to be given admin privileges or  Read-only privileges. Say you are dealing with the account 'mike'.

1. Open the NagVis OMD URL, http://<omd>/<site name>/nagvis e.g. http://monitor.myco.com/mysite/nagvis
2. If you are not already logged on, it will prompt for a username amd password. Enter the OMD/Check_admin administratror, which is usually 'omd'.
3. Choose User Menu on the top right and choose Manage User.
4. In the Modify User section select the user 'mike'
5. Move the role 'Users (Read-only)' from the left column (Available Roles) to the right column (Selected Roles).
6. Move the role 'Guest' from the right column to the left.
7. Click Modify User button to apply the changes.

Try logging in Check_mk with the account 'mike'. You have to do the same for all new users. Until the bug is fixed, at least.