If you want to monitor a specific tcp port on Icinga2 you just need to add another tcp variable:

object Host "host-web" {
     import "generic-host-tpl"

     address = "YOUR_SERVER_IP_ADDRESS"
     vars.os = "Linux"

     vars.tcp["OPTIONAL_NAME"]={
         tcp_port=8181
     }

Here in above sample we are monitoring port 8181 of host-web which has the IP address of YOUR_SERVER_IP_ADDRESS (change it to your server ip address remote or local).

#linux #monitoring #sysadmin #icinga2 #host #tcp #port

In icinga2 monitoring tools, you may have used check disk space command check_snmp_storage. In its default behaviour it lists all
the partitions which is present on disk either mounted by NFS or /run, etc.

In Incinga2 you would have a command called snmp-storage in /etc/icinga2/conf.d/commands. To exclude unnecessary partitions from all disks you can pass -e (e stands for exclude) argument to the perl command. You can use -m to pass regex to say which partitions you want to use.

e, --exclude:
Select all storages except the one(s) selected by -m
No action on storage type selection.

The final config for snmp-storage to exclude a list is like below:

object CheckCommand "snmp-storage" {
         import "snmp-manubulon-command"

         command = [ ManubulonPluginDir + "/check_snmp_storage.pl" ]

         arguments += {
                 "-m" = "$snmp_storage_name$"
                 "-f" = {
                         set_if = "$snmp_perf$"
                 }
                 "-w" = 80
                 "-c" = 90
                 "-H" = "$address$"
                 "-m" = "^Cached|^Shared|^Swap|^/run|^Physical|^Memory|^Virtual|^/dev|^/sys"
                 "-e" = ""
         }

        vars.snmp_perf = true
 }

In brief the combination of -m & -e means: select all partitions except the ones listed in front of -m.

#icinga2 #monitoring #snmp #snmp_storage #exlude #exclude_partition

How to monitor network cards on Icinga2? (part-1)

Yesterday I've been on a task of monitoring network cards of all our servers and infrastructure to check the bandwidth in/out and send alarms based on some criteria. In Icigna2 we have a library from nagios called check_nwc_health. Download the script from https:// labs.consol.de/nagios/check_nwc_health/index.html.

Move the script to /usr/lib/nagios/plugins on a server that you have installed Icinga2. If you run it all alone you will get some helps that you could be useful.

Some important usages of the script:

- list interfaces of a specific server (we assume snmp has been installed on the destination server):

./check_nwc_health --mode list-interfaces --hostname YOUR_TARGET_SERVER_IP  --community YOUR_COMMUNITY_STRING

The output would be something like below (it can be different in your case):

000001 lo
 000002 Device 1af4:0001 2
 000003 Device 1af4:0001 3
 000004 docker0
 OK - have fun

The interface name is given in front of serial numbers which is lo, Device 1af4:0001 2 or docker0. These interface names are important and will be used in icinga2 to add network card to hosts.


Another mode for the script is interface-usage that shows in/out bandwidth. The output can be something like follow:

OK - interface Device 1af4:0001 2 (alias eth0) usage is in:0.00% (7058.67bit/s) out:0.00% (5603.67bit/s) | 'Device 1af4:0001 2_usage_in'=0%;80;90;0;100 'Device 1af4:0001 2_usage_out'=0%;80;90;0;100 'Device 1af4:0001 2_traffic_in'=7058.67;0;0;0;0 'Device 1af4:0001         2_traffic_out'=5603.67;0;0;0;0

OK, the important part is over and we can list all server network interfaces plus the usage of a specific network interface. In the next part we will explain the Icinga2 part to add the command and the service to icinga2.


#icinga2 #icinga #nagios #check_nwc_health #network #monitor

How to monitor network cards on Icinga2? (part-2)

Ok for now we have added the plugin to nagios folder and ran some tests on target server's network interfaces. We need to add a command to Icinga2 to use it in service section of Icinga2. To create a new command create a new file in /etc/icinga2/conf.d/commands/check_nwc_command.conf and with the following content:

object CheckCommand "YOUR_COMMAND_NAME" {
     import "plugin-check-command"
     command = [ PluginDir + "/check_nwc_health", "--mode", "interface-usage" ]
     arguments = {
         "-H" = "$address$"
         "-C" = "$community$"
         "--name" = "$int$"
     }
 }

In brief it creates a new command called YOUR_COMMAND_NAME that calls the script check_nwc_health with interface-usage argument to get the bandwidth data.


Now we need to use this command in a service. We have to create a new service which will be used in our hosts configuration sections /etc/icinga2/conf.d/services/if_traffic.conf:

apply Service for (display_name => config in host.vars.int) {
     import "generic-service"
     check_command = "YOUR_COMMAND_NAME"
     vars += config
     assign where host.vars.int
 }

Again in brief the service will be applied on hosts that have a variable section of int in their configuration that we will see a little bit later. YOUR_COMMAND_NAME is the name that we have given in the first part when creating the command.


The final part is to add this service to your desired host. Go to /etc/icinga2/conf.d/hosts and open the file which relates to your host. Host files content start with:

object Host "host-54 (Infra)" {

Add the service like below into your host:

vars.int["YOUR DISPLAY NAME"] = {
     int = "Device 1af4:0001 2"
     community = "YOUR SERVER COMMUNITY STRING"
 }

int is the part that we give the interface name, this should be given from the output of list-interfaces in part-1.

You can go even further like me :) and add these data into Grafana dashboard to have a better understanding of what is happening around you.

#icinga2 #icinga #service #host #command #nagios #interface #network

How to ignore a specific partition in Icinga2 monitoring server?

Sometimes a specific partition has been mounted on many servers. Now if you monitor disk partitions on Icinga2 and a warning message appears on that specific partition you will get notifications as many as your servers.

Icinga2 uses Snmp Storage Check in order to get disk partitions and their data. The command is located in /etc/icinga2/conf.d/ commands/snmp-storage.conf. (Its name maybe different in your case)

This command will use check_snmp_storage.pl nagios plugin, the overall structure of it is similar to:

object CheckCommand "snmp-storage" {
         import "snmp-manubulon-command"

         command = [ ManubulonPluginDir + "/check_snmp_storage.pl" ]

         arguments += {
                 "-m" = "$snmp_storage_name$"
                 "-f" = {
                         set_if = "$snmp_perf$"
                 }
                 "-w" = 87
                 "-c" = 95
                 "-H" = "$address$"
                 "-m" = "^Cached|^Shared|^Swap|^/run|^Physical|^Memory|^Virtual|^/dev|^/sys|^/mnt/remote_folder"
                 "-e" = ""
         }

         vars.snmp_perf = true
 }

-m parameter will ignore partitions. Here we have provided many partitions like /sys or /mnt/remote_folder. These partitions will be ignored all over. Add your desired partitions to this section.

To read more about this Perl plug-in head over to the below link:
- http://nagios.manubulon.com/snmp_storage.html

#monitoring #icinga2 #snmp #storage #snmp_storage

How do you upgrade Icinga2 from a very old version?

The short answer is that you need to do an incremental upgrade on its database.

But how exactly?

apt-get update

apt-cache policy icinga2-ido-mysql

The candidate section displays what version is available for you system. Do the same for icingaweb2:

apt-cache policy icingaweb2

Upgrade these packages alltogether:

apt-get upgrade -o Dpkg::Options::="--force-confold" -y

Update icinga2 packages:

apt-get upgrade icinga2 icinga2-bin icinga2-common icinga2-ido-mysql libicinga2
 apt-get install icinga2-bin
 apt-get install icinga2

Now check icinga2 log in tail -f /var/log/icinga2/icinga2.log, you may see errors like:

[2017-12-21 12:00:22 -0600] critical/IdoPgsqlConnection: Schema version '1.14.2' does not match the required version '1.14.3' (or newer)! Please check the upgrade documentation at https://docs.icinga.com/icinga2

Here you need to upgrade MySQL schemas, go to /usr/share/icinga2-ido-mysql/schema/upgrade path and now import new schemas to Icinga2 MySQL database. As I was in version 2.6.0 I went for 2.8.0:

mysql -u icinga2 -p icinga2 < 2.8.0.sql

It will prompt for password of user icinga2, enter the password and go on.

The content of MySQL Schemas is as follows:

2.0.2.sql  2.1.0.sql  2.2.0.sql  2.3.0.sql  2.4.0.sql  2.5.0.sql  2.6.0.sql  2.8.0.sql  2.8.1.sql

NOTE: to get the password for user icinga2 open /etc/icinga2/features-enabled/ido-mysql.conf file and get the password from here.


#linux #icinga2 #monitoring #icinga2 #icingaweb #upgrade

In case your upgrade failed, try to start icinga service and in the meantime check the logs in another console:

service icinga2 start

Check logs to see what is the error:

tail -f /var/log/icinga2/icinga2.log

If it point to a specific config file, change the config as reported or in case you don't need it remove it. It happened for me on another Icinga2 server and I removed the old config and tried to apply the new schemas. It gave another error on mysql grants, I gave all permissions to the icinga2 user:

mysql -u root -p
 GRANT ALL PRIVILEGES ON icinga2_database_name.* TO 'icinga2_username'@'localhost'
 flush privileges;

Now all things should be up and working now.

#icinga2 #monitoring #upgrade #schema_update #icinga

How to check MongoDB replication lag in Icinga2 and get notified when it is over 15 seconds?

We assume here that you have a replica set in place. First download the python script for our nagios plugin:

cd /usr/lib/nagios/plugins
 git clone git://github.com/mzupan/nagios-plugin-mongodb.git

Now the Icinga2 part. You first need to create a command for replication lag check:

cd /etc/icinga2/conf.d/commands

Create a new file replication_lag.conf:

object CheckCommand "check_replication_lag" {
     import "plugin-check-command"
     command = [ PluginDir + "/nagios-plugin-mongodb/check_mongodb.py", "-A", "replication_lag" ]
     arguments = {
         "-H" = "$mongo_host$"
         "-P" = "$mongo_port$"
     }
 }

Create a new file in services folder called replication_lag.conf:

apply Service for (display_name => config in host.vars.replication) {
     import "generic-service"
     check_command = "check_replication_lag"
     vars += config
     assign where host.vars.replication
 }

This service gets enabled where it finds replication in host config. Now in secondary mongoDB hosts configuration add the below part:

vars.replication["Secondary DB"] = {
     mongo_host = "slave.example.com"
     mongo_port = 27017
 }

#sysadmin #icinga2 #mongodb #replication #replication_lag #nagios_plugin