SystemAdministration/Icinga: Difference between revisions

From SoylentNews
Jump to navigation Jump to search
No edit summary
 
(10 intermediate revisions by one other user not shown)
Line 1: Line 1:
== Sentinel: Icinga / Server & Services Monitoring ==
== Sentinel: Icinga / Server & Services Monitoring ==


'''Important directories:'''
=== Important Directories ===


{| class="wikitable"
{| class="wikitable"
Line 20: Line 20:
|}
|}


'''Testing Icinga's config and reloading:'''
=== Useful links ===
 
[http://docs.icinga.org/latest/en/ Icinga documentation]
<section begin=testandreload />
=== Testing Icinga's config and reloading: ===


When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:
When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:
Line 29: Line 33:


<code># service icinga reload</code>
<code># service icinga reload</code>
<section end=testandreload />


== Access ==
== Access ==
Line 47: Line 52:


== Servers ==
== Servers ==
[http://docs.icinga.org/latest/en/objectdefinitions.html#host Icinga's "host" documentation]


=== Adding a server ===
=== Adding a server ===
Line 76: Line 83:


If you want to remove a server, remove the config and check '''/etc/icinga/objects/groups/hostgroups_icinga.cfg''' and remove any references to the server.
If you want to remove a server, remove the config and check '''/etc/icinga/objects/groups/hostgroups_icinga.cfg''' and remove any references to the server.
{{#lst:SystemAdministration/Icinga|testandreload}}


=== Conclusion ===
== Host groups ==


Don't forget to check Icinga's config and reload (see the note at the top)
[http://docs.icinga.org/latest/en/objectdefinitions.html#objectdefinitions-hostgroup Icinga's documentation]


== objects/groups ==
=== Introduction ===


Firstly, we've got objects named "host groups" which are simple groups with several servers, for instance all Ubuntu servers belong to the 'ubuntu-servers' group and all servers which have a PostgreSQL service running belong to the 'postgresql-servers' group. One host/server can belong to many groups. All these groups are defined in '''/etc/icinga/objects/groups/hostgroups_icinga.cfg''' the members of these groups are also defined here.
In Icinga we can group hosts, for instance "Ubuntu Servers" and "PostgreSQL servers" and so on. All of our servers are members of the 'all' host group, so instead of defining services checks for memory & harddisk usage (to name a few) per server, we simply define that check in the 'all' host group and effectively, all of our servers are now being monitored for memory & harddisk usage (and a lot more, check '''/etc/icinga/objects/groups/all.cfg''').


Let's take the ubuntu-servers & postgresql-servers (respectively) as examples:
One server can be a member of many host groups and we've got quite a few. We've got the "ubuntu-servers" group, the "postgresql-servers" group and a few more.


<code>/etc/icinga/objects/groups/ $ '''cat hostgroups_icinga.cfg'''</code> [note: only copying 'ubuntu-servers' and 'postgresql-servers' definitions out of file]</code>
=== Define a hostgroup ===


'''ubuntu-servers'''
We define host groups in '''/etc/icinga/objects/groups/hostgroup_icinga.cfg''', these look a lot like (comments in line):


<pre>
<pre>
; Start with the define statement.
define hostgroup {
        hostgroup_name  ubuntu-servers ; Let's give the group a name.
alias          Ubuntu Servers ; Let's give it a "more" friendly name.
members        boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22 ; It's members.
        }
define hostgroup {
define hostgroup {
         hostgroup_name  ubuntu-servers
         hostgroup_name  postgresql-servers ; Let's give the group a name.
alias          Ubuntu Servers
                alias          PostgreSQL servers ; Let's give it a "more" friendly name.
members        boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22
                members        carbon.li694-22, boron.li694-22 ; It's members.
         }
         }
</pre>
</pre>


'''postgresql-servers'''
As you can no doubt see, the definitions are relatively straight forward. Let's say, we install PostgreSQL on our server by the name of "helium"; then we'd simply add helium.li694-22 as a member of the postgresql-servers group and voila, now helium's PostgreSQL is also monitored. (note: monitoring PostgreSQL in itself involves a few more steps which are omitted to keep the explanation simple).
 
=== Adding a new host group ===
 
Adding a new host group is simple, you copy one of the "hostgroup" definitions to the end of '''/etc/icinga/objects/groups/hostgroup_icinga.cfg''' and change the group's name, alias & you define which servers are a member. So let's say, you want to monitor webservers, you would add the following definition (note, we're already monitoring those, it's just an example):


<pre>
<pre>
# A list of our web servers
define hostgroup {
define hostgroup {
         hostgroup_name  postgresql-servers
         hostgroup_name  http-servers
                 alias          PostgreSQL servers
                 alias          HTTP servers
                 members        carbon.li694-22, boron.li694-22
                 members        hydrogen.li694-22, carbon.li694-22, boron.li694-22, beryllium.li694-22
         }
         }
</pre>
</pre>
Now, this alone, won't work, currently we've only defined the host group but we haven't defined any services, let's do that in the next chapter.
=== Modifying a host group ===
Simply modify the appropriate definition in '''/etc/icinga/objects/groups/hostgroup_icinga.cfg'''
=== Removing a host group ===
Simply remove the definition from '''/etc/icinga/objects/groups/hostgroup_icinga.cfg'''
== Monitoring Services ==
=== Introduction ===
=== Adding a service ===
=== Modifying a service ===
=== Removing a service ===
=== junk ===


As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.
As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.
Line 139: Line 183:


Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.
Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.
{{#lst:SystemAdministration/Icinga|testandreload}}
[[Category:System administration]]

Latest revision as of 22:26, 29 May 2014

Sentinel: Icinga / Server & Services Monitoring

Important Directories

Directory Description
/etc/icinga/objects/groups (service definitions for) Host Groups
/etc/icinga/objects/servers Server Definitions
/etc/icinga/objects/templates Various templates docs here
/etc/icinga/objects/contacts Contact & time period config docs here
/etc/nagios-plugins/config/ Plugin configs; be sure to checkout 'remote.cfg'
/usr/lib/nagios/plugins/ Plugins

Useful links

Icinga documentation

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload


Access

Our Icinga installation is named 'Sentinel' which means "a soldier or guard whose job is to stand and keep watch", you can find Sentinel here and login with your Keberos username and password.

Kerberized SSH connections

We use AutoSSH to allow Icinga to connect to other nodes to execute commands and get services information back.

Internally, we can use Kerberos to jump from one host to another our Icinga installation requires HTTP Kerberos auth (via https) but we've also set up AutoSSH which allows our Icinga instance to connect to other nodes passwordless & ssh key-less to fetch information.

more information, show how we've setup the connections, etc

icinga.cfg

The file /etc/icinga/icinga.cfg is heavily commented. I suggest you simply look it over.

Servers

Icinga's "host" documentation

Adding a server

Each server has a file in /etc/icinga/objects/servers/ which looks something like:

define host{
        use                     generic-host            ; Name of host template to use
        host_name               carbon.li694-22         ; The server hostname (ensure it's in our DNS).
        alias                   carbon                  ; A shortname
        address                 carbon.li694-22         ; The server's address.
}

Creating a new one (for instance, 'example.li694-22') should be straight forward, you simply copy carbon.li694-22 to example.li64-22:

/etc/icinga/objects/servers/ $ cp carbon.cfg example.cfg

Once that's done, you can either edit the file manually with the editor of your choice or use sed:

/etc/icinga/objects/servers/ $ sed -i 's/carbon/example/g' example.cfg

Modifying a server

Simply edit the server object in /etc/icinga/objects/servers/

Removing a server

If you want to remove a server, remove the config and check /etc/icinga/objects/groups/hostgroups_icinga.cfg and remove any references to the server.

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload


Host groups

Icinga's documentation

Introduction

In Icinga we can group hosts, for instance "Ubuntu Servers" and "PostgreSQL servers" and so on. All of our servers are members of the 'all' host group, so instead of defining services checks for memory & harddisk usage (to name a few) per server, we simply define that check in the 'all' host group and effectively, all of our servers are now being monitored for memory & harddisk usage (and a lot more, check /etc/icinga/objects/groups/all.cfg).

One server can be a member of many host groups and we've got quite a few. We've got the "ubuntu-servers" group, the "postgresql-servers" group and a few more.

Define a hostgroup

We define host groups in /etc/icinga/objects/groups/hostgroup_icinga.cfg, these look a lot like (comments in line):

; Start with the define statement.
define hostgroup {
        hostgroup_name  ubuntu-servers ; Let's give the group a name.
		alias           Ubuntu Servers ; Let's give it a "more" friendly name.
		members         boron.li694-22, helium.li694-22, carbon.li694-22, lithium.li694-22, nitrogen.li694-22, oxygen.li694-22, hydrogen.li694-22 ; It's members.
        }

define hostgroup {
        hostgroup_name  postgresql-servers ; Let's give the group a name.
                alias           PostgreSQL servers ; Let's give it a "more" friendly name.
                members         carbon.li694-22, boron.li694-22 ; It's members.
        }

As you can no doubt see, the definitions are relatively straight forward. Let's say, we install PostgreSQL on our server by the name of "helium"; then we'd simply add helium.li694-22 as a member of the postgresql-servers group and voila, now helium's PostgreSQL is also monitored. (note: monitoring PostgreSQL in itself involves a few more steps which are omitted to keep the explanation simple).

Adding a new host group

Adding a new host group is simple, you copy one of the "hostgroup" definitions to the end of /etc/icinga/objects/groups/hostgroup_icinga.cfg and change the group's name, alias & you define which servers are a member. So let's say, you want to monitor webservers, you would add the following definition (note, we're already monitoring those, it's just an example):

# A list of our web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         hydrogen.li694-22, carbon.li694-22, boron.li694-22, beryllium.li694-22
        }

Now, this alone, won't work, currently we've only defined the host group but we haven't defined any services, let's do that in the next chapter.

Modifying a host group

Simply modify the appropriate definition in /etc/icinga/objects/groups/hostgroup_icinga.cfg

Removing a host group

Simply remove the definition from /etc/icinga/objects/groups/hostgroup_icinga.cfg

Monitoring Services

Introduction

Adding a service

Modifying a service

Removing a service

junk

As you can see, we give each group a name, an alias (friendly name, mostly) & we define it's members. Pretty straight forward right? Let's move on to the fun part.

Now instead of defining a service per server, we simply define a service per hostgroup, meaning; when we add an Ubuntu server we simply add it to the 'ubuntu-servers' host group and it'll be automatically monitored for updates, and such.

The service definitions for hostgroups are in the same directory, let's take a look at the service definitions for the ubuntu-servers group:

/etc/icinga/objects/groups/ $ cat ubuntu.cfg

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             APT Status              ; Service check name.
        check_command                   remote_one!check_apt    ; This is the command that's being executed.
        }

Let's say, all Ubuntu servers also run a special daemon (let's called it "speciald" - I haven't got a lot of inspiration right now), we would add the following (make sure the command definition/plugin exists) to 'ubuntu.cfg':

define service{
	hostgroup_name			ubuntu-servers          ; Host group
        use                             generic-service         ; Name of service template to use
        service_description             Speciald Status         ; Service check name.
        check_command                   remote_one!check_speciald ; This is the command that's being executed.
        }

and reload Icinga (after testing the configuration, first) and now, 'speciald' is being monitored on all Ubuntu servers.

Note, we also have 'all.cfg' which powers the "All Servers Group" - in which, you guessed it, checks for every server are defined, e.g. ssh, disk space, etc.

Testing Icinga's config and reloading:

When you've made modifications to Icinga's config (you really should first back up the files you're modifying), you can run:

# /usr/sbin/icinga -v /etc/icinga/icinga.cfg

If this command returns "Total Warnings: 0" & "Total Errors: 0" you can then:

# service icinga reload