[dev.icinga.com #5444] allow external tools to check health of icinga 2 {instance,cluster}

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/5444

Created by tgelf on 2014-01-13 15:03:57 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2014-02-17 18:03:03 +00:00)
Target Version: 0.0.8
Last Update: 2014-02-17 18:03:03 +00:00 (in Redmine)

---

Provide a standarized way allowing external tools to check Icinga2 health.

Cheers,
Thomas

Changesets

2014-02-13 15:34:15 +00:00 by (unknown) f5d40ba

Cluster: Periodically dump status json.

Refs #5444

2014-02-13 18:23:38 +00:00 by (unknown) 66aa874

Refactor ClusterCheckTask based on cluster status.

Refs #5444

2014-02-13 18:24:09 +00:00 by (unknown) fc56782

Merge branch 'feature/cluster-status-json-5444' into next

Fixes #5444

2014-02-17 17:51:16 +00:00 by (unknown) d3cdbb5

Implement IcingaStatusWriter object dumping local and feature stats.

Refactored the cluster listener health check too.

Refs #5622
Refs #5444

---

Relations:

• relates #5440
• relates #5438
• relates #5444
• relates #5622
• relates #5654

[icinga-migration]

Updated by elippmann on 2014-01-13 15:09:06 +00:00

Required information:

• endpoint latencies
• loaded modules
• cluster status
• check statistics

[icinga-migration]

Updated by mfriedrich on 2014-01-14 08:13:49 +00:00

• Target Version set to 0.0.7

Per definition as periodically dumped file in json format.

[icinga-migration]

Updated by mfriedrich on 2014-01-15 09:40:25 +00:00

• Subject changed from check_icinga2_health to allow external tools to check health of icinga 2 {instance,cluster}
• Category set to libicinga

[icinga-migration]

Updated by mfriedrich on 2014-01-23 16:19:01 +00:00

• Priority changed from Normal to High

[icinga-migration]

Updated by mfriedrich on 2014-01-27 08:35:49 +00:00

• Target Version changed from 0.0.7 to 0.0.8

[icinga-migration]

Updated by mfriedrich on 2014-02-06 12:27:06 +00:00

• Priority changed from High to Normal

[icinga-migration]

Updated by mfriedrich on 2014-02-06 13:08:29 +00:00

• Priority changed from Normal to High

[icinga-migration]

Updated by mfriedrich on 2014-02-06 13:13:03 +00:00

• Assigned to set to mfriedrich

[icinga-migration]

Updated by mfriedrich on 2014-02-07 15:02:22 +00:00

Requires central information base providing instance check stats in #5617
Furthermore all features must register a health check method in order to be seen in #5622

[icinga-migration]

Updated by mfriedrich on 2014-02-13 15:24:04 +00:00

Example:

object ClusterListener "icinga2-cluster" {

  // certificate for icinga2a
  ca_path = "/etc/icinga2/icinga2-cluster/pki/ca.crt",
  cert_path = "/etc/icinga2/icinga2-cluster/pki/icinga2a.crt",
  key_path = "/etc/icinga2/icinga2-cluster/pki/icinga2a.key",

  // cluster port for icinga2a
  bind_port = 8888,

  // remote instances array
  peers = [ "icinga2b" ],

  status_path = ( IcingaLocalStateDir + "/cache/icinga2/cluster/icinga2a.json")
}

{
        "active_checks":        6.64114e-02,
        "active_checks_15min":  3,
        "active_checks_1min":   3,
        "active_checks_5min":   3,
        "avg_execution_time":   8.949076e-01,
        "avg_latency":  1.4951e-03,
        "conn_endpoints":       ["icinga2b"],
        "feature_CheckResultReader":    0,
        "feature_CheckerComponent":     1,
        "feature_CompatLogger": 1,
        "feature_ExternalCommandListener":      1,
        "feature_FileLogger":   1,
        "feature_GraphiteWriter":       0,
        "feature_IdoMysqlConnection":   1,
        "feature_IdoPgsqlConnection":   0,
        "feature_LivestatusListener":   1,
        "feature_NotificationComponent":        1,
        "feature_PerfdataWriter":       1,
        "feature_StatusDataWriter":     1,
        "feature_SyslogLogger": 1,
        "identity":     "icinga2a",
        "max_execution_time":   2.8632e-03,
        "max_latency":  2.8632e-03,
        "min_execution_time":   4.129e-04,
        "min_latency":  4.129e-04,
        "node": "icinga2a",
        "not_conn_endpoints":   [],
        "num_conn_endpoints":   1,
        "num_endpoints":        2,
        "num_hosts_acknowledged":       0,
        "num_hosts_down":       0,
        "num_hosts_flapping":   0,
        "num_hosts_in_downtime":        0,
        "num_hosts_unreachable":        0,
        "num_hosts_up": 3,
        "num_not_conn_endpoints":       0,
        "num_services_acknowledged":    0,
        "num_services_critical":        2,
        "num_services_flapping":        0,
        "num_services_in_downtime":     0,
        "num_services_ok":      14,
        "num_services_pending": 0,
        "num_services_unknown": 0,
        "num_services_unreachable":     0,
        "num_services_warning": 2,
        "passive_checks":       0,
        "passive_checks_15min": 0,
        "passive_checks_1min":  0,
        "passive_checks_5min":  0
}

[icinga-migration]

Updated by mfriedrich on 2014-02-13 18:25:26 +00:00

A more in-depth cluster remote features check should be done seperately. That one just pulls the local clusterlistener node and writes that to a json blob.

[icinga-migration]

Updated by Anonymous on 2014-02-13 18:25:31 +00:00

• Status changed from New to Resolved
• Done % changed from 0 to 100

Applied in changeset fc56782.

[icinga-migration]

Updated by mfriedrich on 2014-02-17 08:55:33 +00:00

• Status changed from Resolved to Assigned
• Done % changed from 100 to 80

Move status file writing into a seperate configuration object. By default the ClusterListener shall not write that file.

[icinga-migration]

Updated by ccesario on 2014-02-17 15:53:32 +00:00

Remember add user permission in IcingaLocalStateDir + "/cache/icinga2/cluster/ directory.

Without correct permission the status files isn't created.

[icinga-migration]

Updated by mfriedrich on 2014-02-17 15:59:33 +00:00

Thanks but I will move the location away from the cluster dir into the main /var/cache/icinga2 as it also reflects the status of the current local instance.

[icinga-migration]

Updated by mfriedrich on 2014-02-17 18:03:03 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 80 to 100

All features including the clusterlistener will register their values through a to-be-completed stats registry with hooks in #5622

That way we may add/remove feature stats dynamically.

    library "icinga"

    object IcingaStatusWriter "status" {
      status_path = (IcingaLocalStateDir + "/cache/icinga2/status.json),
      update_interval = 15s
    }

{
        "active_checks":        0,
        "active_checks_15min":  0,
        "active_checks_1min":   0,
        "active_checks_5min":   0,
        "avg_execution_time":   8.048826e-01,
        "avg_latency":  1.4144e-03,
        "feature_perfdata":     {
        },
        "feature_status":       {
                "checkercomponent_":    1,
                "checkresultreader_":   1,
                "clusterlistener_":     1,
                "clusterlistenericinga2-cluster":       {
                        "conn_endpoints":       [],
                        "identity":     "icinga2a",
                        "node": "icinga2a",
                        "not_conn_endpoints":   ["icinga2b"],
                        "num_conn_endpoints":   0,
                        "num_endpoints":        2,
                        "num_not_conn_endpoints":       1
                },
                "compatlogger_":        1,
                "externalcommandlisterner_":    1,
                "filelogger_":  1,
                "graphite_writer_":     1,
                "icingaapplication_":   1,
                "icingastatuswriter_":  1,
                "ido_mysql_version_req":        "1.11.0",
                "livestatus_connections":       0,
                "notification_":        1,
                "perfdatawriter_":      1,
                "statusdatawriter_":    1,
                "sysloglogger_":        1
        },
        "max_execution_time":   2.6312e-03,
        "max_latency":  2.6312e-03,
        "min_execution_time":   2.441e-04,
        "min_latency":  2.441e-04,
        "num_hosts_acknowledged":       0,
        "num_hosts_down":       0,
        "num_hosts_flapping":   0,
        "num_hosts_in_downtime":        0,
        "num_hosts_unreachable":        0,
        "num_hosts_up": 2,
        "num_services_acknowledged":    0,
        "num_services_critical":        1,
        "num_services_flapping":        0,
        "num_services_in_downtime":     0,
        "num_services_ok":      8,
        "num_services_pending": 0,
        "num_services_unknown": 0,
        "num_services_unreachable":     0,
        "num_services_warning": 1,
        "passive_checks":       0,
        "passive_checks_15min": 0,
        "passive_checks_1min":  0,
        "passive_checks_5min":  0
}