[dev.icinga.com #9549] Generated certificates cannot be verified w/ openssl 0.9.8j on SLES 11

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/9549

Created by twidhalm on 2015-07-02 20:49:37 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-07-06 11:45:03 +00:00)
Target Version: 2.3.6
Last Update: 2015-07-06 14:36:48 +00:00 (in Redmine)

Icinga Version: 2.3.5
Backport?: Already backported
Include in Changelog: 1

---

Hi,

On SLES 11 SP3 openssl is available in version 0.9.8j-0.70.1 . It seems this version can not deal with the default hashing algorithm of the Icinga 2 CA SHA256 (which was introduced in Issue #7434 ) .

When building a cluster connection, there is one try reported in the logs that says, the connection is unauthenticated and then no more log entries according this connection. (Issue #9489 will fix the missing logevents) When creating a CA to generate all Certificates for all nodes connections are made but the logs of the Agent / Satellite (not on the master) still say the connection is "unauthenticated" although it retries to create a connection regularly. When changing the direction in which the connection is made (by changing the endpoint configuration in zones.conf) you still see the unauthenticated messages it only changes on which Node the reconnection is logged.

When adding the "cluster" and "cluster-zone" check on both nodes, the master says, the satellite zone and endpoint are connected (verified by seeing an alert, when the satellite ist shut down) and the satellite says both zone and endpoint of the master are disconnected at the same time. No configuration (not even global zones) are copied to the satellite.

I've sent some logs and more information directly to Gunnar and I know some people who had the same problem with the same combination Icinga 2.3.5 on SLES 11 SP3, so if you need more information, tell me what you need and I will try to provide it.

Cheers,
Thomas

Attachments

• combined_logs twidhalm - 2015-07-02 22:21:22 +00:00 - Excerpts of various logs on the hosts

Changesets

2015-07-06 09:43:26 +00:00 by (unknown) 6810737

Require libopenssl1-devel on sles11sp3

refs #9549

2015-07-06 11:37:39 +00:00 by (unknown) 3c1aec4

Require openssl1 on sles11sp3 from Security Module

There is a severe problem with SSL certificate verification
described in #9549 which renders client communication
unusable. The OpenSSL 0.9.8j version in SLES11 is buggy
and does not allow to verify clients on the master,
nor does a manual openssl verification work.

We'll therefore switch to the openssl1 package provided
by the SLES 11 Security Module and link against their
working API in order to resolve the problem in clusters
and client setups.

fixes #9549

2015-07-06 11:47:48 +00:00 by (unknown) eac73e1

Require openssl1 on sles11sp3 from Security Module

There is a severe problem with SSL certificate verification
described in #9549 which renders client communication
unusable. The OpenSSL 0.9.8j version in SLES11 is buggy
and does not allow to verify clients on the master,
nor does a manual openssl verification work.

We'll therefore switch to the openssl1 package provided
by the SLES 11 Security Module and link against their
working API in order to resolve the problem in clusters
and client setups.

fixes #9549

2015-07-07 07:20:12 +00:00 by (unknown) ecd9876

Fix typo in INSTALL.md

refs #9549

2015-07-07 07:20:32 +00:00 by (unknown) 86b5e20

Fix typo in INSTALL.md

refs #9549

---

Relations:

• relates #8844
• relates #7434

[icinga-migration]

Updated by mfriedrich on 2015-07-02 21:25:30 +00:00

Pleas add logs, configuration and cli output directly to this issue so that others may also look into this issue, not only Gunnar. Thanks.

[icinga-migration]

Updated by twidhalm on 2015-07-02 22:21:35 +00:00

• File added combined_logs

I added excerpts of debug.log and icinga2.log of both hosts here (anonymised). Most of the events just repeat themselves, so I uploaded just a part of them.

[icinga-migration]

Updated by mfrosch on 2015-07-03 07:29:02 +00:00

The actual problem seems to have been caused by #8844 - which fixed another Issue with OpenSSL on SuSE.

Whats really bothering me, what OpenSSL are we linking against, when EVP_sha256() seems to be resolved in the ABI.

What version of OpenSSL is the user in #8844 using, and what version here...

[icinga-migration]

Updated by mfrosch on 2015-07-03 07:29:11 +00:00

• Relates set to 8844

[icinga-migration]

Updated by mfriedrich on 2015-07-03 07:31:34 +00:00

• Relates set to 7434

[icinga-migration]

Updated by mfriedrich on 2015-07-03 07:32:15 +00:00

• Target Version set to Backlog

[icinga-migration]

Updated by mfriedrich on 2015-07-03 08:10:43 +00:00

• Subject changed from Automatic CA can not work with older openssl versions to CA generated by node wizard cannot be verified w/ openssl 0.9.8

[icinga-migration]

Updated by mfriedrich on 2015-07-03 15:29:32 +00:00

• Status changed from New to Feedback
• Assigned to set to twidhalm

Analysis

CreateCert() with EVP_sha256 fixed in #8844 is called in CreateCertIcingaCA() which gets invoked to request a client certificate in RequestCertificateHandler().

EVP_sha256 was introduced with 0.9.8 and is not available in 0.9.7.
https://kea.isc.org/ticket/3482

So we build and link against a valid source, but there is a problem with openssl 0.9.8j on SLES11SP3 regarding the verification preventing preverify_ok being set to 1.

SHA-2 support

https://support.globalsign.com/customer/portal/articles/1499561-sha-256-compatibility

Support for SHA-2 was introduced in OpenSSL 0.9.8, but is not enabled by default with SSL_library_init(). In 0.9.8, SHA-2 hash functions must be called specifically or by using OpenSSL_add_all_algorithms() which may not be desired. OpenSSL 0.9.8o enables the SHA-2 hash algorithms in the default configuration.

Calling those hash functions specifically would somehow explain the segfault encountered with #8844 having that set to NULL previously without initialization of the algorithms.

http://linux.die.net/man/3/openssl\_add\_ssl\_algorithms

OpenSSL 0.9.8o and 1.0.0a and later added SHA2 algorithms to SSL_library_init(). Applications which need to use SHA2 in earlier versions of OpenSSL should call OpenSSL_add_all_algorithms() as well.

A slightly different approach is to add the cipher

http://lists.freebsd.org/pipermail/freebsd-bugs/2009-August/036249.html

        SSL_library_init();
+#ifndef OPENSSL_NO_SHA256
+       EVP_add_digest(EVP_sha256());
+#endif /* OPENSSL_NO_SHA256 */

Although this might require a version check as well.
http://lists.shmoo.com/pipermail/hostap/2011-April/023002.html

                 SSL_library_init();
-#ifndef OPENSSL_NO_SHA256
+#if (OPENSSL_VERSION_NUMBER >= 0x0090800fL) && !defined(OPENSSL_NO_SHA256)
                 EVP_add_digest(EVP_sha256());
  #endif /* OPENSSL_NO_SHA256 */

Update SLES11 packages

SLES provides openssl1 as package inside its security repository. Changing the package requirements and rebuilding the packages linking to openssl 1.0+ will solve the problem.
https://www.suse.com/communities/conversations/introducing-the-suse-linux-enterprise-11-security-module/

In terms of the ABI, I don't see an issue here.
http://upstream.rosalinux.ru/compat\_reports/openssl/0.9.8\_to\_1.0.1e/compat\_report.html

More material

More on that topic on the internet:
https://www.digicert.com/sha-2-compatibility.htm
http://marc.info/?l=openssl-users&m=135355590501495
http://comments.gmane.org/gmane.linux.drivers.hostap/21107

https://projects.puppetlabs.com/issues/21029#note-3

Conclusion

I wouldn't go for a simple patch solution only, but rather enforce users to use a more recent openssl library version where we may link against for sles11 sp3 (we're doing that already for boost153 too).

In any attempt, this requires proper tests and feedback on that ancient platform.

Please test the attached patch on your sles11 platform.

diff --git a/lib/base/tlsutility.cpp b/lib/base/tlsutility.cpp
index 5090d4d..d4c334a 100644
--- a/lib/base/tlsutility.cpp
+++ b/lib/base/tlsutility.cpp
@@ -57,6 +57,11 @@ void InitializeOpenSSL(void)
                return;

        SSL_library_init();
+
+#if (OPENSSL_VERSION_NUMBER >= 0x0090800fL) && !defined(OPENSSL_NO_SHA256)
+       EVP_add_digest(EVP_sha256());
+#endif /* OPENSSL_NO_SHA256 */
+
        SSL_load_error_strings();

        SSL_COMP_get_compression_methods();

[icinga-migration]

Updated by mfrosch on 2015-07-03 16:26:15 +00:00

Since SHA1 should really be deprecated, and is declared to by many vendors, we should keep sha256.

The initialization issue is rather weird, but well, OpenSSL API...

Old platforms like Debian squeeze, RH 5 and SLES 11 SP3(!) seems to support that.

Even on weird platforms the user should make sure to have a secure OpenSSL version...

[icinga-migration]

Updated by mfriedrich on 2015-07-05 17:47:41 +00:00

Applying the patch to a freshly installed sles11sp3 vagrant box environment does not resolve the issue, unfortunately.
I've built a small build environment for that - https://github.com/dnsmichi/vagrant-boxes/tree/master/sles11sp3\_cluster

The generated client certificate signed with the master's ca is still invalid.

# openssl verify -verbose -CAfile ca.crt icinga2b.crt
icinga2b.crt: /CN=Icinga CA
error 7 at 1 depth lookup:certificate signature failure
31384:error:0408D068:rsa routines:FIPS_RSA_VERIFY:bad signature:fips_rsa_sign.c:391:
31384:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:178:

Maybe related for leading zeros and encoding, but our serial is just an integer.

http://openssl.6102.n7.nabble.com/Re-Verify-X-509-certificate-openssl-verify-returns-bad-signature-td38193.html

The problem you are encountering is partly caused by the way OpenSSL 
handles integers whose DER encoded value starts with one or more zeros : 
in this case, OpenSSL removes the leading zero when creating the 
corresponding ASN1_INTEGER structure thus leading to the fact that 
computed DER of this structure and the original one will be different!! 

Available Ciphers

It seems that sha-2 is not available regardless what you specify on the command line.

icinga2b:/etc/icinga2/pki # openssl ciphers -v
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-CAMELLIA256-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(256) Mac=SHA1
DHE-DSS-CAMELLIA256-SHA SSLv3 Kx=DH       Au=DSS  Enc=Camellia(256) Mac=SHA1
CAMELLIA256-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(256) Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
DES-CBC3-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=MD5 
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-CAMELLIA128-SHA SSLv3 Kx=DH       Au=RSA  Enc=Camellia(128) Mac=SHA1
DHE-DSS-CAMELLIA128-SHA SSLv3 Kx=DH       Au=DSS  Enc=Camellia(128) Mac=SHA1
CAMELLIA128-SHA         SSLv3 Kx=RSA      Au=RSA  Enc=Camellia(128) Mac=SHA1
RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5 
RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1
RC4-MD5                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5 
EDH-RSA-DES-CBC-SHA     SSLv3 Kx=DH       Au=RSA  Enc=DES(56)   Mac=SHA1
EDH-DSS-DES-CBC-SHA     SSLv3 Kx=DH       Au=DSS  Enc=DES(56)   Mac=SHA1
DES-CBC-SHA             SSLv3 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=SHA1
DES-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=MD5 
EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=DSS  Enc=DES(40)   Mac=SHA1 export
EXP-DES-CBC-SHA         SSLv3 Kx=RSA(512) Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-RC2-CBC-MD5         SSLv3 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv3 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export

Test with custom openssl1

Building the icinga2 package against openssl1 from an external repository seems to make it work. Although that proof-of-concept needs to be redone with the sles11 openssl1 package being officially supported.

Only the master setup:

# icinga2 node wizard

icinga2a:/etc/icinga2/pki # openssl1 verify -verbose -CAfile ca.crt icinga2a.crt 
icinga2a.crt: OK

icinga2a:/etc/icinga2/pki # openssl verify -verbose -CAfile ca.crt icinga2a.crt 
icinga2a.crt: /CN=Icinga CA
error 7 at 1 depth lookup:certificate signature failure
549:error:0408D068:rsa routines:FIPS_RSA_VERIFY:bad signature:fips_rsa_sign.c:391:
549:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:178:

Going further to the client

icinga2b:/etc/icinga2/pki # openssl verify -verbose -CAfile ca.crt icinga2b.crt
icinga2b.crt: /CN=Icinga CA
error 7 at 1 depth lookup:certificate signature failure
4139:error:0408D068:rsa routines:FIPS_RSA_VERIFY:bad signature:fips_rsa_sign.c:391:
4139:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:178:

icinga2b:/etc/icinga2/pki # openssl1 verify -verbose -CAfile ca.crt icinga2b.crt
icinga2b.crt: OK

Master-Client communication exchanging check results also works (after invoking node update-config on the master and restart the service).

Conclusion

The possible safest solution is to enable the sles updates with the security repository and change the spec file to require openssl1-devel. There's a pending request for that already for gcc47 in #9560.
For some reason, the evaluation code request form does not work though - https://www.suse.com/products/server/download/amd64-v11.html - we will need to add sles11 subscriptions then anyways.

[icinga-migration]

Updated by mfriedrich on 2015-07-06 11:43:39 +00:00

• Subject changed from CA generated by node wizard cannot be verified w/ openssl 0.9.8 to Generated certificates cannot be verified w/ openssl 0.9.8 on SLES 11
• Status changed from Feedback to Assigned
• Assigned to changed from twidhalm to mfriedrich
• Priority changed from Normal to High
• Target Version changed from Backlog to 2.4.0
• Estimated Hours set to 8

Solution

Link against openssl1 packages from Security Module.

[icinga-migration]

Updated by mfriedrich on 2015-07-06 11:45:00 +00:00

• Subject changed from Generated certificates cannot be verified w/ openssl 0.9.8 on SLES 11 to Generated certificates cannot be verified w/ openssl 0.9.8j on SLES 11

[icinga-migration]

Updated by Anonymous on 2015-07-06 11:45:03 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset 3c1aec4.

[icinga-migration]

Updated by mfriedrich on 2015-07-06 11:51:32 +00:00

• Target Version changed from 2.4.0 to 2.3.6
• Backport? changed from TBD to Yes

Requires Changelog entry for 2.3.6

[icinga-migration]

Updated by mfriedrich on 2015-07-06 12:29:03 +00:00

Upgrading

The problem is the certificate generation (master, or CSR auto-signing). All certificates generated with 0.9.8j are invalid and cannot be verified - that needs to be re-done with an uptodate icinga2 version linking against openssl1

Old:

# ldd `which icinga2` | egrep "ssl|crypto"
    libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007f82089c5000)
    libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007f82058c6000)

new

# ldd `which icinga2` | egrep "ssl|crypto"
    libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007f11d27cd000)
    libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007f11cf1f6000)

[icinga-migration]

Updated by mfriedrich on 2015-07-06 14:36:48 +00:00

Tests

icinga2a:/etc # icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.



Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specifiy the common name (CN) [icinga2a]: 
information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
information/cli: Initializing serial file in '/var/lib/icinga2/ca/serial.txt'.
information/cli: Generating new CSR in '/etc/icinga2/pki/icinga2a.csr'.
information/base: Writing private key to '/etc/icinga2/pki/icinga2a.key'.
information/base: Writing certificate signing request to '/etc/icinga2/pki/icinga2a.csr'.
information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/icinga2a.crt'.
information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
warning/cli: Backup file '/etc/icinga2/zones.conf.orig' already exists. Skipping backup.
Please specify the API bind host/port (optional):
Bind Host []: 
Bind Port []: 
information/cli: Enabling the APIlistener feature.
warning/cli: Feature 'api' already enabled.
warning/cli: Backup file '/etc/icinga2/features-available/api.conf.orig' already exists. Skipping backup.
information/cli: Updating constants.conf.
warning/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

icinga2a:~ # service icinga2 restart
Checking configuration: Done
Stopping Icinga 2: The PID file '/var/run/icinga2/icinga2.pid' does not exist.
Starting Icinga 2: Done
icinga2a:~ # icinga2 pki ticket --cn 'icinga2b'
730fdaa2a5bf7af6bbff641906c9bdae6ab2432d

icinga2b:~ # icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.



Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: 
Starting the Node setup routine...
Please specifiy the common name (CN) [icinga2b]: 
Please specifiy the local zone name [icinga2b]: 
Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga2a
Do you want to establish a connection to the master from this node? [Y/n]: 
Please fill out the master connection information:
Master endpoint host (Your master's IP address or FQDN): 192.168.56.210
Master endpoint port [5665]: 
Add more master endpoints? [y/N]: 
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [192.168.56.210]: 
Port [5665]: 
warning/cli: Backup file '/etc/icinga2/pki/icinga2b.key.orig' already exists. Skipping backup.
warning/cli: Backup file '/etc/icinga2/pki/icinga2b.crt.orig' already exists. Skipping backup.
information/base: Writing private key to '/etc/icinga2/pki/icinga2b.key'.
information/base: Writing X509 certificate to '/etc/icinga2/pki/icinga2b.crt'.
information/cli: Generating self-signed certifiate:
information/cli: Fetching public certificate from master (192.168.56.210, 5665):

information/cli: Writing trusted certificate to file '/etc/icinga2/pki/trusted-master.crt'.
information/cli: Stored trusted master certificate in '/etc/icinga2/pki/trusted-master.crt'.

Please specify the request ticket generated on your Icinga 2 master.
 (Hint: # icinga2 pki ticket --cn 'icinga2b'): 730fdaa2a5bf7af6bbff641906c9bdae6ab2432d
information/cli: Processing self-signed certificate request. Ticket '730fdaa2a5bf7af6bbff641906c9bdae6ab2432d'.

information/cli: Created backup file '/etc/icinga2/pki/ca.crt.orig'.
warning/cli: Backup file '/etc/icinga2/pki/icinga2b.crt.orig' already exists. Skipping backup.
information/cli: Writing signed certificate to file '/etc/icinga2/pki/icinga2b.crt'.
information/cli: Writing CA certificate to file '/etc/icinga2/pki/ca.crt'.
Please specify the API bind host/port (optional):
Bind Host []: 
Bind Port []: 
Accept config from master? [y/N]: y
Accept commands from master? [y/N]: y
information/cli: Disabling the Notification feature.
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Enabling the Apilistener feature.
warning/cli: Feature 'api' already enabled.
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Generating local zones.conf.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

# service icinga2 restart

icinga2a:~ # icinga2 feature enable debuglog
Enabling feature debuglog. Make sure to restart Icinga 2 for these changes to take effect.
icinga2a:~ # service icinga2 restart
Checking configuration: Done
Stopping Icinga 2: .Done
Starting Icinga 2: Done
icinga2a:~ # tail -f /var/log/icinga2/debug.log 
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::SetNextCheck' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::SetNextCheck' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::CheckResult' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::CheckResult' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::SetNextCheck' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::SetNextCheck' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::CheckResult' message from 'icinga2b'
[2015-07-06 16:33:48 +0200] notice/ApiClient: Received 'event::CheckResult' message from 'icinga2b'