[dev.icinga.com #10688] Endpoint not created

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10688

Created by bert2002 on 2015-11-20 10:00:36 +00:00

Assignee: (none)
Status: Rejected (closed on 2016-03-18 16:11:17 +00:00)
Target Version: (none)
Last Update: 2016-03-18 16:11:17 +00:00 (in Redmine)

Icinga Version: 2.3.10
Backport?: Not yet backported
Include in Changelog: 1

---

Hi,

we have Icinga 2.3.10 running and get clients use the api to connect and push results. We just created a new environment from a staging environment and no client in the new environment gets any endpoint/zone/hosts on the icinga server.

The only difference between the two environments are the names:

stage: stage-app-stats-1.srv.company.com
prod: app-stats-1.srv.company.com

The staging clients have no problem and everyone is happy. The production clients cant connect and get refused from the icinga server with this error (debuglog):

[2015-11-20 09:54:14 +0100] notice/ApiClient: Received 'icinga::Hello' message from 'app-stats-1.srv.company.com'
[2015-11-20 09:54:14 +0100] notice/ApiClient: Received 'event::SetNextCheck' message from 'app-stats-1.srv.company.com'
[2015-11-20 09:54:14 +0100] notice/ApiEvents: Discarding 'next check changed' message from 'app-stats-1.srv.company.com': Invalid endpoint origin (client not allowed).

Apparently this is a problem with the certificates, but according to the troubleshooting guide they are okay:

root@app-stats-1:~# openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/app-stats-1.srv.company.com.crt
/etc/icinga2/pki/app-stats-1.srv.company.com.crt: OK
root@app-stats-1:~#

root@app-stats-1:~# md5sum /etc/icinga2/pki/ca.crt 
082c8ecd93e7c44e3825e2ebe08eba06 /etc/icinga2/pki/ca.crt
root@app-stats-1:~#

root@icinga ~# md5sum /etc/icinga2/pki/ca.crt 
082c8ecd93e7c44e3825e2ebe08eba06 /etc/icinga2/pki/ca.crt

When I try to create a new endpoint with the name of the production environment i get told that the endpoint already exist:

root@icinga repository.d# icinga2 repository endpoint add name=app-stats-1.srv.company.com 
information/Utility: Loading library 'libicinga.so'
warning/cli: Endpoint 'app-stats-1.srv.company.com' already exists. Skipping creation.

But I cant find it in any way on the system (status file, etc.).

root@icinga repository.d# find . | grep stats
./endpoints/stage-app-stats-1.srv.company.com.conf
./zones/stage-app-stats-1.srv.company.com.conf
./hosts/stage-app-stats-1.srv.company.com.conf
./hosts/stage-app-stats-1.srv.company.com
./hosts/stage-app-stats-1.srv.company.com/ping6.conf
./hosts/stage-app-stats-1.srv.company.com/icinga.conf
./hosts/stage-app-stats-1.srv.company.com/disk.conf
./hosts/stage-app-stats-1.srv.company.com/apt.conf
./hosts/stage-app-stats-1.srv.company.com/load.conf
./hosts/stage-app-stats-1.srv.company.com/procs.conf
./hosts/stage-app-stats-1.srv.company.com/swap.conf
./hosts/stage-app-stats-1.srv.company.com/ping4.conf
./hosts/stage-app-stats-1.srv.company.com/ssh.conf
./hosts/stage-app-stats-1.srv.company.com/disk %2F.conf
root@icinga repository.d#

I dont understand the problem, or what I am missing (we have an automated deployment for the client and its working for other name schemas/new server without any problem). Any idea where/what I could check? May be dump some data from Icinga, database, etc.?

Any help appreciated.

bert2002

[icinga-migration]

Updated by jflach on 2015-11-20 10:33:11 +00:00

icinga2 object list --type Endpoint

Lists all your endpoints and the files they come from.

[icinga-migration]

Updated by bert2002 on 2015-11-20 11:27:44 +00:00

Hi,

the endpoint for the hosts is not created.

[root@icinga repository.d]# icinga2 object list --type Endpoint |grep app-stats-1.srv.company.com  |grep -v stage
[root@icinga repository.d]#

Running update-config, etc. does not help either. :(

[icinga-migration]

Updated by jflach on 2015-11-20 11:43:53 +00:00

Disregard my earlier version of the comment, please

Run icinga2 object list --type Endpoint --name app-stats-1.srv.company.com

Don't grep, you need the whole output

[icinga-migration]

Updated by bert2002 on 2015-11-20 12:45:50 +00:00

Hi,

same issue. no output, only for the staging servers.

[icinga-migration]

Updated by gbeutner on 2015-11-21 03:37:40 +00:00

FWIW I'm unable to reproduce this on a new installation.

[icinga-migration]

Updated by mfriedrich on 2015-11-25 14:16:34 +00:00

• Status changed from New to Feedback
• Assigned to set to bert2002

Please re-test this issue with the latest 2.4.0 release.

[icinga-migration]

Updated by bert2002 on 2015-12-01 14:26:00 +00:00

dnsmichi wrote:

Please re-test this issue with the latest 2.4.0 release.

okay I updated to 2.4.0 (centos) but problem still exist.

[icinga-migration]

Updated by bert2002 on 2015-12-08 08:51:28 +00:00

When a, not working, client connects. I get following error:

[2015-12-08 09:41:20 +0100] warning/JsonRpcConnection: Error while reading JSON-RPC message for identity 'app-stats-1.srv.company.com': Error: std::exception

        (0) libbase.so: void boost::throw_exception(icinga::openssl_error const&) (+0x97) [0x3fbf130a57]
        (1) libbase.so: void boost::exception_detail::throw_exception_(icinga::openssl_error const&, char const*, char const*, int) (+0x40) [0x3fbf130b00]
        (2) libbase.so: icinga::TlsStream::HandleError() const (+0xbc) [0x3fbf0cf46c]
        (3) libbase.so: icinga::TlsStream::Read(void*, unsigned long, bool) (+0x7e) [0x3fbf0cf5de]
        (4) libbase.so: icinga::StreamReadContext::FillFromStream(boost::intrusive_ptr const&, bool) (+0x55) [0x3fbf0de6e5]
        (5) libbase.so: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr const&, icinga::String*, icinga::StreamReadContext&, bool) (+0xce) [0x3fbf0ecdde]
        (6) libremote.so: icinga::JsonRpc::ReadMessage(boost::intrusive_ptr const&, boost::intrusive_ptr*, icinga::StreamReadContext&, bool) (+0x3d) [0x3fbd4c24ad]
        (7) libremote.so: icinga::JsonRpcConnection::ProcessMessage() (+0x65) [0x3fbd4e6555]
        (8) libremote.so: icinga::JsonRpcConnection::DataAvailableHandler() (+0x38) [0x3fbd502e68]
        (9) libbase.so: boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (+0x1bb) [0x3fbf1630ab]
        (10) libbase.so: icinga::Stream::SignalDataAvailable() (+0x27) [0x3fbf113a17]
        (11) libbase.so: icinga::TlsStream::OnEvent(int) (+0x3af) [0x3fbf113e9f]
        (12) libbase.so: icinga::SocketEvents::ThreadProc() (+0x23a) [0x3fbf110aca]
        (13) /usr/lib64/libboost_thread.so.1.53.0() [0x3fbcc0c5c3]
        (14) /lib64/libpthread.so.0() [0x30fa407a51]
        (15) libc.so.6: clone (+0x6d) [0x30fa0e893d]

I double checked the certificates and they are identical. Installed openssl versions:

icinga server:

# rpm -qa |grep openssl
openssl-1.0.1e-42.el6.x86_64
openssl-devel-1.0.1e-42.el6.x86_64

client:

# dpkg -l openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                            Version              Architecture         Description
+++-===============================-====================-====================-====================================================================
ii  openssl                         1.0.1f-1ubuntu2.15   amd64                Secure Sockets Layer toolkit - cryptographic utility

Other clients (with a different name schema) are working with the same openssl version.

[icinga-migration]

Updated by bert2002 on 2015-12-08 10:17:00 +00:00

Okay something "funny" happened. We configured some custom checks on the client and they are getting created on the icinga server. Of course its not working, because the zone for the host does not really exist. I would have expected that the custom checks get ignored as all other checks.

critical/config: Error: Validation failed for object 'app-stats-1.srv.company.com!http_cdh_namenode_2' of type 'Service'; Attribute 'zone': Object 'app-stats-1.srv.company.com' of type 'Zone' does not exist.
Location: in /etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf: 5:2-5:38
/etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf(3):  check_command = "dummy"
/etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf(4):  host_name = "app-stats-1.srv.company.com"
/etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf(5):  zone = "app-stats-1.srv.company.com"
                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf(6): }
/etc/icinga2/repository.d/hosts/app-stats-1.srv.company.com/http_cdh_namenode_2.conf(7): 

[icinga-migration]

Updated by mfriedrich on 2016-02-24 21:55:28 +00:00

We configured some custom checks on the client and they are getting created on the icinga server.

I can't follow here. How are these config objects created on the master? Is the client node's zone configured with the master zone being its parent?

[icinga-migration]

Updated by mfriedrich on 2016-03-18 16:11:17 +00:00

• Status changed from Feedback to Rejected
• Assigned to deleted bert2002

Since we cannot reproduce the issue here and it is not obvious which cli command causes the error, I'll close the issue.