[dev.icinga.com #12808] Endpoint not created

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12808

Created by jsmartel68 on 2016-09-27 10:09:58 +00:00

Assignee: jsmartel68
Status: Closed (closed on 2016-12-07 21:35:54 +00:00)
Target Version: (none)
Last Update: 2017-01-14 13:05:58 +00:00 (in Redmine)

Icinga Version: 2.5.4
Backport?: Not yet backported
Include in Changelog: 1

---

Hi,

This issue is the same as the rejected issue #10688. We have just upgraded to 2.5.4 and still have the problem where we try to add a node but the icinga server does not seem to create it properly. But we have a similarly named server in our staging environment that does work.

Server:

gap-batch-ingestion-1.srv.company.com
stage-gap-batch-ingestion-1.srv.company.com

To add the node we use "icinga2 node wizard".

When the node tries to connect to the master we see the following.

 [2016-09-27 10:10:31 +0200] notice/ApiListener: New JSON-RPC client
 [2016-09-27 10:10:31 +0200] notice/JsonRpcConnection: Received 'icinga::Hello' message from 'gap-batch-ingestion-1.srv.company.com'
 [2016-09-27 10:10:31 +0200] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from 'gap-batch-ingestion-1.srv.company.com'
 [2016-09-27 10:10:31 +0200] notice/ClusterEvents: Discarding 'next check changed' message from 'gap-batch-ingestion-1.srv.company.com': Invalid endpoint origin (client not allowed).
 [2016-09-27 10:10:31 +0200] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from 'gap-batch-ingestion-1.srv.comapny.com'
 [2016-09-27 10:10:31 +0200] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from 'gap-batch-ingestion-1.srv.company.com'
 [2016-09-27 10:10:31 +0200] notice/ClusterEvents: Discarding 'next check changed' message from 'gap-batch-ingestion-1.srv.company.com': Invalid endpoint origin (client not allowed).

The node seems to be at least partially created on the master.

# icinga2 repository endpoint add name=gap-batch-ingestion-1.srv.company.com
warning/cli: Endpoint 'gap-batch-ingestion-1.srv.company.com' already exists. Skipping creation.

But nothing shows up in the repository.d/endpoints for this node.

I have followed the suggestions in http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/troubleshooting and confirmed the SSL stuff is correct.

root@gap-batch-ingestion-1:~# openssl verify -verbose -CAfile /etc/icinga2/pki/ca.crt /etc/icinga2/pki/gap-batch-ingestion-1.srv.company.com.crt
/etc/icinga2/pki/gap-batch-ingestion-1.srv.company.com.crt: OK

Could you suggest some more troubleshooting steps or perhaps a way to create the node on the server side?

Thank you in advance,
-Jay

Attachments

• etc-repo-dir.txt jsmartel68 - 2016-09-27 12:15:34 +00:00
• var-lib-repo-dir.txt jsmartel68 - 2016-09-27 12:15:43 +00:00

---

Parent Task: #13257

Relations:

• relates #12808

[icinga-migration]

Updated by mfriedrich on 2016-09-27 11:08:22 +00:00

• Category set to CLI
• Status changed from New to Feedback
• Assigned to set to jsmartel68

Did you run "node list" and "node update-config" prior to checking the repository.d/endpoints directory?

Can you show the output of

ls -lR /etc/icinga2/repository.d
ls -lR /var/lib/icinga2/repository/

[icinga-migration]

Updated by jsmartel68 on 2016-09-27 12:16:29 +00:00

• File added etc-repo-dir.txt
• File added var-lib-repo-dir.txt

Yes, I did run "node update-config" but could not see the client. It does, however, show up in "node list".

# icinga2 node list | grep batch
Node 'stage-gap-batch-ingestion-1.srv.company.com' (last seen: Tue Sep 27 14:08:06 2016)
    * Host 'stage-gap-batch-ingestion-1.srv.company.com'
Node 'gap-batch-ingestion-1.srv.company.com' (last seen: Tue Sep 27 14:08:02 2016)
    * Host 'gap-batch-ingestion-1.srv.company.com'

I have attached as text files the output of the two ls commands you requested.

Kind Regards,
-Jay

[icinga-migration]

Updated by jsmartel68 on 2016-09-27 12:33:15 +00:00

Hi,

I forgot to also mention the pattern in the problem hosts that we have seen. We first create a host in our staging environment with name stage-client-1.srv.company.com. Then after testing, we create a host in our production environment without stage at the front, such as client-1.srv.company.com. While it seems unlikely that this would be causing the problem, I did want to point it out. Here are two examples where the stage server works fine in Icinga but the non-stage has the same error as described:

stage-gap-batch-ingestion-1.srv.company.com
gap-batch-ingestion-1.srv.company.com

stage-app-stats-1.srv.company.com
app-stats-1.srv.company.com

Kind Regards,
-Jay

[icinga-migration]

Updated by jsmartel68 on 2016-11-01 13:03:18 +00:00

Hello,

Are there any updates to this? Can you suggest a way to add the node directly on the server instead of using the wizard on the client?

Best Regards,
-Jay

[icinga-migration]

Updated by mfriedrich on 2016-11-07 15:41:02 +00:00

I do see that endpoint config file 'stage-gap-batch-ingestion-1.srv.company.com.conf' in your etc/repository.d directory. So node update-config is working fine and creating things. Probably your Icinga 2 node isn't aware of the configuration changes and was not restarted properly.

I'd recommend using the "top down" approach though where you'll either distributed the config from the master to the clients entirely, or at least use them as command endpoints.

[icinga-migration]

Updated by jsmartel68 on 2016-11-08 08:18:22 +00:00

Hi, thanks for the reply. However, it is not the stage server that is the problem. The problem is the server without the "stage-" in the front. If you look in the repository.d directory there should be two hosts: stage-gap-batch-ingestion-1.srv.company.com.conf and gap-batch-ingestion-1.srv.company.com.conf, but there is only the one.

Regards,
-Jay

[icinga-migration]

Updated by mfriedrich on 2016-11-18 17:11:01 +00:00

• Relates set to 13255

[icinga-migration]

Updated by mfriedrich on 2016-12-07 21:35:54 +00:00

• Status changed from Feedback to Closed

No idea about that sorry. But as you asked, adding this directly on the node itself instead of using node update-config, I'd suggest to migrate to top down while at it. Additional docs have been added for 2.6 too - https://docs.icinga.com/icinga2/snapshot/doc/module/icinga2/toc\#!/icinga2/snapshot/doc/module/icinga2/chapter/distributed-monitoring#distributed-monitoring-bottom-up-migration-top-down

Closing here.

[icinga-migration]

Updated by mfriedrich on 2017-01-14 13:05:58 +00:00

• Parent Id set to 13257

[dnsmichi]

#4799