[dev.icinga.com #5773] icinga silently exits when trying to reload a config with errors

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/5773

Created by gvegidy on 2014-03-15 22:05:07 +00:00

Assignee: (none)
Status: Closed (closed on 2014-03-19 13:24:30 +00:00)
Target Version: (none)
Last Update: 2014-03-19 13:24:30 +00:00 (in Redmine)

Icinga Version: 0.0.8-40b88422

---

# systemctl start icinga2.service
# sleep 5
# echo "some garbage"  >>/etc/icinga2/icinga2.conf
# systemctl reload icinga2.service
# tail -n 7 /var/log/messages 
Mar 15 22:53:36 f20 icinga2: Starting Icinga 2:
Mar 15 22:53:36 f20 icinga2: [2014-03-15 22:53:36 +0100]  information/config: Adding include search dir: /usr/share/icinga2
Mar 15 22:53:36 f20 icinga2: Done
Mar 15 22:53:36 f20 systemd: Started LSB: icinga2 host/service/network monitoring and management system.
Mar 15 22:53:42 f20 systemd: Reloading LSB: icinga2 host/service/network monitoring and management system.
Mar 15 22:53:42 f20 systemd: Reloaded LSB: icinga2 host/service/network monitoring and management system.
Mar 15 22:53:42 f20 icinga2: Reloading Icinga 2: Done
# ps ax | grep icinga

-> icinga is not running anymore due to the syntax error in the config file
That is ok with me.

But there is no message in the syslog that icinga2 has been terminated.
But there is no message in the syslog why icinga2 has been terminated.

I have the syslog-feature enabled with level "warning" (the default).

---

Relations:

• relates #5781
• relates #5773
• relates #5773

[icinga-migration]

Updated by gvegidy on 2014-03-16 18:56:49 +00:00

I looked into this a bit, this is what I found:

there is a big red error message written out in this case, but you usually can't see it ;)
You can only see it if you run icinga2 in non-daemon mode (start the program by hand in a console and without the -d option).

Even the messages about shutdown/restart:

 information/base: Shutting down Icinga...
 information/base: Dumping program state to file '/var/lib/icinga2/icinga2.state'
 information/icinga: Icinga has shut down.
 information/base: Restarting application.

aren't written out into syslog or the file log.

I think the reason is the logging system is being shut down too early:

• before every log message received is written out
• before new loggers (from the new config) have taken over successfully.

I guess the second point requires some deeper changes. So another idea would be to check the config on SIGHUP first and only start the whole reload procedure if the config check is successful.

[icinga-migration]

Updated by mfriedrich on 2014-03-16 22:35:10 +00:00

try the fix from #5781

[icinga-migration]

Updated by mfriedrich on 2014-03-18 16:23:34 +00:00

• Target Version set to 0.0.10

[icinga-migration]

Updated by mfriedrich on 2014-03-19 13:24:30 +00:00

• Status changed from New to Closed

• Target Version deleted 0.0.10

  Add support for specifying argument types (string, number, array, ...) in commands #5800 will allow to skip validation on the daemon itself, but let the init script handle it before any restart/reload action. That way the error will be visible without any daemon interaction.