[dev.icinga.com #9312] reload timeout with " Reload failed for Icinga host/service/network monitoring system" #3034
Comments
|
Updated by mfriedrich on 2015-05-25 11:59:01 +00:00
No, there isn't. But I'd prefer fixing the real bug, so could you please attach the gdb backtrace of that crash? |
|
Updated by PowellEB on 2015-05-25 12:10:34 +00:00 Thank you for quick response. I am happy to do a backtrace. Best Regards / Mit freundlichen Grüßen, |
|
Updated by PowellEB on 2015-05-27 02:07:48 +00:00 dnsmichi, Myself and one of our sysadmins watched logs over and over and found: (1) message log has reload command issued (2) 90 seconds later May 25 14:01:08 fitc09v205 systemd: icinga2.service: control process exited, code=exited status=11 (3) 8 minutes later, watching procs, a new pid for icinga2 is now up with a reload from old pid. So it looks like even though console gives error message, the reload does occur, just total time about 9 minutes. If you can provide steps on how I provide a backtrace of all of this, I am happy to comply. For now, we were using this https://github.com/Icinga/icinga2/blob/master/doc/21-debug.md as a guide for gdb, but no luck. Best Regards / Mit freundlichen Grüßen, |
|
Updated by mfriedrich on 2015-05-27 07:52:40 +00:00 Attaching to systemctl isn't what you want. You'll need to attach to the process (use e.g. ps aux) directly using "-p ". In that case the parent process and the forked child process are interesting, so fire up two terminals. You may then fetch the backtrace as usual. |
|
Updated by itbess on 2015-07-08 17:22:46 +00:00 This happens in gdb when the reload error occurs. ~]# gdb program 23482 |
|
Updated by mfriedrich on 2015-08-21 19:47:25 +00:00 Please don't use external paste sites, but attach files here directly. |
|
Updated by mfriedrich on 2015-09-04 09:22:28 +00:00
|
|
Updated by PowellEB on 2015-09-27 15:17:37 +00:00 Michael, finally, thinking thru this again, what is the problem? The timeout is not icinga2, but system parameter (centos) systemctl show icinga2.service -p TimeoutStartUSec In centos this is controlled by (/etc/systemd/system.conf /etc/systemd/user.conf) In our system, modified /etc/systemd/system.conf DefaultTimeoutStartSec=240s So far no problems on reloads or full stops and starts. |
|
Updated by mfriedrich on 2015-09-28 08:20:53 +00:00
|
|
Updated by mfriedrich on 2016-03-09 13:03:57 +00:00
I'm closing this as duplicate of #10226 which has been fixed and released with 2.4.3. |
icinga-migration
added
bug
This issue has been migrated from Redmine: https://dev.icinga.com/issues/9312
Created by PowellEB on 2015-05-25 11:57:25 +00:00
Assignee: (none)
Status: Closed (closed on 2016-03-09 13:03:57 +00:00)
Target Version: (none)
Last Update: 2016-03-09 13:03:57 +00:00 (in Redmine)
We are starting to scale up our icinga2 server in preparation for going
live in Production. Currently 3100 hosts, 11000 services.
Config Reload now takes over 4 minutes, but now crashes.
"..... Reload failed for Icinga host/service/network monitoring system"
Restart works fine.
found issue #7306, #7368 that looks to be our issue as well.
Is there anyway to increase the reload timeout?
Relations:
The text was updated successfully, but these errors were encountered: