New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dev.icinga.com #9312] reload timeout with " Reload failed for Icinga host/service/network monitoring system" #3034
Comments
Updated by mfriedrich on 2015-05-25 11:59:01 +00:00
No, there isn't. But I'd prefer fixing the real bug, so could you please attach the gdb backtrace of that crash? |
Updated by PowellEB on 2015-05-25 12:10:34 +00:00 Thank you for quick response. I am happy to do a backtrace. Best Regards / Mit freundlichen Grüßen, |
Updated by PowellEB on 2015-05-27 02:07:48 +00:00 dnsmichi, Myself and one of our sysadmins watched logs over and over and found: (1) message log has reload command issued (2) 90 seconds later May 25 14:01:08 fitc09v205 systemd: icinga2.service: control process exited, code=exited status=11 (3) 8 minutes later, watching procs, a new pid for icinga2 is now up with a reload from old pid. So it looks like even though console gives error message, the reload does occur, just total time about 9 minutes. If you can provide steps on how I provide a backtrace of all of this, I am happy to comply. For now, we were using this https://github.com/Icinga/icinga2/blob/master/doc/21-debug.md as a guide for gdb, but no luck. Best Regards / Mit freundlichen Grüßen, |
Updated by mfriedrich on 2015-05-27 07:52:40 +00:00 Attaching to systemctl isn't what you want. You'll need to attach to the process (use e.g. ps aux) directly using "-p ". In that case the parent process and the forked child process are interesting, so fire up two terminals. You may then fetch the backtrace as usual. |
Updated by itbess on 2015-07-08 17:22:46 +00:00 This happens in gdb when the reload error occurs. ~]# gdb program 23482
|
Updated by mfriedrich on 2015-08-21 19:47:25 +00:00 Please don't use external paste sites, but attach files here directly. |
Updated by mfriedrich on 2015-09-04 09:22:28 +00:00
|
Updated by PowellEB on 2015-09-27 15:17:37 +00:00 Michael, finally, thinking thru this again, what is the problem? The timeout is not icinga2, but system parameter (centos) systemctl show icinga2.service -p TimeoutStartUSec In centos this is controlled by (/etc/systemd/system.conf /etc/systemd/user.conf) In our system, modified /etc/systemd/system.conf DefaultTimeoutStartSec=240s So far no problems on reloads or full stops and starts. |
Updated by mfriedrich on 2015-09-28 08:20:53 +00:00
|
Updated by mfriedrich on 2016-03-09 13:03:57 +00:00
I'm closing this as duplicate of #10226 which has been fixed and released with 2.4.3. |
This issue has been migrated from Redmine: https://dev.icinga.com/issues/9312
Created by PowellEB on 2015-05-25 11:57:25 +00:00
Assignee: (none)
Status: Closed (closed on 2016-03-09 13:03:57 +00:00)
Target Version: (none)
Last Update: 2016-03-09 13:03:57 +00:00 (in Redmine)
We are starting to scale up our icinga2 server in preparation for going
live in Production. Currently 3100 hosts, 11000 services.
Config Reload now takes over 4 minutes, but now crashes.
"..... Reload failed for Icinga host/service/network monitoring system"
Restart works fine.
found issue #7306, #7368 that looks to be our issue as well.
Is there anyway to increase the reload timeout?
Relations:
The text was updated successfully, but these errors were encountered: