[dev.icinga.com #13567] SIGPIPE shutdown on config reload #4867

icinga-migration · 2016-12-14T14:05:27Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13567

Created by Markus on 2016-12-14 14:05:27 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2017-01-12 09:55:02 +00:00)
Target Version: 2.6.1
Last Update: 2017-01-12 09:55:38 +00:00 (in Redmine)

Icinga Version: 2.6.0-1
Backport?: Not yet backported
Include in Changelog: 1

Today we updated our icinga2 from v2.5.4 to the latest 2.6.0-1.

We have master <-> satellite config with one master, one satellite in a remote site and multiple clients connected.

On every config reload icinga2 does a shutdown but does not start up again. When we start icinga2 again everything works as it did with version 2.5.4. We have not made any config changes during the update.

The same happens when we do a restart (not reload!) after config changes on the satellite. Satellite becomes reload signal via API call and does a shutdown.

Any help is welcome.
Markus

#/etc/init.d/icinga2 checkconfig
[ ok ] checking Icinga2 configuration.
#
# /etc/init.d/icinga2 reload
[ ok ] checking Icinga2 configuration.
[ ok ] icinga2 is running.
[ ok ] Reloading icinga2 monitoring daemon: icinga2.

icinga.log on master

[2016-12-14 14:34:47 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 14:34:47 +0100] information/Application: Received request to shut down.
[2016-12-14 14:34:48 +0100] information/Application: Shutting down...
[2016-12-14 14:34:48 +0100] information/CheckerComponent: Checker stopped.

# /etc/init.d/icinga2 status
[FAIL] icinga2 is not running ... failed!

icinga.log on satellite

[2016-12-14 14:34:48 +0100] warning/TlsStream: TLS stream was disconnected.
[2016-12-14 14:34:48 +0100] warning/JsonRpcConnection: API client disconnected for identity 'zipsmon2001.siepha.net'
[2016-12-14 14:34:48 +0100] warning/ApiListener: Removing API client for endpoint 'zipsmon2001.siepha.net'. 0 API clients left.
[2016-12-14 14:34:58 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'zipsmon2001.siepha.net' via host 'zipsmon2001.siepha.net' and port '5665'
[2016-12-14 14:34:58 +0100] critical/TcpSocket: Invalid socket: Connection refused

icinga.log on satellite after config change

[2016-12-14 13:51:47 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 13:51:47 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 13:51:48 +0100] information/Application: Received request to shut down.
[2016-12-14 13:51:48 +0100] information/Application: Shutting down...
[2016-12-14 13:51:48 +0100] information/CheckerComponent: Checker stopped.

Version informations of master system:

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.0-1)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-4-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers

Version informations of satellite system:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.0-1)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-4-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers

Attachments

strace.log mfrosch - 2016-12-14 16:54:57 +00:00

Changesets

2016-12-15 10:47:07 +00:00 by mfriedrich fb8f410

Block SIGPIPE signal for check processes

refs #13567

2017-01-12 09:50:43 +00:00 by gbeutner 751ca67

Ignore SIGPIPE earlier in the start-up process

fixes #13567

Relations:

relates #8900
duplicates #13567
duplicates #13567
duplicates #13567

The text was updated successfully, but these errors were encountered:

icinga-migration · 2016-12-14T15:02:28Z

Updated by seferovic on 2016-12-14 15:02:28 +00:00

Hi... I can confirm and reproduce this behavior... Icinga Director "kills" my master instance on deployment (I deployed an UserGroup object :) )

[2016-12-14 15:50:34 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 15:50:34 +0100] information/Application: Received request to shut down.
[2016-12-14 15:50:35 +0100] information/Application: Shutting down...
[2016-12-14 15:50:35 +0100] information/CheckerComponent: Checker stopped.

icinga-migration · 2016-12-14T15:42:22Z

Updated by mfrosch on 2016-12-14 15:42:22 +00:00

Whats the systemd output / status after that exit?

Icinga 2 is doing an restart internally, meaning a new process comes up and replaces the old process in sync.

The full log of one of mine servers is:

[2016-12-14 16:37:26 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 16:37:28 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 16:37:28 +0100] information/Application: Received request to shut down.
[2016-12-14 16:37:29 +0100] information/Application: Shutting down...
[2016-12-14 16:37:29 +0100] information/CheckerComponent: Checker stopped.
[2016-12-14 16:37:29 +0100] information/LivestatusListener: Created UNIX socket in '/run/icinga2/cmd/livestatus'.
[2016-12-14 16:37:29 +0100] information/ConfigItem: Activated all objects.
[2016-12-14 16:38:26 +0100] information/ApiListener: New client connection for identity 'XXXXX' from XXXX

The re-connection could be faster possibly, but the service restarts properly...

icinga-migration · 2016-12-14T15:42:30Z

Updated by mfrosch on 2016-12-14 15:42:30 +00:00

Status changed from New to Feedback

icinga-migration · 2016-12-14T15:48:35Z

Updated by mfriedrich on 2016-12-14 15:48:35 +00:00

Category set to libbase
Assigned to set to Markus

Works on CentOS 7 with Systemd, as well as Debian Wheezy in a Docker container with SystemVinit too.

root@066ecc51d6aa:/# service icinga2 restart
[ ok ] checking Icinga2 configuration.
[....] Restarting icinga2 monitoring daemon: icinga2[2016-12-14 15:45:22 +0000] information/cli: Icinga application loader (version: v2.6.0)
[2016-12-14 15:45:22 +0000] information/cli: Loading configuration file(s).
[2016-12-14 15:45:22 +0000] information/ConfigItem: Committing config item(s).
[2016-12-14 15:45:22 +0000] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Zone.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 13 Notifications.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 201 CheckCommands.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Host.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 User.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 12 Services.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2016-12-14 15:45:22 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2016-12-14 15:45:22 +0000] information/ConfigItem: Triggering Start signal for config items
[2016-12-14 15:45:22 +0000] information/ConfigItem: Committing config item(s).
[2016-12-14 15:45:22 +0000] information/ConfigItem: Activated all objects.
. ok
root@066ecc51d6aa:/# service icinga2 status
[ ok ] icinga2 is running.
root@066ecc51d6aa:/# service icinga2 reload
[ ok ] checking Icinga2 configuration.
[ ok ] icinga2 is running.
[ ok ] Reloading icinga2 monitoring daemon: icinga2.
root@066ecc51d6aa:/# service icinga2 status
[ ok ] icinga2 is running.

Any specific OS version/release you're using this on?

icinga-migration · 2016-12-14T15:50:36Z

Updated by Markus on 2016-12-14 15:50:36 +00:00

I posted the last line of the log. There is no more icinga process running. It is like you would stop icinga manually.

Status of icinga is

# /etc/init.d/icinga2 status
[FAIL] icinga2 is not running ... failed!

icinga-migration · 2016-12-14T15:51:18Z

Updated by mfriedrich on 2016-12-14 15:51:18 +00:00

Please add the entire output of "icinga2 --version" (both of you).

icinga-migration · 2016-12-14T15:55:29Z

Updated by Markus on 2016-12-14 15:55:29 +00:00

Version infos are already in my first post.

icinga-migration · 2016-12-14T15:58:49Z

Updated by mfriedrich on 2016-12-14 15:58:49 +00:00

Ah, yes. So you are using SystemVinit instead of the Systemd default on Debian Jessie, right?

icinga-migration · 2016-12-14T16:02:22Z

Updated by Markus on 2016-12-14 16:02:22 +00:00

Yes, sysvinit is used on both systems.

icinga-migration · 2016-12-14T16:09:19Z

Updated by mfrosch on 2016-12-14 16:09:19 +00:00

Tried it with the bare initscript on Jessie:

_SYSTEMCTL_SKIP_REDIRECT=true /etc/init.d/icinga2 start
_SYSTEMCTL_SKIP_REDIRECT=true /etc/init.d/icinga2 reload

The satellite's log ends with:

[2016-12-14 17:03:38 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 17:03:39 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 17:03:39 +0100] information/Application: Received request to shut down.
[2016-12-14 17:03:40 +0100] information/Application: Shutting down...
[2016-12-14 17:03:40 +0100] information/CheckerComponent: Checker stopped.

icinga-migration · 2016-12-14T16:10:58Z

Updated by mfrosch on 2016-12-14 16:10:58 +00:00

Status changed from Feedback to Assigned
Assigned to changed from Markus to mfrosch

icinga-migration · 2016-12-14T16:17:32Z

Updated by rwaffen on 2016-12-14 16:17:32 +00:00

can confirm this on CentOS 6.8 ... with initV ... service icinga2 reload stops the icinga2 deamon

 rpm -qa| grep icinga
icinga2-2.6.0-1.el6.x86_64
icingaweb2-vendor-dompdf-0.6.2-1.el6.noarch
icingaweb2-common-2.3.4-1.el6.noarch
icinga2-common-2.6.0-1.el6.x86_64
icinga2-bin-2.6.0-1.el6.x86_64
icinga2-ido-mysql-2.6.0-1.el6.x86_64
icingaweb2-2.3.4-1.el6.noarch
icingaweb2-vendor-HTMLPurifier-4.7.0-1.el6.noarch
icingaweb2-vendor-Parsedown-1.0.0-1.el6.noarch
icingaweb2-vendor-JShrink-1.0.1-1.el6.noarch
icinga2-libs-2.6.0-1.el6.x86_64
icingaweb2-vendor-lessphp-0.4.0-1.el6.noarch

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.0)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /var/run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /var/run/icinga2/icinga2.pid

System information:
  Platform: CentOS
  Platform version: 6.8 (Final)
  Kernel: Linux
  Kernel version: 2.6.32-642.3.1.el6.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.2
  Build host: 4d3d57293387

icinga-migration · 2016-12-14T16:55:14Z

Updated by mfrosch on 2016-12-14 16:55:14 +00:00

File added strace.log
Assigned to deleted ~~mfrosch~~
Priority changed from Normal to High

To sum up the problem.

When the icinga2 daemon is started in foreground, or by sysvinit, and a reload is issued, icinga shuts down. And the new process does not take over.

Tested on Debian Jessie:

cat >> /etc/icinga2/icinga2.conf <&1 | tee strace.log &
sleep 5
kill -HUP $(cat /run/icinga2/icinga2.pid)

Interesting parts of the strace:

execve("/usr/sbin/icinga2", ["icinga2", "daemon"], [/* 25 vars */]) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/sbin/icinga2", O_RDONLY)     = 3
Process 17276 attached

# child
[pid 17275] execve("/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2", ["/usr/lib/x86_64-linux-gnu/icinga"..., "--no-stack-rlimit", "daemon"], [/* 25 vars */] 

# SIGHUP
[pid 17275] --- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=6359, si_uid=0} ---

Process 17336 attached
[pid 17336] execve("/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2", ["/usr/lib/x86_64-linux-gnu/icinga"..., "--no-stack-rlimit", "daemon", "--reload-internal", "17275"], [/* 26 vars */]) = 0

# new child killing old
[pid 17275] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=17336, si_uid=109} ---

[pid 17275] open("/run/icinga2/icinga2.pid", O_RDWR) = 11
[pid 17275] open("/var/lib/icinga2/icinga2.state.jE9TzE", O_RDWR|O_CREAT|O_EXCL, 0600) = 11
[pid 17275] open("/var/lib/icinga2/icinga2.state.jE9TzE", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 20
[pid 17275] open("/var/lib/icinga2/modified-attributes.conf.CFMKZF", O_RDWR|O_CREAT|O_EXCL, 0600) = 11
[pid 17275] open("/var/lib/icinga2/modified-attributes.conf.CFMKZF", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 20

## NOT in sync with strace output
[2016-12-14 17:38:40 +0100] information/ConfigItem: Activated all objects.
[2016-12-14 17:38:46 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 17:38:46 +0100] information/Application: Received request to shut down.
[2016-12-14 17:38:47 +0100] information/Application: Shutting down...
[2016-12-14 17:38:47 +0100] information/CheckerComponent: Checker stopped.
[2016-12-14 17:38:47 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2016-12-14 17:38:47 +0100] information/IcingaApplication: Icinga has shut down.

# all get killed by SIGPIPE
[pid 17373] open("/var/lib/icinga2/api/log/current", O_WRONLY|O_CREAT|O_APPEND, 0666) = 11
[pid 17373] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=17336, si_uid=109} ---
[pid 17373] +++ killed by SIGPIPE +++
[pid 17372] +++ killed by SIGPIPE +++
[pid 17371] +++ killed by SIGPIPE +++
[pid 17370] +++ killed by SIGPIPE +++
[pid 17369] +++ killed by SIGPIPE +++
[pid 17368] +++ killed by SIGPIPE +++
[pid 17367] +++ killed by SIGPIPE +++
[pid 17342] +++ killed by SIGPIPE +++
[pid 17341] +++ killed by SIGPIPE +++
[pid 17340] +++ killed by SIGPIPE +++
[pid 17339] +++ killed by SIGPIPE +++
[pid 17338] +++ killed by SIGPIPE +++
[pid 17337] +++ killed by SIGPIPE +++
[pid 17366] +++ killed by SIGPIPE +++
[pid 17336] +++ killed by SIGPIPE +++
+++ exited with 0 +++

This does not happen when:

on my master (also run in foreground)
started by systemd

icinga-migration · 2016-12-14T18:48:52Z

Updated by seferovic on 2016-12-14 18:48:52 +00:00

Sorry for not posting the info before...

[root@master ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.0)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /var/run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /var/run/icinga2/icinga2.pid

System information:
  Platform: CentOS
  Platform version: 6.8 (Final)
  Kernel: Linux
  Kernel version: 2.6.32-642.11.1.el6.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.2
  Build host: 4d3d57293387

If you need anything else (mfrosch has a pretty good analysis) I am willing to provide feedback...

icinga-migration · 2016-12-14T23:34:08Z

Updated by seferovic on 2016-12-14 23:34:08 +00:00

Interesting behavior.. if I clean out the conf.d directory and only leave api-users.conf and app.conf .. the reload is working as expected.

icinga-migration · 2016-12-15T10:52:15Z

Updated by mfriedrich on 2016-12-15 10:52:15 +00:00

Relates set to 8900

icinga-migration · 2016-12-15T10:58:02Z

Updated by gbeutner on 2016-12-15 10:58:02 +00:00

The problem is that the writing side of the pipes that are passed to the new Icinga child process for stdout/stderr are closed once the old Icinga process dies. This problem has been around for a while, however we didn't notice this because we were ignoring SIGPIPE by default.

This is no longer the case - at least until after the config validation is finished. Only then do we reset SIGPIPE to SIG_IGN.

The result of this is that depending on the length of the log output for the config validation we might end up write()-ing to a pipe whose writing side has been closed. The SIGPIPE signal we get from that causes the new Icinga process to die.

There are two possible fixes:

Ignore SIGPIPE in the child processes (quick & dirty).
Don't use new pipe FDs for the Icinga child process and use the "real" stdout/stderr from the parent process instead. This would also solve the issue of missing journald output after a reload.

icinga-migration · 2016-12-15T11:07:07Z

Updated by mfriedrich on 2016-12-15 11:07:07 +00:00

Systemd ignores SIGPIPE by default, that's probably the reason why you only experience that with SystemVinit or running in foreground.

https://lists.freedesktop.org/archives/systemd-devel/2014-August/021982.html

icinga-migration · 2016-12-15T11:10:29Z

Updated by mfrosch on 2016-12-15 11:10:29 +00:00

Quick Fix for sysVinit

Please insert into initscript or default/sysconfig:

trap "" SIGPIPE
# or when not using bash for initscript (check shebang):
trap "" 13

This ignores SIGPIPE for all child processes spawned. And mimics the behavior of systemd.exec.

icinga-migration · 2016-12-15T11:51:18Z

Updated by mfriedrich on 2016-12-15 11:51:18 +00:00

The quick&dirty fix is available in git master & inside the snapshot packages (specifically Centos6 and Debian8 for you, tested CentOS7 over here).

You can also reproduce the issue using the "icinga2x-cluster" Vagrant box using an old snapshot. The current revision will not have this behaviour as it already pulls in the latest snapshot packages.

icinga-migration · 2016-12-15T20:27:34Z

Updated by Markus on 2016-12-15 20:27:34 +00:00

mfrosch wrote:

h1. Quick Fix for sysVinit

Please insert into initscript or default/sysconfig:

[...]

This ignores SIGPIPE for all child processes spawned. And mimics the behavior of systemd.exec.

I've tested the quick fix in initscript. Reload is working now.
Many thanks.

icinga-migration · 2016-12-16T08:41:35Z

Updated by mfriedrich on 2016-12-16 08:41:35 +00:00

Duplicated set to 13631

icinga-migration · 2016-12-16T10:20:08Z

Updated by denny on 2016-12-16 10:20:08 +00:00

hi,

I have the same problem:

Debian Jessie with Sysvinit
Icinga2 2.6.0-1~debmon8+1

===
[...]
[2016-12-16 11:10:56 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2016-12-16 11:10:56 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2016-12-16 11:10:56 +0100] information/Application: Shutting down...
[2016-12-16 11:10:56 +0100] information/CheckerComponent: Checker stopped.

/var/log/icinga2/icinga2.log <
[...]
[2016-12-16 11:10:56 +0100] information/Application: Shutting down...
[2016-12-16 11:10:56 +0100] information/CheckerComponent: Checker stopped.
==

The problem starts with the latest 2.5.x version. I disabled feature by feature to see, when reload works again. Reload works, if I disable the api feature.

=
# icinga2 daemon -C
information/cli: Icinga application loader (version: r2.6.0-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: mon3.f00.net
information/ConfigItem: Instantiated 4 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 12 Zones.
information/ConfigItem: Instantiated 2 FileLoggers.
information/ConfigItem: Instantiated 8 Endpoints.
information/ConfigItem: Instantiated 4 NotificationCommands.
information/ConfigItem: Instantiated 122 Notifications.
information/ConfigItem: Instantiated 209 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 9 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 3 HostGroups.
information/ConfigItem: Instantiated 1 EventCommand.
information/ConfigItem: Instantiated 2 UserGroups.
information/ConfigItem: Instantiated 2 Users.
information/ConfigItem: Instantiated 130 Services.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 CompatLogger.
information/ConfigItem: Instantiated 1 StatusDataWriter.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 InfluxdbWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

cu denny

icinga-migration · 2016-12-16T10:21:45Z

Updated by mfriedrich on 2016-12-16 10:21:45 +00:00

Status changed from Assigned to Feedback

Does the workaround work for you? I'd suggest switching to Systemd on Debian Jessie as well, but that probably results in a flame war.

icinga-migration · 2016-12-16T10:37:14Z

Updated by denny on 2016-12-16 10:37:14 +00:00

hi,

I changed the line "158"

trap 'status=2;' INT # handle intr here

with

trap "" SIGPIPE

It doesn't change the behavior. If i put it anywhere else:

===
trap: SIGPIPE: bad trap

===

icinga-migration · 2016-12-17T08:31:02Z

Updated by MarcoKl on 2016-12-17 08:31:02 +00:00

Hello,
In my case, the following workaround works with Ubuntu 16.04.
Insert

trap "" 13

at the beginning of the icinga2 init script.

icinga-migration · 2016-12-18T16:22:15Z

Updated by denny on 2016-12-18 16:22:15 +00:00

hi,

thanks @marcokl works for me too :-)

icinga-migration · 2016-12-19T20:38:54Z

Updated by rnekkanti on 2016-12-19 20:38:54 +00:00

Hi..,

I am running Ubuntu 14.04 with icinga2 2.6.0-1. I have tried updating the init script by adding trap "" 13 at the start of the init script and in line 158 but icinga2 does not seem to restart after deploying configs through director. It is just shutting down.

[2016-12-19 20:35:21 +0000] information/ApiListener: New client connection from [127.0.0.1]:60518 (no client certificate)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/ops-monitor-test-1-1482179721-0 (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:22 +0000] information/Application: Got reload command: Starting new instance.
[2016-12-19 20:35:22 +0000] information/Application: Received request to shut down.
[2016-12-19 20:35:23 +0000] information/Application: Shutting down...
[2016-12-19 20:35:23 +0000] information/CheckerComponent: Checker stopped.
^C

icinga-migration · 2016-12-21T10:38:34Z

Updated by tgelf on 2016-12-21 10:38:34 +00:00

Relates set to 13713

icinga-migration · 2016-12-21T10:38:41Z

Updated by tgelf on 2016-12-21 10:38:41 +00:00

Relates deleted ~~13713~~

icinga-migration · 2016-12-21T10:38:47Z

Updated by tgelf on 2016-12-21 10:38:47 +00:00

Duplicated set to 13713

icinga-migration · 2016-12-21T10:56:11Z

Updated by spillerm on 2016-12-21 10:56:11 +00:00

Hi,

adding trap "" SIGPIPE at the beginning of init script solves it for me using Ubuntu 14.04.5 LTS.
Sorry for the duplicated bug report.

Cheers, Marianne

icinga-migration · 2016-12-21T10:57:37Z

Updated by tgelf on 2016-12-21 10:57:37 +00:00

spillerm wrote:

adding trap "" SIGPIPE at the beginning of init script solves it for me using Ubuntu 14.04.5 LTS.

Good to hear that!

Sorry for the duplicated bug report.

Well... sorry for the bug.

icinga-migration · 2016-12-21T17:23:56Z

Updated by rnekkanti on 2016-12-21 17:23:56 +00:00

Adding trap "" SIGPIPE toinit script does not seem to work when deploying configs through director. Iicinga2 is still failing to restart.

[2016-12-21 17:19:26 +0000] information/ApiListener: New client connection from [127.0.0.1]:35286 (no client certificate)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: DELETE /v1/config/stages/director/ops-monitor-test-1-1482184649-0 (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: POST /v1/config/stages/director (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/director-global/001-director-basics.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/director-global/commands.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/slave-emv/host_templates.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/slave-emv/hosts.conf
[2016-12-21 17:19:27 +0000] information/ApiListener: New client connection from [127.0.0.1]:35287 (no client certificate)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/ops-monitor-test-1-1482340767-0 (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:28 +0000] information/Application: Got reload command: Starting new instance.
[2016-12-21 17:19:28 +0000] information/Application: Received request to shut down.
[2016-12-21 17:19:28 +0000] information/Application: Shutting down...
[2016-12-21 17:19:28 +0000] information/CheckerComponent: Checker stopped.

icinga-migration · 2016-12-22T08:18:30Z

Updated by Nagodar on 2016-12-22 08:18:30 +00:00

Same bug I reported here I think: https://dev.icinga.com/issues/13317

icinga-migration · 2016-12-22T10:49:54Z

Updated by mfrosch on 2016-12-22 10:49:54 +00:00

Duplicated set to 13317

icinga-migration · 2016-12-22T16:38:27Z

Updated by plarivee on 2016-12-22 16:38:27 +00:00

Had same issue, testing adding
sig "" 13
ubuntu 14.04

seems already more stable. Will report back later

icinga-migration · 2016-12-22T18:21:32Z

Updated by rnekkanti on 2016-12-22 18:21:32 +00:00

sig "" 13 on top of the icinga2 init script seems to fix the issue.

icinga-migration · 2016-12-22T19:21:53Z

Updated by plarivee on 2016-12-22 19:21:53 +00:00

have been hamering for the last hour with multiple deploy and it did not stopped.

sig "" 13

seems to be the way in Ubuntu 14.04 for now.

icinga-migration · 2016-12-23T08:39:10Z

Updated by marcof on 2016-12-23 08:39:10 +00:00

We installed the 2.6.0-2 version on Ubuntu Trusty and Xenial and RHEL7, this fixed the issue. one strange thing though: "icinga2 --version" still shows 2.6.0-1 but the bug is gone.

icinga-migration · 2016-12-25T10:20:02Z

Updated by mfrosch on 2016-12-25 10:20:02 +00:00

marcof wrote:

We installed the 2.6.0-2 version on Ubuntu Trusty and Xenial and RHEL7, this fixed the issue. one strange thing though: "icinga2 --version" still shows 2.6.0-1 but the bug is gone.

icinga2 -~~version does not reflect the Debian version :~~)

icinga-migration · 2017-01-02T11:44:48Z

Updated by fugstrolch on 2017-01-02 11:44:48 +00:00

fyi: we have some centos5 Clients with Icinga2 V2.0.6.

Icinga2 -Process on these servers stopped when we've changed our Master's "global-templates"-config.

Adding

trap "" 13

seems to solve this problem.
We had to do this on all Icinga2-Centos-5-Clients.

#!/bin/sh
trap "" 13
#
# chkconfig: 35 90 12
# description: Icinga 2
...

icinga-migration · 2017-01-09T15:20:48Z

Updated by mfriedrich on 2017-01-09 15:20:48 +00:00

Target Version set to 2.6.1

icinga-migration · 2017-01-11T16:43:19Z

Updated by mfriedrich on 2017-01-11 16:43:19 +00:00

Subject changed from icinga2 does shutdown on config reload to SIGPIPE shutdown on config reload

icinga-migration · 2017-01-12T09:55:02Z

Updated by gbeutner on 2017-01-12 09:55:02 +00:00

Status changed from Feedback to Resolved
Done % changed from 0 to 100

Applied in changeset 751ca67.

icinga-migration · 2017-01-12T09:55:38Z

Updated by mfriedrich on 2017-01-12 09:55:38 +00:00

Assigned to set to gbeutner

Dark-o · 2017-05-05T06:37:35Z

Hi,
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)

on Ubuntu 14.04.5 LTS

Since about a week I'm experiencing the same problem that is driving me crazy.

In the init script I've found:

# Block/ignore SIGPIPE inside Icinga2
# Workaround for a known bug in 2.6.0
# see https://dev.icinga.com/issues/13567#note-19
trap '' 13 # SIGPIPE

But the problem still there.
Any Suggestion?
thx.

icinga-migration closed this as completed Jan 12, 2017

icinga-migration added blocker Blocks a release or needs immediate attention bug Something isn't working libbase labels Jan 17, 2017

icinga-migration added this to the 2.6.1 milestone Jan 17, 2017

dnsmichi mentioned this issue Jan 23, 2017

Problem with StatusDataWriter in v2.6.0 #4938

Closed

Thomas-Gelf mentioned this issue Jan 24, 2017

Icinga2 daemon is stopped when configs are deployed. Icinga/icingaweb2-module-director#747

Closed

dnsmichi mentioned this issue Jan 26, 2017

Icinga2 Service on RHEL5 Satellite/Agent stops after reload/config change from master #4961

Closed

mtdeguzis mentioned this issue Feb 9, 2018

Icingaweb2 crashes upon clicking "deploy" Icinga/icingaweb2-module-director#1386

Closed

dnsmichi mentioned this issue Jan 28, 2019

Resetting SIGPIPE for checkcommand child processes #6912

Closed

[dev.icinga.com #13567] SIGPIPE shutdown on config reload #4867

[dev.icinga.com #13567] SIGPIPE shutdown on config reload #4867

Comments

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 14, 2016

icinga-migration commented Dec 15, 2016

icinga-migration commented Dec 15, 2016

icinga-migration commented Dec 15, 2016

icinga-migration commented Dec 15, 2016

Quick Fix for sysVinit

icinga-migration commented Dec 15, 2016

icinga-migration commented Dec 15, 2016

icinga-migration commented Dec 16, 2016

icinga-migration commented Dec 16, 2016

icinga-migration commented Dec 16, 2016

icinga-migration commented Dec 16, 2016

icinga-migration commented Dec 17, 2016

icinga-migration commented Dec 18, 2016

icinga-migration commented Dec 19, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 21, 2016

icinga-migration commented Dec 22, 2016

icinga-migration commented Dec 22, 2016

icinga-migration commented Dec 22, 2016

icinga-migration commented Dec 22, 2016

icinga-migration commented Dec 22, 2016

icinga-migration commented Dec 23, 2016

icinga-migration commented Dec 25, 2016

icinga-migration commented Jan 2, 2017

icinga-migration commented Jan 9, 2017

icinga-migration commented Jan 11, 2017

icinga-migration commented Jan 12, 2017

icinga-migration commented Jan 12, 2017

Dark-o commented May 5, 2017 • edited

Dark-o commented May 5, 2017 •

edited