[dev.icinga.com #13693] icinga 2 segfault

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13693

Created by tobiasvdk on 2016-12-20 07:46:04 +00:00

Assignee: tobiasvdk
Status: Feedback
Target Version: (none)
Last Update: 2017-01-11 17:17:23 +00:00 (in Redmine)

Icinga Version: 2.6.0
Backport?: Not yet backported
Include in Changelog: 1

---

My satellite crashed with the following log:

Application information:
  Application version: r2.6.0-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 4.7.0-0.bpo.1-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: buildbot
Stacktrace:

    (0) libpthread.so.0:  (+0xf8d0) [0x2ab384ccc8d0]
    (1) libc.so.6: gsignal (+0x37) [0x2ab38767c067]
    (2) libc.so.6: abort (+0x148) [0x2ab38767d448]
    (3) libc.so.6:  (+0x2e266) [0x2ab387675266]
    (4) libc.so.6:  (+0x2e312) [0x2ab387675312]
    (5) libbase.so.2.6.0:  (+0xd7bbf) [0x2ab384fb1bbf]
    (6) libbase.so.2.6.0: icinga::Array::~Array() (+0x4f) [0x2ab384fb29cf]
    (7) libicinga.so.2.6.0:  (+0x95d4f) [0x2ab38d8ccd4f]
    (8) libicinga.so.2.6.0: icinga::ObjectImpl::~ObjectImpl() (+0x85) [0x2ab38d8f04c5]
    (9) libicinga.so.2.6.0: icinga::CheckResult::~CheckResult() (+0x17) [0x2ab38d9ea4f7]
    (10) libicinga.so.2.6.0: icinga::Checkable::ProcessCheckResult(boost::intrusive_ptr const&, boost::intrusive_ptr const&) (+0xe04) [0x2ab38d9c4ee4]
    (11) libmethods.so.2.6.0: icinga::PluginCheckTask::ProcessFinishedHandler(boost::intrusive_ptr const&, boost::intrusive_ptr const&, icinga::Value const&, icinga::ProcessResult const&) (+0x4cd) [0x2ab38dd716bd]
    (12) libicinga.so.2.6.0:  (+0x1b7c73) [0x2ab38d9eec73]
    (13) libbase.so.2.6.0:  (+0xdd950) [0x2ab384fb7950]
    (14) libbase.so.2.6.0: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x369) [0x2ab384f68789]
    (15) libboost_thread.so.1.55.0:  (+0xdaea) [0x2ab38431faea]
    (16) libpthread.so.0:  (+0x80a4) [0x2ab384cc50a4]
    (17) libc.so.6: clone (+0x6d) [0x2ab38772f62d]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://dev.icinga.org/ and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[icinga-migration]

Updated by de1m on 2016-12-20 11:48:08 +00:00

yes, I've the same error in 2.6.0 and Ubuntu 16.04.1 TLS

And in /var/log/icinga2/icinga2.err

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:174: T* boost::intrusive_ptr::operator->() const [with T = icinga::Socket]: Assertion `px != 0' failed.
icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:174: T* boost::intrusive_ptr::operator->() const [with T = icinga::Socket]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-12-20 11:34:02 +0100

[icinga-migration]

Updated by akrus on 2016-12-30 07:27:33 +00:00

Same as here:
https://dev.icinga.com/issues/11061

And we have the same on 2.5.6, crashed four times already.

[icinga-migration]

Updated by mfriedrich on 2017-01-09 16:58:06 +00:00

• Category set to Cluster
• Status changed from New to Feedback
• Assigned to set to tobiasvdk

Can you please add more details on how to reliably reproduce the issue? Is this a HA cluster setup, how many objects are checked, which type, possible things which could influence the crash like a remote instance restart, etc.?

[icinga-migration]

Updated by tobiasvdk on 2017-01-09 17:22:55 +00:00

It's a distributed setup without any HA. No config/icinga instance reload involved. I still use NRPE for remote checks. I also experienced this only once and the performance monitoring shows nothing abnormal.

Hosts / Services:

$ sudo icinga2 object list --type Host | grep -c ^Object
728
$ sudo icinga2 object list --type Service | grep -c ^Object
31139

[icinga-migration]

Updated by mfriedrich on 2017-01-11 17:17:23 +00:00

@de1m
Yours seems to be a different error, loosely related Create a new issue for that, if that is not #13655.

@akrus
That's not the same error, #11061 fails with a FIFO stream error. Which exact error do you get? Btw - 2.5.6 has never been released. Please show the output of "icinga2 --version" as well.

@tobiasvdk
Are those upstream packages or custom built ones?

[dnsmichi]

Can you please test again with v2.6.1?

[dnsmichi]

@tobiasvdk ping?