Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #12898] icinga2 process crash #4727

Closed
icinga-migration opened this issue Oct 12, 2016 · 11 comments
Closed

[dev.icinga.com #12898] icinga2 process crash #4727

icinga-migration opened this issue Oct 12, 2016 · 11 comments
Labels
area/api REST API bug Something isn't working

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12898

Created by ebisu on 2016-10-12 12:54:32 +00:00

Assignee: (none)
Status: New
Target Version: (none)
Last Update: 2016-11-21 10:18:00 +00:00 (in Redmine)

Icinga Version: r2.5.4-1
Backport?: Not yet backported
Include in Changelog: 1

Hello,

after some hours my Icinga2 Instance is crashing. In the error log is:
icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Stream]: Assertion `px != 0' failed.
Caught SIGABRT.

This happend after the last update. I use the latest version from:
http://ppa.launchpad.net/formorer/icinga/ubuntu

Icinga2 version:
icinga2 - The Icinga 2 network monitoring daemon (version: r2.5.4-1)

System information:
Platform: Ubuntu
Platform version: 14.04.5 LTS, Trusty Tahr
Kernel: Linux
Kernel version: 3.13.0-96-generic
Architecture: x86_64

When I keep restarting it with a cronjob it never crashes.

If there is any Info you need just ask.

Thanks for the help!

Relations:
@icinga-migration
Copy link
Author

Updated by ebisu on 2016-10-12 13:07:19 +00:00

The Crash log:

System information:
Platform: Ubuntu
Platform version: 14.04.5 LTS, Trusty Tahr
Kernel: Linux
Kernel version: 3.13.0-96-generic
Architecture: x86_64

Build information:
Compiler: GNU 4.8.4
Build host: lgw01-18
Stacktrace:

(0) libpthread.so.0: (+0x10330) [0x7fe0f8403330]
(1) libc.so.6: gsignal (+0x37) [0x7fe0f7167c37]
(2) libc.so.6: abort (+0x148) [0x7fe0f716b028]
(3) libc.so.6: (+0x2fbf6) [0x7fe0f7160bf6]
(4) libc.so.6: (+0x2fca2) [0x7fe0f7160ca2]
(5) libremote.so: (+0xb5a83) [0x7fe0f55aba83]
(6) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7bb) [0x7fe0f55f0dbb]
(7) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x29) [0x7fe0f55f1219]
(8) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x70) [0x7fe0f55f1c10]
(9) libremote.so: icinga::HttpServerConnection::Start() (+0x18f) [0x7fe0f55f1eff]
(10) libremote.so: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptricinga::Socket const&, icinga::String const&, icinga::ConnectionRole) (+0x449) [0x7fe0f5608409]
(11) libremote.so: icinga::ApiListener::NewClientHandler(boost::intrusive_ptricinga::Socket const&, icinga::String const&, icinga::ConnectionRole) (+0x12) [0x7fe0f56096e2]
(12) libboost_thread.so.1.54.0: (+0xba4a) [0x7fe0f8a8ea4a]
(13) libpthread.so.0: (+0x8184) [0x7fe0f83fb184]
(14) libc.so.6: clone (+0x6d) [0x7fe0f722b37d]

  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://dev.icinga.org/ and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.
    *****

@icinga-migration
Copy link
Author

Updated by akrus on 2016-10-14 07:33:16 +00:00

Confirming the same, but crashed only once. Installed from http://packages.icinga.org/ubuntu

Details:

Application information:
  Application version: r2.5.4-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 14.04.5 LTS, Trusty Tahr
  Kernel: Linux
  Kernel version: 4.2.0-18-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.4
  Build host: lgw01-18
Stacktrace:

        (0) libpthread.so.0:  (+0x10330) [0x2b5745cfb330]
        (1) libc.so.6: gsignal (+0x37) [0x2b5746e3cc37]
        (2) libc.so.6: abort (+0x148) [0x2b5746e40028]
        (3) libc.so.6:  (+0x2fbf6) [0x2b5746e35bf6]
        (4) libc.so.6:  (+0x2fca2) [0x2b5746e35ca2]
        (5) libremote.so:  (+0xb5a83) [0x2b5748ae3a83]
        (6) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7bb) [0x2b5748b28dbb]
        (7) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x29) [0x2b5748b29219]
        (8) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x70) [0x2b5748b29c10]
        (9) libbase.so: boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (+0x1d4) [0x2b57460993f4]
        (10) libbase.so: icinga::Stream::SignalDataAvailable() (+0x30) [0x2b5746040730]
        (11) libbase.so: icinga::TlsStream::OnEvent(int) (+0x244) [0x2b5746040ab4]
        (12) libbase.so: icinga::SocketEventEngineEpoll::ThreadProc(int) (+0x45b) [0x2b574603ca7b]
        (13) libboost_thread.so.1.54.0:  (+0xba4a) [0x2b574566ea4a]
        (14) libpthread.so.0:  (+0x8184) [0x2b5745cf3184]
        (15) libc.so.6: clone (+0x6d) [0x2b5746f0037d]

@icinga-migration
Copy link
Author

Updated by ebisu on 2016-10-14 08:25:40 +00:00

In the Logs I had lots of "Ignoring invalid perfdata value".
I fixed most of them. Now It is running since 14h without a crash.

@icinga-migration
Copy link
Author

Updated by rhillmann on 2016-10-18 13:56:55 +00:00

I can confirm random crashes with the same issue, running sometimes for days before crash:

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-09-30 15:09:15 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-09-30 19:06:45 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Host]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-04 14:03:23 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Host]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-04 21:38:51 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-05 17:06:10 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-14 09:32:36 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-18 15:49:48 +0200

@icinga-migration
Copy link
Author

Updated by obuchmann on 2016-10-19 07:43:42 +00:00

Same here

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Stream]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-13 22:49:09 +0200

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Stream]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-10-18 13:06:09 +0200

Application information:
  Application version: r2.5.4-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 14.04.5 LTS, Trusty Tahr
  Kernel: Linux
  Kernel version: 3.13.0-83-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.4
  Build host: lgw01-18
Stacktrace:

        (0) libpthread.so.0:  (+0x10330) [0x2ab7b131c330]
        (1) libc.so.6: gsignal (+0x37) [0x2ab7b245dc37]
        (2) libc.so.6: abort (+0x148) [0x2ab7b2461028]
        (3) libc.so.6:  (+0x2fbf6) [0x2ab7b2456bf6]
        (4) libc.so.6:  (+0x2fca2) [0x2ab7b2456ca2]
        (5) libremote.so:  (+0xb5a83) [0x2ab7b4104a83]
        (6) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7bb) [0x2ab7b4149dbb]
        (7) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x29) [0x2ab7b414a219]
        (8) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x70) [0x2ab7b414ac10]
        (9) libremote.so: icinga::HttpServerConnection::Start() (+0x18f) [0x2ab7b414aeff]
        (10) libremote.so: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x449) [0x2ab7b4161409]
        (11) libremote.so: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x12) [0x2ab7b41626e2]
        (12) libboost_thread.so.1.54.0:  (+0xba4a) [0x2ab7b0c8fa4a]
        (13) libpthread.so.0:  (+0x8184) [0x2ab7b1314184]
        (14) libc.so.6: clone (+0x6d) [0x2ab7b252137d]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://dev.icinga.org/ and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

@icinga-migration
Copy link
Author

Updated by vince000 on 2016-11-02 10:58:56 +00:00

Same here. Any fix or workaround available?

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Stream]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-11-02 10:40:44 +0100

Application information:
Application version: r2.5.4-1
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

System information:
Platform: Ubuntu
Platform version: 14.04.5 LTS, Trusty Tahr
Kernel: Linux
Kernel version: 3.13.0-100-generic
Architecture: x86_64

Build information:
Compiler: GNU 4.8.4
Build host: lgw01-18
Stacktrace:

(0) libpthread.so.0: (+0x10330) [0x2b263a385330]
(1) libc.so.6: gsignal (+0x37) [0x2b263b4c6c37]
(2) libc.so.6: abort (+0x148) [0x2b263b4ca028]
(3) libc.so.6: (+0x2fbf6) [0x2b263b4bfbf6]
(4) libc.so.6: (+0x2fca2) [0x2b263b4bfca2]
(5) libremote.so: (+0xb5a83) [0x2b263d16da83]
(6) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7bb) [0x2b263d1b2dbb]
(7) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x29) [0x2b263d1b3219]
(8) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x70) [0x2b263d1b3c10]
(9) libremote.so: icinga::HttpServerConnection::Start() (+0x18f) [0x2b263d1b3eff]
(10) libremote.so: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptricinga::Socket const&, icinga::String const&, icinga::ConnectionRole) (+0x449) [0x2b263d1ca409]
(11) libremote.so: icinga::ApiListener::NewClientHandler(boost::intrusive_ptricinga::Socket const&, icinga::String const&, icinga::ConnectionRole) (+0x12) [0x2b263d1cb6e2]
(12) libboost_thread.so.1.54.0: (+0xba4a) [0x2b2639cf8a4a]
(13) libpthread.so.0: (+0x8184) [0x2b263a37d184]
(14) libc.so.6: clone (+0x6d) [0x2b263b58a37d]

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-16 15:34:04 +00:00

  • Category set to API

Can you do me a favour and add the following details to help nail down the issue please :)

Output of

icinga2 feature list

Which tools and endpoints are you using inside the REST API?

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-16 15:40:53 +00:00

  • Relates set to 12667

@icinga-migration
Copy link
Author

Updated by vince000 on 2016-11-17 07:58:22 +00:00

icinga2 feature list
Disabled features: compatlog debuglog gelf graphite icingastatus influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

@icinga-migration
Copy link
Author

Updated by gbeutner on 2016-11-21 10:18:00 +00:00

I believe this should be fixed in the master branch. Unfortunately I am unable to reproduce this problem here so I'm going to have to rely on feedback to see if this issue reoccurs. Please test the snapshot packages. :)

@icinga-migration icinga-migration added bug Something isn't working area/api REST API labels Jan 17, 2017
@coreaut
Copy link

coreaut commented Jan 17, 2017

In the latest Version (r2.6.0-1) this seems to be fixed.
No more crashes since 22.12.2016.

Before it crashed at least once or twice a week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/api REST API bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dnsmichi @coreaut @icinga-migration