[dev.icinga.com #10783] segfault in libremote.so #3751

icinga-migration · 2015-12-07T13:31:15Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10783

Created by mnardin on 2015-12-07 13:31:15 +00:00

Assignee: gbeutner
Status: Rejected (closed on 2016-02-09 08:13:25 +00:00)
Target Version: (none)
Last Update: 2016-02-09 08:13:25 +00:00 (in Redmine)

Icinga Version: v2.4.1
Backport?: Not yet backported
Include in Changelog: 1

Hi,
we just started installing a new icinga2 environment. The setup has 2 master and 2 zones (2 satellites + 1 satellite).
At the moment we just have configured the zones.conf file on all nodes and the ido-feature on the 2 masters.
That means that we have still the example config in conf.d included on all nodes. The setup will feature a config master when the setup is completed.

We use puppet to ensure the running state of the icinga2 service on all servers.

We have noticed that in some way the problem could be related to the service status query during the puppet agent run.

See the following log extract as example:

----
[root@icingas01-p ~]# grep -a1 segfault /var/log/messages
<29>1 2015-12-07T02:24:12.108426+01:00 icingas01-p puppet-agent 12411 - -  Finished catalog run in 1.85 seconds
<6>1 2015-12-07T02:24:14.628722+01:00 icingas01-p kernel  - - icinga2[17044]: segfault at 0 ip 00007f12d8e96774 sp 00007f12d2ca78b0 error 4 in libremote.so[7f12d8dab000+18c000]
<29>1 2015-12-07T02:54:14.755692+01:00 icingas01-p puppet-agent 12756 - -  (/Stage[main]/Risrole_icinga::Base/Service[icinga2]/ensure) ensure changed 'stopped' to 'running'
----
[root@icingas02-p ~]# grep -a1 segfault /var/log/messages
<29>1 2015-12-07T01:54:13.949345+01:00 icingas02-p puppet-agent 8683 - -  Finished catalog run in 1.75 seconds
<6>1 2015-12-07T02:00:11.326386+01:00 icingas02-p kernel  - - icinga2[30619]: segfault at 0 ip 00007f95cc3be774 sp 00007f95cd9e48b0 error 4 in libremote.so[7f95cc2d3000+18c000]
<29>1 2015-12-07T02:24:14.683177+01:00 icingas02-p puppet-agent 9274 - -  (/Stage[main]/Risrole_icinga::Base/Service[icinga2]/ensure) ensure changed 'stopped' to 'running'
----
[root@icingam02-p ~]# grep -a1 segfault /var/log/messages
<29>1 2015-12-07T10:48:29.461788+01:00 icingam02-p puppet-agent 11590 - -  Run of Puppet configuration client already in progress; skipping  (/var/lib/puppet/state/agent_catalog_run.lock exists)
<6>1 2015-12-07T10:48:30.439800+01:00 icingam02-p kernel  - - icinga2[11967]: segfault at 0 ip 00007fb735638774 sp 00007fb715bbd8b0 error 4 in libremote.so[7fb73554d000+18c000]
<29>1 2015-12-07T10:48:30.459385+01:00 icingam02-p puppet-agent 11301 - -  (/Stage[main]/Risrole_icinga::Base/Service[icinga2]/ensure) ensure changed 'stopped' to 'running'
----

This are the all the relevant log lines on the impacted servers:

<6>1 2015-12-07T10:48:30.439800+01:00 icingam02-p kernel  - - icinga2[11967]: segfault at 0 ip 00007fb735638774 sp 00007fb715bbd8b0 error 4 in libremote.so[7fb73554d000+18c000]
<6>1 2015-12-07T02:24:14.628722+01:00 icingas01-p kernel  - - icinga2[17044]: segfault at 0 ip 00007f12d8e96774 sp 00007f12d2ca78b0 error 4 in libremote.so[7f12d8dab000+18c000]
<6>1 2015-12-07T02:00:11.326386+01:00 icingas02-p kernel  - - icinga2[30619]: segfault at 0 ip 00007f95cc3be774 sp 00007f95cd9e48b0 error 4 in libremote.so[7f95cc2d3000+18c000]
<6>1 2015-12-06T02:24:12.374710+01:00 icingas02-p kernel  - - icinga2[17290]: segfault at 0 ip 00007f11f0ee4774 sp 00007f11f250a8b0 error 4 in libremote.so[7f11f0df9000+18c000]
<6>1 2015-12-06T02:00:13.585583+01:00 icingas01-p kernel  - - icinga2[22269]: segfault at 0 ip 00007fd939070774 sp 00007fd93a6d78b0 error 4 in libremote.so[7fd938f85000+18c000]
<6>1 2015-12-05T02:24:12.214500+01:00 icingas02-p kernel  - - icinga2[31656]: segfault at 0 ip 00007fb945e2b774 sp 00007fb9474518b0 error 4 in libremote.so[7fb945d40000+18c000]
<6>1 2015-12-05T02:00:29.666730+01:00 icingam02-p kernel  - - icinga2[3226]: segfault at 0 ip 00007f9d9e458774 sp 00007f9d9fabe8b0 error 4 in libremote.so[7f9d9e36d000+18c000]
<6>1 2015-12-05T02:00:02.099638+01:00 icingas01-p kernel  - - icinga2[16473]: segfault at 0 ip 00007f9776840774 sp 00007f977044b8b0 error 4 in libremote.so[7f9776755000+18c000]
<6>1 2015-12-04T02:25:13.556569+01:00 icingas01-p kernel  - - icinga2[4620]: segfault at 0 ip 00007f66d119f774 sp 00007f66cadaa8b0 error 4 in libremote.so[7f66d10b4000+18c000]
<6>1 2015-12-04T02:00:14.469420+01:00 icingam02-p kernel  - - icinga2[2729]: segfault at 0 ip 00007fdd46e72774 sp 00007fdd484988b0 error 4 in libremote.so[7fdd46d87000+18c000]
<6>1 2015-12-04T02:00:13.652376+01:00 icingas02-p kernel  - - icinga2[5552]: segfault at 0 ip 00007f05ef195774 sp 00007f05f07bb8b0 error 4 in libremote.so[7f05ef0aa000+18c000]
<6>1 2015-12-03T16:30:57.465815+01:00 icingas01-p kernel  - - icinga2[4010]: segfault at 0 ip 00007fe54f9a6774 sp 00007fe550fcc8b0 error 4 in libremote.so[7fe54f8bb000+18c000]
<6>1 2015-12-03T16:30:57.465121+01:00 icingas02-p kernel  - - icinga2[4117]: segfault at 0 ip 00007f3a02121774 sp 00007f3a037478b0 error 4 in libremote.so[7f3a02036000+18c000]
<6>1 2015-12-03T16:29:15.000390+01:00 icingam02-p kernel  - - icinga2[32321]: segfault at 0 ip 00007f76444ce774 sp 00007f763d3398b0 error 4 in libremote.so[7f76443e3000+18c000]

Since the installation only the 2 hosts (icingam01-p, icinga01-d) didn't crash.

What other information should I provide?

Thanks
Mirko

System information:
  Platform: Red
  Platform version: Hat Enterprise Linux Server release 6.7 (Santiago)
  Kernel: Linux
  Kernel version: 2.6.32-573.8.1.el6.x86_64
  Architecture: x86_64

Relations:

duplicates #10783

The text was updated successfully, but these errors were encountered:

icinga-migration · 2015-12-07T13:52:55Z

Updated by mfriedrich on 2015-12-07 13:52:55 +00:00

Description updated

icinga-migration · 2015-12-07T13:57:40Z

Updated by mfriedrich on 2015-12-07 13:57:40 +00:00

ls /var/log/icinga2/crash

Can you run icinga 2 with gdb (gdb --args /usr/lib64/icinga2/sbin/icinga2 daemon and then 'run')?
Attach the full backtrace to this ticket. Details in the docs: http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/development#development-debug-gdb-backtrace

icinga-migration · 2015-12-07T14:00:49Z

Updated by mfriedrich on 2015-12-07 14:00:49 +00:00

Category set to Cluster
Status changed from New to Feedback
Assigned to set to mnardin

icinga-migration · 2015-12-08T08:15:22Z

Updated by mnardin on 2015-12-08 08:15:22 +00:00

Here are the additional information:
No files in the crash directory:

[root@icingas01-p ~]# ls -la /var/log/icinga2/crash
total 8
drwxr-x---. 2 icinga icinga    4096 Nov 26 13:32 .
drwxr-x---. 4 icinga icingacmd 4096 Dec  8 04:09 ..

gdb
last few output lines:

[2015-12-08 02:00:19 +0100] information/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/icingas01-p                                           -1449152116-1/conf.d/downtimes/icingas02-p.ad.raiffeisen.it!load!icingas02-p-1449536419-1.conf
[2015-12-08 02:00:19 +0100] information/ConfigItem: Committing config items
[New Thread 0x7fffb3fbe700 (LWP 27764)]
[2015-12-08 02:00:19 +0100] warning/ConfigObject: Ignoring config object 'icingas02-p-1449536419-1' of type 'Downtime' due to                                            errors: Error: Validation failed for object 'icingas02-p.ad.raiffeisen.it!load!icingas02-p-1449536419-1' of type 'Downtime';                                            Attribute 'host_name': Object 'icingas02-p.ad.raiffeisen.it' of type 'Host' does not exist.
Location: in /var/lib/icinga2/api/packages/_api/icingas01-p-1449152116-1/conf.d/downtimes/icingas02-p.ad.raiffeisen.it!load!i                                           cingas02-p-1449536419-1.conf: 1:0-1:57
/var/lib/icinga2/api/packages/_api/icingas01-p-1449152116-1/conf.d/downtimes/icingas02-p.ad.raiffeisen.it!load!icingas02-p-14                                           49536419-1.conf(1): object Downtime "icingas02-p-1449536419-1" ignore_on_error {
                                                                                                                                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/packages/_api/icingas01-p-1449152116-1/conf.d/downtimes/icingas02-p.ad.raiffeisen.it!load!icingas02-p-14                                           49536419-1.conf(2):  author = "icingaadmin"
/var/lib/icinga2/api/packages/_api/icingas01-p-1449152116-1/conf.d/downtimes/icingas02-p.ad.raiffeisen.it!load!icingas02-p-14                                           49536419-1.conf(3):  comment = "Scheduled downtime for backup"

[2015-12-08 02:00:19 +0100] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41                                           ) for type 'Dependency' does not match anywhere!
[2015-12-08 02:00:19 +0100] information/ConfigItem: Triggering Start signal for config items
[2015-12-08 02:00:19 +0100] information/ConfigItem: Activated all objects.
[Thread 0x7fffb3fbe700 (LWP 27764) exited]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7e54700 (LWP 16094)]
0x00007ffff682c774 in icinga::ApiListener::ConfigUpdateObjectAPIHandler(boost::intrusive_ptr const&, b                                           oost::intrusive_ptr const&) () from /usr/lib64/icinga2/libremote.so
Missing separate debuginfos, use: debuginfo-install icinga2-bin-2.4.1-1.el6.x86_64

bt

(gdb) bt

#0  0x00007ffff682c774 in icinga::ApiListener::ConfigUpdateObjectAPIHandler(boost::intrusive_ptr const&, boost::intrusive_ptr const&) () from /usr/lib64/icinga2/libremote.so
#1  0x00007ffff684904f in boost::detail::function::function_invoker2 const&, boost::intrusive_ptr const&), icinga::Value, boost::intrusive_ptr const&, boost::intrusive_ptr const&>::invoke(boost::detail::function::function_buffer&, boost::intrusive_ptr const&, boost::intrusive_ptr const&) ()

from /usr/lib64/icinga2/libremote.so

#2  0x00007ffff67fb7ed in icinga::ApiFunction::Invoke(boost::intrusive_ptr const&, boost::intrusive_ptr const&) () from /usr/lib64/icinga2/libremote.so
#3  0x00007ffff68279d8 in icinga::JsonRpcConnection::ProcessMessage() () from /usr/lib64/icinga2/libremote.so
#4  0x00007ffff6843e68 in icinga::JsonRpcConnection::DataAvailableHandler() () from /usr/lib64/icinga2/libremote.so
#5  0x00007ffff71f00ab in boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) ()

from /usr/lib64/icinga2/libbase.so

#6  0x00007ffff71a0a17 in icinga::Stream::SignalDataAvailable() () from /usr/lib64/icinga2/libbase.so
#7  0x00007ffff71a0d00 in icinga::TlsStream::OnEvent(int) () from /usr/lib64/icinga2/libbase.so
#8  0x00007ffff719daca in icinga::SocketEvents::ThreadProc() () from /usr/lib64/icinga2/libbase.so
#9  0x00007ffff7bd25c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
#10 0x00007ffff4340a51 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff484693d in clone () from /lib64/libc.so.6

bt full

(gdb) bt full

#0  0x00007ffff682c774 in icinga::ApiListener::ConfigUpdateObjectAPIHandler(boost::intrusive_ptr const&, boost::intrusive_ptr const&) () from /usr/lib64/icinga2/libremote.so

No symbol table info available.

#1  0x00007ffff684904f in boost::detail::function::function_invoker2 const&, boost::intrusive_ptr const&), icinga::Value, boost::intrusive_ptr const&, boost::intrusive_ptr const&>::invoke(boost::detail::function::function_buffer&, boost::intrusive_ptr const&, boost::intrusive_ptr const&) ()

from /usr/lib64/icinga2/libremote.so
No symbol table info available.

#2  0x00007ffff67fb7ed in icinga::ApiFunction::Invoke(boost::intrusive_ptr const&, boost::intrusive_ptr const&) () from /usr/lib64/icinga2/libremote.so

No symbol table info available.

#3  0x00007ffff68279d8 in icinga::JsonRpcConnection::ProcessMessage() () from /usr/lib64/icinga2/libremote.so

No symbol table info available.

#4  0x00007ffff6843e68 in icinga::JsonRpcConnection::DataAvailableHandler() () from /usr/lib64/icinga2/libremote.so

No symbol table info available.

#5  0x00007ffff71f00ab in boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) ()

from /usr/lib64/icinga2/libbase.so
No symbol table info available.

#6  0x00007ffff71a0a17 in icinga::Stream::SignalDataAvailable() () from /usr/lib64/icinga2/libbase.so

No symbol table info available.

#7  0x00007ffff71a0d00 in icinga::TlsStream::OnEvent(int) () from /usr/lib64/icinga2/libbase.so

No symbol table info available.

#8  0x00007ffff719daca in icinga::SocketEvents::ThreadProc() () from /usr/lib64/icinga2/libbase.so

No symbol table info available.

#9  0x00007ffff7bd25c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0

No symbol table info available.

#10 0x00007ffff4340a51 in start_thread () from /lib64/libpthread.so.0

No symbol table info available.

#11 0x00007ffff484693d in clone () from /lib64/libc.so.6

No symbol table info available.

icinga-migration · 2015-12-10T08:45:18Z

Updated by gbeutner on 2015-12-10 08:45:18 +00:00

Can you please install the debug symbols (i.e. by installing the "icinga2-debuginfo" package) and re-run icinga2 with gdb?

icinga-migration · 2015-12-11T07:46:30Z

Updated by mnardin on 2015-12-11 07:46:30 +00:00

Here are the the outputs of gdb with the package "icinga2-debuginfo" installed:

(gdb) bt

#0  icinga::ApiListener::ConfigUpdateObjectAPIHandler (origin=..., params=...) at ../remote/apilistener-configsync.cpp:133
#1  0x00007ffff684904f in boost::detail::function::function_invoker2 const&, boost::intrusive_ptr const&), icinga::Value, boost::intrusive_ptr const&, boost::intrusive_ptr const&>::invoke (function_ptr=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:95

#2  0x00007ffff67fb7ed in operator() (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:767

#3  icinga::ApiFunction::Invoke (this=Unhandled dwarf expression opcode 0xf3

) at ../remote/apifunction.cpp:31

#4  0x00007ffff68279d8 in icinga::JsonRpcConnection::ProcessMessage (this=0x7fffc8002c50) at ../remote/jsonrpcconnection.cpp:173
#5  0x00007ffff6843e68 in icinga::JsonRpcConnection::DataAvailableHandler (this=0x7fffc8002c50) at ../remote/jsonrpcconnection.cpp:197
#6  0x00007ffff71f00ab in operator() (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:767

#7  m_invoke&)>, 0u, const boost::intrusive_ptr&> (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89

#8  operator()&)>, const boost::intrusive_ptr&> (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78

#9  m_invoke >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114

#10 operator() >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107

#11 dereference (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82

#12 dereference&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514

#13 operator* (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639

#14 operator()&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55

#15 operator(), boost::signals2::detail::slot_call_iterator_t&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64

#16 boost::signals2::detail::signal_impl&), boost::signals2::optional_last_value, int, std::less, boost::function&)>, boost::function&)>, boost::signals2::mutex>::operator()(const boost::intrusive_ptr &) (this=Unhandled dwarf expression opcode 0xf3

)
at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246

#17 0x00007ffff71a0a17 in operator() (this=0x7fffc80011b0) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695
#18 icinga::Stream::SignalDataAvailable (this=0x7fffc80011b0) at ../base/stream.cpp:55
#19 0x00007ffff71a0d00 in icinga::TlsStream::OnEvent (this=0x7fffc80011b0, revents=Unhandled dwarf expression opcode 0xf3

) at ../base/tlsstream.cpp:197

#20 0x00007ffff719daca in icinga::SocketEvents::ThreadProc () at ../base/socketevents.cpp:147
#21 0x00007ffff7bd25c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
#22 0x00007ffff4340a51 in start_thread () from /lib64/libpthread.so.0
#23 0x00007ffff484693d in clone () from /lib64/libc.so.6

(gdb) bt full

#0  icinga::ApiListener::ConfigUpdateObjectAPIHandler (origin=..., params=...) at ../remote/apilistener-configsync.cpp:133

errors = {px = 0x7fffcc032900}
objType = {static NPos = 18446744073709551615, m_Data = "Downtime"}
objName = {static NPos = 18446744073709551615, m_Data = "icingas02-p.ad.raiffeisen.iticingas02-p-1449795613-1"}
endpoint = {px = 0x7fffd8031b40}
objOriginalAttributes = {px = 0x0}
objVersion = 1449795613.608484
dtype = {px = 0x72b8e0}
modified_attributes = {px = 0x0}
listener = {px = 0x7fffd8000c50}
object = {px = 0x0}
config = {static NPos = 18446744073709551615, m_Data =
"object Downtime \"icingas02-p-1449795613-1\" ignore_on_error {\n\tauthor = \"icingaadmin\"\n\tcomment = \"Scheduled downtime for backup\"\n\tconfig_owner = \"icingas02-p.ad.raiffeisen.itbackup-downtime\"\n\tdur"...}
newOriginalAttributes = {px = 0x0}

#1  0x00007ffff684904f in boost::detail::function::function_invoker2 const&, boost::intrusive_ptr const&), icinga::Value, boost::intrusive_ptr const&, boost::intrusive_ptr const&>::invoke (function_ptr=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:95
f =

#2  0x00007ffff67fb7ed in operator() (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:767
No locals.

#3  icinga::ApiFunction::Invoke (this=Unhandled dwarf expression opcode 0xf3

) at ../remote/apifunction.cpp:31
No locals.

#4  0x00007ffff68279d8 in icinga::JsonRpcConnection::ProcessMessage (this=0x7fffc8002c50) at ../remote/jsonrpcconnection.cpp:173

afunc = {px = 0x62fda0}
message = {px = 0x7fffcc031b40}
origin = {px = 0x7fffcc021740}

PRETTY_FUNCTION = "bool icinga::JsonRpcConnection::ProcessMessage()"
srs =
method = {static NPos = 18446744073709551615, m_Data = "config::UpdateObject"}
resultMessage = {px = 0x7fffcc022b60}

#5  0x00007ffff6843e68 in icinga::JsonRpcConnection::DataAvailableHandler (this=0x7fffc8002c50) at ../remote/jsonrpcconnection.cpp:197

lock = {m = 0x7fffc8002cc8, is_locked = true}

#6  0x00007ffff71f00ab in operator() (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/function/function_template.hpp:767
No locals.

#7  m_invoke&)>, 0u, const boost::intrusive_ptr&> (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89
No locals.

#8  operator()&)>, const boost::intrusive_ptr&> (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78
resolver = 0x0

#9  m_invoke >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114
No locals.

#10 operator() >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107
resolver = 0x0

#11 dereference (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82
No locals.

#12 dereference&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514
No locals.

#13 operator* (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639
No locals.

#14 operator()&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55
No locals.

#15 operator(), boost::signals2::detail::slot_call_iterator_t&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (this=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64
No locals.

#16 boost::signals2::detail::signal_impl&), boost::signals2::optional_last_value, int, std::less, boost::function&)>, boost::function&)>, boost::signals2::mutex>::operator()(const boost::intrusive_ptr &) (this=Unhandled dwarf expression opcode 0xf3

)
at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246
local_state = {px = , pn = {pi_ = 0x7fffc80014a0}}
janitor = {_cache = 0x7ffff04390c0, _sig = 0x7fffc8001370, _connection_bodies = 0x7fffc80013f0}
it = Traceback (most recent call last):
File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 99, in to_string
itype = self.val.type.template_argument(0)
RuntimeError: syntax error, near `const boost::intrusive_ptr<icinga::St'

    invoker = <error reading variable invoker (Unhandled dwarf expression opcode 0xf3)>

cache = {result = {<boost::optional_detail::optional_baseboost::signals2::detail::void_type> = {boost::optional_detail::optional_tag = {}, m_initialized = false, m_storage = {dummy_ = {data = "\003",
aligner_ = 3 '\003'}}}, },
tracked_ptrs = {<std::allocator<boost::variant<boost::shared_ptr, boost::signals2::detail::foreign_void_shared_ptr, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*> >> = {<_gnu_cxx::new_allocator<boost::variant<boost::shared_ptr, boost::signals2::detail::foreign_void_shared_ptr, boost::detail::variant::void, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void*, boost::detail::variant::void_> >> = {}, },
static is_stack_buffer_empty = , members_ = {<boost::aligned_storage<240ul, 8ul>> = {<boost::detail::aligned_storage::aligned_storage_imp<240ul, 8ul>> = {data_ = {
buf = "\340\b\000\314\377\177\000\000\220\225C\360\377\177\000\000\364\341/\366\377\177\000\000\000\000\000\000\000\000\326\031\027\003\001", '\000' <repeats 13 times>, "E\000\000\000\000\000\000\000\000\000\000\000\000\000\020^{\000\260\027\000\000\000\020\"\000\310\377\177\000\000\240\221C\360\377\177\000\000\000\000\000\000\000\000\000\000P\001\000\000\000\000\000\000\200b,\366\377\177\000\000\060#\000\310\377\177\000\000\031\000\000\000\000\000\000\000`\221C\360\377\177\000\000\000\000\000\000\000\002\000\000\220\225C\360\377\177\000\000\000\000\000\000\000\000\000\000\220\225C\360\377\177\000\000\071.\005\314\377\177\000\000\323E\353X08\023i\210%4\364\377\177\000\000\020}\000\260\377\177\000\000\200b\000\260\377\177\000\000\200b\000\260\377\177\000\000\027\000\000\000\000\000\000\000X\203\033\367\377\177\000\000\000\000\000\000\000\000\000", align_ = {}}}, static size = ,
static alignment = }, capacity_ = 10}, buffer_ = 0x7ffff04390c8, size_ = 0}, f = {_args = std::tuple containing = {[1] = @0x7ffff0439230}}, connected_slot_count = 1, disconnected_slot_count = 0}

#17 0x00007ffff71a0a17 in operator() (this=0x7fffc80011b0) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695

No locals.

#18 icinga::Stream::SignalDataAvailable (this=0x7fffc80011b0) at ../base/stream.cpp:55

No locals.

#19 0x00007ffff71a0d00 in icinga::TlsStream::OnEvent (this=0x7fffc80011b0, revents=Unhandled dwarf expression opcode 0xf3

) at ../base/tlsstream.cpp:197
err =
count =
buffer = "me = 1449882000.000000\\n\\ttriggered_by = \\\"\\\"\\n\\tversion = 1449795613.608484\\n}\\n\",\modified_attributes\.
rc =
lock = {m = 0x7fffc80012a8, is_locked = false}

#20 0x00007ffff719daca in icinga::SocketEvents::ThreadProc () at ../base/socketevents.cpp:147

desc = {Events = , EventInterface = , LifesupportObject = 0x7fffc80011b0}
ltref = {px = 0x7fffc80011b0}
i =
pfds = 0x7fffcc01cd00
pfdcount =

#21 0x00007ffff7bd25c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0

No symbol table info available.

#22 0x00007ffff4340a51 in start_thread () from /lib64/libpthread.so.0

No symbol table info available.

#23 0x00007ffff484693d in clone () from /lib64/libc.so.6

No symbol table info available.

icinga-migration · 2015-12-16T06:57:02Z

Updated by elimat on 2015-12-16 06:57:02 +00:00

Hi,

I just wanted to inform I have similar issue with icinga2 crashing with below segfault error:

Dec 15 17:13:05 prod-icinga01 kernel: icinga2[3260]: segfault at 0 ip 00002adb1291b892 sp 00002adb4c801720 error 4 in libremote.so[2adb12828000+191000]
Dec 15 17:13:14 prod-icinga01 systemd: icinga2.service: main process exited, code=killed, status=11/SEGV
Dec 15 17:13:14 prod-icinga01 systemd: Unit icinga2.service entered failed state.
Dec 15 17:13:14 prod-icinga01 systemd: icinga2.service failed.

I started to get this after trying the icinga-dashing through API to icinga2. I suspect it have something to do with the API.

icinga-migration · 2015-12-22T11:44:33Z

Updated by tgelf on 2015-12-22 11:44:33 +00:00

Assigned to changed from mnardin to gbeutner

Hi Gunnar,

did the bt help, should a current snapshot fix this?

Thanks,
Thomas

icinga-migration · 2016-01-06T11:02:23Z

Updated by viniciusfs on 2016-01-06 11:02:23 +00:00

elimat wrote:

Hi,

I just wanted to inform I have similar issue with icinga2 crashing with below segfault error:

Dec 15 17:13:05 prod-icinga01 kernel: icinga2[3260]: segfault at 0 ip 00002adb1291b892 sp 00002adb4c801720 error 4 in libremote.so[2adb12828000+191000]
Dec 15 17:13:14 prod-icinga01 systemd: icinga2.service: main process exited, code=killed, status=11/SEGV
Dec 15 17:13:14 prod-icinga01 systemd: Unit icinga2.service entered failed state.
Dec 15 17:13:14 prod-icinga01 systemd: icinga2.service failed.

I started to get this after trying the icinga-dashing through API to icinga2. I suspect it have something to do with the API.

Same behaviour here after trying icinga-dashing.

icinga-migration · 2016-01-06T17:22:34Z

Updated by viniciusfs on 2016-01-06 17:22:34 +00:00

icinga2[29444]: segfault at 0 ip 00007f33f97dde63 sp 00007f33ddc816f0 error 4 in libremote.so[7f33f96ec000+18c000]
icinga2[19593]: segfault at 0 ip 00007fa0d22f2e63 sp 00007fa0b677c6f0 error 4 in libremote.so[7fa0d2201000+18c000]

icinga-migration · 2016-02-09T08:13:25Z

Updated by gbeutner on 2016-02-09 08:13:25 +00:00

Status changed from Feedback to Rejected

This is most likely the same bug like #11006.

icinga-migration · 2016-02-09T08:13:34Z

Updated by gbeutner on 2016-02-09 08:13:34 +00:00

Duplicated set to 11006

icinga-migration closed this as completed Feb 9, 2016

icinga-migration added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) labels Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #10783] segfault in libremote.so #3751

[dev.icinga.com #10783] segfault in libremote.so #3751

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 8, 2015

icinga-migration commented Dec 10, 2015

icinga-migration commented Dec 11, 2015

icinga-migration commented Dec 16, 2015

icinga-migration commented Dec 22, 2015

icinga-migration commented Jan 6, 2016

icinga-migration commented Jan 6, 2016

icinga-migration commented Feb 9, 2016

icinga-migration commented Feb 9, 2016

[dev.icinga.com #10783] segfault in libremote.so #3751

[dev.icinga.com #10783] segfault in libremote.so #3751

Comments

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 7, 2015

icinga-migration commented Dec 8, 2015

icinga-migration commented Dec 10, 2015

icinga-migration commented Dec 11, 2015

icinga-migration commented Dec 16, 2015

icinga-migration commented Dec 22, 2015

icinga-migration commented Jan 6, 2016

icinga-migration commented Jan 6, 2016

icinga-migration commented Feb 9, 2016

icinga-migration commented Feb 9, 2016