Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #13151] Crash w/ SendNotifications cluster handler and check result with empty perfdata #4775

Closed
icinga-migration opened this issue Nov 10, 2016 · 5 comments
Closed

[dev.icinga.com #13151] Crash w/ SendNotifications cluster handler and check result with empty perfdata #4775

icinga-migration opened this issue Nov 10, 2016 · 5 comments
Labels
area/distributed Distributed monitoring (master, satellites, clients) bug Something isn't working
Milestone
2.6.0

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13151

Created by mfriedrich on 2016-11-10 18:20:09 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-11-10 19:03:56 +00:00)
Target Version: 2.6.0
Last Update: 2016-11-10 19:03:56 +00:00 (in Redmine)

Icinga Version: 2.5.4 / 5dd4898eb353fa7751543e2af4647b66818eb901
Backport?: Not yet backported
Include in Changelog: 1

Probably the memory problem we're hunting in git master.

[2016-11-10 18:36:08 +0100] information/ConfigItem: Committing config item(s).
[2016-11-10 18:36:08 +0100] information/ApiListener: My API identity: icinga2a
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 3 Endpoints.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 3 Zones.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 61 CheckCommands.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1019 Services.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1019 Downtimes.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 32 Hosts.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 HostGroup.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2016-11-10 18:36:09 +0100] information/ScriptGlobal: Dumping variables to file './icinga2a/cache/icinga2/icinga2.vars'
[2016-11-10 18:36:09 +0100] information/ConfigObject: Restoring program state from file './icinga2a/lib/icinga2/icinga2.state'
[2016-11-10 18:36:09 +0100] information/ConfigObject: Restored 1699 objects. Loaded 447 new objects without state.
[2016-11-10 18:36:09 +0100] information/ConfigItem: Triggering Start signal for config items
[2016-11-10 18:36:09 +0100] information/ApiListener: Copying 1 zone configuration files for zone 'global-templates' to './icinga2a/lib/icinga2/api/zones/global-templates'.
[2016-11-10 18:36:09 +0100] information/ApiListener: Copying 7 zone configuration files for zone 'master' to './icinga2a/lib/icinga2/api/zones/master'.
[2016-11-10 18:36:09 +0100] information/ApiListener: Copying 1 zone configuration files for zone 'satellite' to './icinga2a/lib/icinga2/api/zones/satellite'.
[2016-11-10 18:36:09 +0100] information/ApiListener: Adding new listener on port '7000'
[2016-11-10 18:36:10 +0100] information/ConfigItem: Activated all objects.
[2016-11-10 18:36:10 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2b' via host 'localhost' and port '8000'
[2016-11-10 18:36:10 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2016-11-10 18:36:10 +0100] information/ApiListener: New client connection for identity 'icinga2b' to [127.0.0.1]:8000
[2016-11-10 18:36:10 +0100] information/ApiListener: Sending config updates for endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing configuration files for zone 'satellite' to endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: New client connection for identity 'icinga2c' to [127.0.0.1]:9000
[2016-11-10 18:36:10 +0100] information/ApiListener: Sending config updates for endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing configuration files for zone 'satellite' to endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Finished sending config updates for endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Sending replay log for endpoint 'icinga2c'.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Discarding 'config update object' message from 'icinga2c'.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Discarding 'config update object' message from 'icinga2c'.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Discarding 'config update object' message from 'icinga2c'.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Ignoring config update for zone 'global-templates' because we have an authoritative version of the zone's config.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.
[2016-11-10 18:36:10 +0100] warning/ApiListener: Ignoring config update for zone 'satellite' because we have an authoritative version of the zone's config.
[2016-11-10 18:36:10 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Finished sending config updates for endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Sending replay log for endpoint 'icinga2b'.
[2016-11-10 18:36:10 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga2b'.
[2016-11-10 18:36:13 +0100] information/ApiListener: Replayed 546 messages.
[2016-11-10 18:36:13 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga2c'.
Assertion failed: (px != 0), function operator->, file /usr/local/include/boost/smart_ptr/intrusive_ptr.hpp, line 174.
Process 26157 stopped
* thread #8: tid = 0x338269, 0x00007fffdb48bdda libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGABRT
    frame #0: 0x00007fffdb48bdda libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fffdb48bdda <+10>: jae    0x7fffdb48bde4            ; <+20>
    0x7fffdb48bddc <+12>: movq   %rax, %rdi
    0x7fffdb48bddf <+15>: jmp    0x7fffdb484d6f            ; cerror_nocancel
    0x7fffdb48bde4 <+20>: retq
(lldb) bt
* thread #8: tid = 0x338269, 0x00007fffdb48bdda libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGABRT
  * frame #0: 0x00007fffdb48bdda libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fffdb577787 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fffdb3f1420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fffdb3b8893 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x00000001040064c9 libicinga.2.5.4.dylib`boost::intrusive_ptr::operator->(this=0x000070000291d2b8) const + 73 at intrusive_ptr.hpp:174
    frame #5: 0x0000000104157fb2 libicinga.2.5.4.dylib`icinga::ClusterEvents::SendNotificationsAPIHandler(origin=0x000070000291e008, params=0x000070000291ddd8) + 2546 at clusterevents.cpp:840
    frame #6: 0x000000010415c46c libicinga.2.5.4.dylib`boost::detail::function::function_invoker2 const&, boost::intrusive_ptr const&), icinga::Value, boost::intrusive_ptr const&, boost::intrusive_ptr const&>::invoke(function_ptr=0x0000000101d17cd8, a0=0x000070000291e008, a1=0x000070000291ddd8) + 92 at function_template.hpp:101
    frame #7: 0x00000001006b72af libremote.2.5.4.dylib`boost::function2 const&, boost::intrusive_ptr const&>::operator(this=0x0000000101d17cd0, a0=0x000070000291e008, a1=0x000070000291ddd8)(boost::intrusive_ptr const&, boost::intrusive_ptr const&) const + 191 at function_template.hpp:770
    frame #8: 0x00000001006b71db libremote.2.5.4.dylib`icinga::ApiFunction::Invoke(this=0x0000000101d17cb0, origin=0x000070000291e008, arguments=0x000070000291ddd8) + 59 at apifunction.cpp:31
    frame #9: 0x000000010083f5e8 libremote.2.5.4.dylib`icinga::JsonRpcConnection::MessageHandler(this=0x00000001061d9f50, jsonString=0x0000000106873f58) + 3096 at jsonrpcconnection.cpp:202
    frame #10: 0x000000010083e7e8 libremote.2.5.4.dylib`icinga::JsonRpcConnection::MessageHandlerWrapper(this=0x00000001061d9f50, jsonString=0x0000000106873f58) + 88 at jsonrpcconnection.cpp:148
    frame #11: 0x0000000100843113 libremote.2.5.4.dylib`void boost::_mfi::mf1::call, icinga::String const>(this=0x0000000106873f40, u=0x0000000106873f50, (null)=0x0000000000000000, b1=0x0000000106873f58) const + 131 at mem_fn_template.hpp:156
    frame #12: 0x0000000100843081 libremote.2.5.4.dylib`void boost::_mfi::mf1::operator(this=0x0000000106873f40, u=0x0000000106873f50, a1=0x0000000106873f58) >(boost::intrusive_ptr&, icinga::String const&) const + 49 at mem_fn_template.hpp:171
    frame #13: 0x000000010084301c libremote.2.5.4.dylib`void boost::_bi::list2 >, boost::_bi::value >::operator(this=0x0000000106873f50, (null)=type @ 0x000070000291e598, f=0x0000000106873f40, a=0x000070000291e5c0, (null)=0), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) + 108 at bind.hpp:319
    frame #14: 0x0000000100842faa libremote.2.5.4.dylib`boost::_bi::bind_t, boost::_bi::list2 >, boost::_bi::value > >::operator(this=0x0000000106873f40)() + 58 at bind.hpp:1222
    frame #15: 0x0000000100842d10 libremote.2.5.4.dylib`boost::detail::function::void_function_obj_invoker0, boost::_bi::list2 >, boost::_bi::value > >, void>::invoke(function_obj_ptr=0x000070000291e7d8) + 32 at function_template.hpp:159
    frame #16: 0x0000000100f80678 libbase.2.5.4.dylib`boost::function0::operator(this=0x000070000291e7d0)() const + 120 at function_template.hpp:770
    frame #17: 0x00000001011c5674 libbase.2.5.4.dylib`icinga::WorkQueue::WorkerThreadProc(this=0x0000000102005a08) + 3364 at workqueue.cpp:233
    frame #18: 0x00000001011c81c9 libbase.2.5.4.dylib`boost::_mfi::mf0::operator(this=0x00000001069db640, p=0x0000000102005a08)(icinga::WorkQueue*) const + 105 at mem_fn_template.hpp:49
    frame #19: 0x00000001011c8b10 libbase.2.5.4.dylib`void boost::_bi::list1 >::operator(this=0x00000001069db650, (null)=type @ 0x000070000291ee48, f=0x00000001069db640, a=0x000070000291ee70, (null)=0), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) + 80 at bind.hpp:259
    frame #20: 0x00000001011c8aba libbase.2.5.4.dylib`boost::_bi::bind_t, boost::_bi::list1 > >::operator(this=0x00000001069db640)() + 58 at bind.hpp:1222
    frame #21: 0x00000001011c8a4c libbase.2.5.4.dylib`boost::detail::thread_data, boost::_bi::list1 > > >::run(this=0x00000001069db440) + 28 at thread.hpp:116
    frame #22: 0x0000000100109834 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 164
    frame #23: 0x00007fffdb574aab libsystem_pthread.dylib`_pthread_body + 180
    frame #24: 0x00007fffdb5749f7 libsystem_pthread.dylib`_pthread_start + 286
    frame #25: 0x00007fffdb574221 libsystem_pthread.dylib`thread_start + 13
(lldb) up
frame #1: 0x00007fffdb577787 libsystem_pthread.dylib`pthread_kill + 90
libsystem_pthread.dylib`pthread_kill:
    0x7fffdb577787 <+90>:  cmpl   $-0x1, %eax
    0x7fffdb57778a <+93>:  jne    0x7fffdb577793            ; <+102>
    0x7fffdb57778c <+95>:  callq  0x7fffdb5791fa            ; symbol stub for: __error
    0x7fffdb577791 <+100>: movl   (%rax), %eax
(lldb)
frame #2: 0x00007fffdb3f1420 libsystem_c.dylib`abort + 129
libsystem_c.dylib`abort:
    0x7fffdb3f1420 <+129>: movl   $0x2710, %edi             ; imm = 0x2710
    0x7fffdb3f1425 <+134>: callq  0x7fffdb3c2e68            ; usleep$NOCANCEL
    0x7fffdb3f142a <+139>: callq  0x7fffdb3f142f            ; __abort

libsystem_c.dylib`__abort:
    0x7fffdb3f142f <+0>:   cmpq   $0x0, 0x8c6dcf1(%rip)     ; gCRAnnotations + 7
(lldb)
frame #3: 0x00007fffdb3b8893 libsystem_c.dylib`__assert_rtn + 320
libsystem_c.dylib`basename_r:
    0x7fffdb3b8893 <+0>: pushq  %rbp
    0x7fffdb3b8894 <+1>: movq   %rsp, %rbp
    0x7fffdb3b8897 <+4>: pushq  %r15
    0x7fffdb3b8899 <+6>: pushq  %r14
(lldb)
frame #4: 0x00000001040064c9 libicinga.2.5.4.dylib`boost::intrusive_ptr::operator->(this=0x000070000291d2b8) const + 73 at intrusive_ptr.hpp:174
   171
   172      T * operator->() const
   173      {
-> 174          BOOST_ASSERT( px != 0 );
   175          return px;
   176      }
   177
(lldb)
frame #5: 0x0000000104157fb2 libicinga.2.5.4.dylib`icinga::ClusterEvents::SendNotificationsAPIHandler(origin=0x000070000291e008, params=0x000070000291ddd8) + 2546 at clusterevents.cpp:840
   837      if (params->Contains("cr")) {
   838          cr = new CheckResult();
   839          Dictionary::Ptr vcr = params->Get("cr");
-> 840          Array::Ptr vperf = vcr->Get("performance_data");
   841          vcr->Remove("performance_data");
   842          Deserialize(cr, vcr, true);
   843      }

Changesets

2016-11-10 19:00:38 +00:00 by mfriedrich 8ae172e

Fix crash in CheckResult cluster handlers

fixes #13151

2016-11-11 15:29:51 +00:00 by mfriedrich e7a6124

Fix crash in CheckResult cluster handlers

fixes #13151

2016-11-17 12:51:02 +00:00 by mfriedrich 67ebcc2

Fix crash in CheckResult cluster handlers

fixes #13151

Relations:
@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-10 18:59:08 +00:00

  • Relates set to 12718

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-10 19:03:56 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 8ae172e.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-16 15:35:50 +00:00

  • Relates set to 12861

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-11-18 14:04:43 +00:00

  • Relates set to 12677

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-12-05 09:56:14 +00:00

  • Relates set to 13371

@icinga-migration icinga-migration added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.6.0 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/distributed Distributed monitoring (master, satellites, clients) bug Something isn't working
Projects
None yet
Milestone
2.6.0
Development

No branches or pull requests
1 participant
@icinga-migration