[dev.icinga.com #12022] Icinga crashes when using include_recursive in an object definition

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12022

Created by bebehei on 2016-06-22 13:57:56 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2016-06-22 15:04:14 +00:00)
Target Version: 2.5.0
Last Update: 2016-06-22 16:31:48 +00:00 (in Redmine)

Icinga Version: r2.4.10-1
Backport?: Not yet backported
Include in Changelog: 1

---

Hi guys,

I tested a configuration change at our system with an include_recursive. It turns out to make icinga2 crash with SIGABRT.

Steps to reproduce are simple:

1. Ubuntu 16.04 with plain icinga2 from formorer ppa
2. Inside conf.d-directory, you create a new host with an include_recursive inside.
3. Create under given include_recursive directory a configuration-file, which matches the glob and it will crash.

root@mon:/etc/icinga2/conf.d/hosts# cat ubuntu.conf 
object Host "ubuntu" {
        import "satellite-host"

        address = "123.45.67.89"
        vars.os = "Linux"
        vars.sla = "24x7"

        vars.remote_client = "ubuntu"

        check_command = "ping"

        # 
        include_recursive "/srv/icinga2/hosts/ubuntu", "*.cfg"
}

root@mon:/srv/icinga2/hosts/ubuntu# cat drive-vda.cfg 
        vars.disks["disk vda"] = {
                disk = "/dev/vda"
        }

It will not crash, if there is no file inside the include_recursive from the host. In my case: If there is no file inside /srv/icinga2/hosts/ubuntu, icinga won't fail.

If this is not supported, it would be great, icinga would show me this without a SIGABRT.

Answering the question Why are you supposed to use icinga2 this way? I want to generate the icinga2 host information automated via saltstack. I thought putting the single bits in serperate files might prevent some special cases, where if e.g. a harddrive dies and is not recognized by the system anymore and I regenerate the configuration without seperate files, icinga would not miss the harddrive anymore.

Cheers,
Bene

Attachments

• report.1466601943.763545 bebehei - 2016-06-22 13:30:25 +00:00 - crash report
• failure-log.txt bebehei - 2016-06-22 13:38:10 +00:00 - output of icinga2 daemon --validate (including stracktrace)

Changesets

2016-06-22 15:01:36 +00:00 by gbeutner 906ab61

Fix crash when using include_recursive in an object definition

fixes #12022

[icinga-migration]

Updated by mfriedrich on 2016-06-22 14:07:49 +00:00

• Category set to Configuration
• Status changed from New to Feedback
• Assigned to set to bebehei

Ewww.

Could you install gdb in order to let icinga2 generate a full crash dump?

[icinga-migration]

Updated by bebehei on 2016-06-22 14:18:50 +00:00

dnsmichi wrote:

Ewww.

Could you install gdb in order to let icinga2 generate a full crash dump?

Of course I could. Could you please provide some information, how to do it? I'm not that experienced with gdb. I only find examples, which want to use gcore with the specific PID. But icinga2 crashes on startup.

BTW: Have you seen the stacktrace in failure-log.txt?

[icinga-migration]

Updated by mfriedrich on 2016-06-22 14:22:24 +00:00

http://docs.icinga.org/icinga2/latest/doc/module/icinga2/toc\#!/icinga2/latest/doc/module/icinga2/chapter/development#debug-requirements

Once the crash again happens, icinga2 will invoke gdb in order to generate such automatically. If you prefer to run icinga2 yourself inside gdb, there's also information on the docs for that.

[icinga-migration]

Updated by gbeutner on 2016-06-22 14:31:41 +00:00

(lldb) bt
* thread #8: tid = 0x6befac, 0x00007fff934e2f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGABRT
  * frame #0: 0x00007fff934e2f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff835594ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff8fbdd6e7 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fff811c0041 libsystem_malloc.dylib`free + 425
    frame #4: 0x0000000100daa570 libconfig.dylib`icinga::IndexerExpression::GetReference(this=0x00000001043c9e90, frame=0x0000700000498458, init_dict=true, parent=0x0000700000495868, index=0x00007000004953e8, dhint=0x0000700000495400) const + 1712 at expression.cpp:675
    frame #5: 0x0000000100da881c libconfig.dylib`icinga::SetExpression::DoEvaluate(this=0x00000001043bf490, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 396 at expression.cpp:524
    frame #6: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x00000001043bf490, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #7: 0x0000000100da811d libconfig.dylib`icinga::DictExpression::DoEvaluate(this=0x00000001043c8450, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 973 at expression.cpp:484
    frame #8: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x00000001043c8450, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #9: 0x0000000100da811d libconfig.dylib`icinga::DictExpression::DoEvaluate(this=0x00000001043b1bb0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 973 at expression.cpp:484
    frame #10: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x00000001043b1bb0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #11: 0x0000000100dad1fd libconfig.dylib`icinga::IncludeExpression::DoEvaluate(this=0x000000010462a890, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 3149 at expression.cpp:876
    frame #12: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x000000010462a890, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #13: 0x0000000100da811d libconfig.dylib`icinga::DictExpression::DoEvaluate(this=0x000000010462a970, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 973 at expression.cpp:484
    frame #14: 0x0000000100db58bb libconfig.dylib`icinga::OwnedExpression::DoEvaluate(this=0x000000010462bc10, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 91 at expression.hpp:219
    frame #15: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x000000010462bc10, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #16: 0x0000000100da811d libconfig.dylib`icinga::DictExpression::DoEvaluate(this=0x00000001046315f0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 973 at expression.cpp:484
    frame #17: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x00000001046315f0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #18: 0x0000000100da811d libconfig.dylib`icinga::DictExpression::DoEvaluate(this=0x00000001046316b0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 973 at expression.cpp:484
    frame #19: 0x0000000100da195a libconfig.dylib`icinga::Expression::Evaluate(this=0x00000001046316b0, frame=0x0000700000498458, dhint=0x0000700000497d78) const + 138 at expression.cpp:65
    frame #20: 0x0000000100d4125a libconfig.dylib`icinga::ConfigItem::Commit(this=0x0000000104631730, discard=false) + 2186 at configitem.cpp:191
    frame #21: 0x0000000100d5d73d libconfig.dylib`boost::intrusive_ptr boost::_mfi::mf1, icinga::ConfigItem, bool>::call, bool>(this=0x00000001043b9840, u=0x00000001043b9850, (null)=0x0000000000000000, b1=0x000070000049851f) const + 157 at mem_fn_template.hpp:156
    frame #22: 0x0000000100d5d651 libconfig.dylib`boost::intrusive_ptr boost::_mfi::mf1, icinga::ConfigItem, bool>::operator(this=0x00000001043b9840, u=0x00000001043b9850, a1=false) >(boost::intrusive_ptr&, bool) const + 65 at mem_fn_template.hpp:171
    frame #23: 0x0000000100d5d5c8 libconfig.dylib`boost::intrusive_ptr boost::_bi::list2 >, boost::_bi::value >::operator(this=0x00000001043b9850, (null)=type > @ 0x0000700000498588, f=0x00000001043b9840, a=0x00007000004985c0, (null)=0), boost::_mfi::mf1, icinga::ConfigItem, bool>, boost::_bi::list0>(boost::_bi::type >, boost::_mfi::mf1, icinga::ConfigItem, bool>&, boost::_bi::list0&, long) + 136 at bind.hpp:305
    frame #24: 0x0000000100d5d50c libconfig.dylib`boost::_bi::bind_t, boost::_mfi::mf1, icinga::ConfigItem, bool>, boost::_bi::list2 >, boost::_bi::value > >::operator(this=0x00000001043b9840)() + 76 at bind.hpp:895
    frame #25: 0x0000000100d5d257 libconfig.dylib`boost::detail::function::void_function_obj_invoker0, boost::_mfi::mf1, icinga::ConfigItem, bool>, boost::_bi::list2 >, boost::_bi::value > >, void>::invoke(function_obj_ptr=0x00007000004987e8) + 39 at function_template.hpp:159
    frame #26: 0x0000000100faf0e8 libbase.dylib`boost::function0::operator(this=0x00007000004987e0)() const + 120 at function_template.hpp:770
    frame #27: 0x00000001011bbfa4 libbase.dylib`icinga::WorkQueue::WorkerThreadProc(this=0x00007fff5fbf9ee0) + 3364 at workqueue.cpp:228
    frame #28: 0x00000001011bef49 libbase.dylib`boost::_mfi::mf0::operator(this=0x000000010373b2d0, p=0x00007fff5fbf9ee0)(icinga::WorkQueue*) const + 105 at mem_fn_template.hpp:49
    frame #29: 0x00000001011bf890 libbase.dylib`void boost::_bi::list1 >::operator(this=0x000000010373b2e0, (null)=type @ 0x0000700000498e58, f=0x000000010373b2d0, a=0x0000700000498e80, (null)=0), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) + 80 at bind.hpp:255
    frame #30: 0x00000001011bf83a libbase.dylib`boost::_bi::bind_t, boost::_bi::list1 > >::operator(this=0x000000010373b2d0)() + 58 at bind.hpp:895
    frame #31: 0x00000001011bf7cc libbase.dylib`boost::detail::thread_data, boost::_bi::list1 > > >::run(this=0x000000010373b0d0) + 28 at thread.hpp:116
    frame #32: 0x0000000100082715 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 53
    frame #33: 0x00007fff8355699d libsystem_pthread.dylib`_pthread_body + 131
    frame #34: 0x00007fff8355691a libsystem_pthread.dylib`_pthread_start + 168
    frame #35: 0x00007fff83554351 libsystem_pthread.dylib`thread_start + 13
(lldb)

[icinga-migration]

Updated by bebehei on 2016-06-22 14:33:11 +00:00

dnsmichi wrote:

http://docs.icinga.org/icinga2/latest/doc/module/icinga2/toc\#!/icinga2/latest/doc/module/icinga2/chapter/development#debug-requirements

Once the crash again happens, icinga2 will invoke gdb in order to generate such automatically. If you prefer to run icinga2 yourself inside gdb, there's also information on the docs for that.

Generated a complete core. This file is about ~650MB big!?

http://dl.b3be.de/icinga2-coredump-bug12022.core

[icinga-migration]

Updated by gbeutner on 2016-06-22 14:53:57 +00:00

• Status changed from Feedback to Assigned
• Assigned to changed from bebehei to gbeutner

We don't need the core dump - in fact you should probably remove it because it contains a copy of icinga2's memory (including parts of your config). We just need a new report.* file after you've installed GDB.

However, I have already been able to reproduce this here locally as you can see from the stacktrace I posted earlier.

[icinga-migration]

Updated by gbeutner on 2016-06-22 15:04:10 +00:00

• Target Version set to 2.5.0

[icinga-migration]

Updated by gbeutner on 2016-06-22 15:04:14 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset 906ab61.

[icinga-migration]

Updated by gbeutner on 2016-06-22 15:04:55 +00:00

• Subject changed from icinga2 fails with SIGABRT to Icinga crashes when using include_recursive in an object definition

[icinga-migration]

Updated by bebehei on 2016-06-22 16:31:48 +00:00

gunnarbeutner wrote:

Applied in changeset 906ab61.

Thanks, it works like a charm.