[dev.icinga.com #10948] Icinga state file corruption with temporary file creation

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10948

Created by vytenis on 2016-01-11 12:21:01 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-02-23 11:35:03 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 14:47:01 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 1

---

Icinga2 v2.4.1-90-g95e62df corrupts statefile if regenerating configuration quite often. Seems like it always gets stuck on objects with very long dependency object names, eg `appdocker37-sjc1dep-cluster_appdocker_team_udeploy-sjc1-udeploy-config-validation-check-slave`, `schemalessdb615-sjc1depends-on-check-results`

Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: [2016-01-11 11:55:06 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: [2016-01-11 11:55:06 +0000] information/ConfigObject: Restoring program state from file '/var/lib/icinga2/icinga2.state'
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: [2016-01-11 11:55:10 +0000] critical/cli: Failed to restore state file: Error: Invalid NetString (missing ,)
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (0) libbase.so: void boost::throw_exception >(boost::exception_detail::error_info_injector const&) (+0x104) [0x7f764bb37df4]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (1) libbase.so: void boost::exception_detail::throw_exception_(std::invalid_argument const&, char const*, char const*, int) (+0x67) [0x7f764bb37ea7]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (2) libbase.so: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr const&, icinga::String*, icinga::StreamReadContext&, bool) (+0x304) [0x7f764baebf94]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (3) libbase.so: icinga::ConfigObject::RestoreObjects(icinga::String const&, int) (+0x20d) [0x7f764bb1049d]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (4) libcli.so: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector > const&) const (+0x687) [0x7f764b492e67]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (5) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x412d1b]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (6) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x40f6da]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (7) libc.so.6: __libc_start_main (+0xf5) [0x7f76491f1b45]
Jan 11 11:55:10 icinga03-sjc1 icinga2[125096]: (8) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x40f7e0]
Jan 11 11:55:10 icinga03-sjc1 systemd[1]: icinga2.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:55:10 icinga03-sjc1 systemd[1]: Unit icinga2.service entered failed state.

(gdb) set print elements 1024
(gdb) bt full

#0  __cxa_throw (obj=0x1cb5080, pvtinfo=0x43f920 <_ZTIN5boost16exception_detail10clone_implINS0_19error_info_injectorISt16invalid_argumentEEEE>,

dest=0x7ffff6e9850a <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorstd::invalid_argument >::~clone_impl()>)
at ../../../lib/base/exception.cpp:65
real_cxa_throw = 0x7ffff45badb0 <__cxa_throw>
user_exc = 0x7ffff7fe6000
tinfo = 0x7ffff6e7c6b3 <boost::exception_detail::error_info_injectorstd::invalid_argument::~error_info_injector()+77>
thrown_ptr = 0x18009be
boost_exc = 0x1cb5080

#1  0x00007ffff6e7c88c in boost::throw_exception > (e=warning: can't find linker symbol for virtual table for `boost::exception_detail::error_info_injector' value

...) at /usr/include/boost/throw_exception.hpp:70
No locals.

#2  0x00007ffff6e61a2e in boost::exception_detail::throw_exception_ (x=...,

current_function=0x7ffff6f54980 <icinga::NetString::ReadStringFromStream(boost::intrusive_ptricinga::Stream const&, icinga::String**, icinga::StreamReadContext&, bool)::PRETTY_FUNCTION> "static icinga::StreamReadStatus icinga::NetString::ReadStringFromStream(const Ptr&, icinga::String**, icinga::StreamReadContext&, bool)",
file=0x7ffff6f39ee0 "../../../lib/base/netstring.cpp", line=95) at /usr/include/boost/throw_exception.hpp:87
No locals.

#3  0x00007ffff6e21032 in icinga::NetString::ReadStringFromStream (stream=..., str=0x7fffffffaa60, context=..., may_wait=false) at ../../../lib/base/netstring.cpp:95

PRETTY_FUNCTION = "static icinga::StreamReadStatus icinga::NetString::ReadStringFromStream(const Ptr&, icinga::String*, icinga::StreamReadContext&, bool)"
len = 0
i = 0
data_length = 1
data = 0x1558bb7c "0.0}},208:{\name\.
header_length = 11

#4  0x00007ffff6e118bd in icinga::ConfigObject::RestoreObjects (filename=..., attributeTypes=4) at ../../../lib/base/configobject.cpp:581

srs = icinga::StatusNewItem
fp =
sfp = {px = 0x1d34f90}
restored = 229203
message = {static NPos = 18446744073709551615,
m_Data = "{\name\
upq = {m_ID = 5, static m_NextID = 6, m_ThreadCount = 24, m_Spawned = true, m_Mutex = {m = {_data = {lock = 0,count = 0,owner = 0,nusers = 0,_kind = 0,
_spins = 0,elision = 0,list = {prev = 0x0,next = 0x0}},size = '\000' <repeats 39 times>,align = 0}}, m_CVEmpty = {internal_mutex = {_data = {
_lock = 0,count = 0,owner = 0,nusers = 24,kind = 0,spins = 0,elision = 0,list = {prev = 0x0,_next = 0x0}},
_size = '\000' <repeats 12 times>, "\030", '\000' <repeats 26 times>,align = 0}, cond = {data = {lock = 0,futex = 257432,_total_seq = 128728,
_wakeup_seq = 128704,woken_seq = 128704,mutex = 0x7fffffffa728,nwaiters = 48,_broadcast_seq = 0},
_size = "\000\000\000\000\230\355\003\000\330\366\001\000\000\000\000\000\300\366\001\000\000\000\000\000\300\366\001\000\000\000\000\000(\247\377\377\377\177\000\000\060\000\000\000\000\000\000",align = 1105662020943872}}, m_CVFull = {internal_mutex = {data = {lock = 0,count = 0,owner = 0,nusers = 0,kind = 0,_spins = 0,
_elision = 0,list = {prev = 0x0,next = 0x0}},size = '\000' <repeats 39 times>,align = 0}, cond = {data = {lock = 0,futex = 0,_total_seq = 0,
_wakeup_seq = 0,woken_seq = 0,mutex = 0x0,nwaiters = 0,broadcast_seq = 0},size = '\000' <repeats 47 times>,_align = 0}}, m_CVStarved = {
internal_mutex = {_data = {lock = 0,count = 0,owner = 0,nusers = 0,kind = 0,spins = 0,elision = 0,list = {prev = 0x0,_next = 0x0}},
_size = '\000' <repeats 39 times>,align = 0}, cond = {data = {lock = 0,futex = 0,total_seq = 0,wakeup_seq = 0,woken_seq = 0,_mutex = 0x0,
_nwaiters = 0,broadcast_seq = 0},size = '\000' <repeats 47 times>,_align = 0}}, m_Threads = {threads = std::list = {[0] = 0x1dd0620, [1] = 0x1dcf7d0,
[2] = 0x1dce940, [3] = 0x1dcdaa0, [4] = 0x1dccc30, [5] = 0x1dcbd90, [6] = 0x1dcaf00, [7] = 0x1dca0b0, [8] = 0x1dc9270, [9] = 0x1dc83e0, [10] = 0x1dc7550,
[11] = 0x1dc66c0, [12] = 0x1dc5890, [13] = 0x1dc4a40, [14] = 0x1dc3c20, [15] = 0x1dc2e20, [16] = 0x1dc2010, [17] = 0x1dc1200, [18] = 0x1dc03d0, [19] = 0x1dbf560,
[20] = 0x1dbe730, [21] = 0x1dbd8b0, [22] = 0x1dbca50, [23] = 0x1dbbbc0}, m = {state = {shared_count = 0, exclusive = false, upgrade = false,
exclusive_waiting_blocked = false}, state_change = {m = {_data = {lock = 0,count = 0,owner = 0,nusers = 0,kind = 0,spins = 0,_elision = 0,
_list = {prev = 0x0,next = 0x0}},size = '\000' <repeats 39 times>,align = 0}}, shared_cond = {internal_mutex = {data = {lock = 0,_count = 0,
_owner = 0,nusers = 0,kind = 0,spins = 0,elision = 0,list = {prev = 0x0,next = 0x0}},size = '\000' <repeats 39 times>,_align = 0},
cond = {_data = {lock = 0,futex = 0,total_seq = 0,wakeup_seq = 0,woken_seq = 0,mutex = 0x0,nwaiters = 0,_broadcast_seq = 0},
_size = '\000' <repeats 47 times>,align = 0}}, exclusive_cond = {internal_mutex = {data = {lock = 0,count = 0,owner = 0,nusers = 0,_kind = 0,
_spins = 0,elision = 0,list = {prev = 0x0,next = 0x0}},size = '\000' <repeats 39 times>,align = 0}, cond = {data = {lock = 0,_futex = 0,
_total_seq = 0,wakeup_seq = 0,woken_seq = 0,mutex = 0x0,nwaiters = 0,broadcast_seq = 0},size = '\000' <repeats 47 times>,_align = 0}},
upgrade_cond = {internal_mutex = {_data = {lock = 0,count = 0,owner = 0,nusers = 0,kind = 0,spins = 0,elision = 0,list = {_prev = 0x0,
_next = 0x0}},size = '\000' <repeats 39 times>,align = 0}, cond = {data = {lock = 0,futex = 0,total_seq = 0,_wakeup_seq = 0,
_woken_seq = 0,mutex = 0x0,nwaiters = 0,broadcast_seq = 0},size = '\000' <repeats 47 times>,_align = 0}}}}, m_MaxItems = 25000,
m_Stopped = false, m_Processing = 0, m_Tasks = std::priority_queue wrapping: std::deque with 0 elements, m_NextTaskID = 229208,
m_ExceptionCallback = {<boost::function1<void, boost::exception_ptr>> = {boost::function_base = {vtable = 0x0, functor = {obj_ptr = 0x1, type = {type = 0x1,
const_qualified = false, volatile_qualified = 170}, func_ptr = 0x1, bound_memfunc_ptr = {memfunc_ptr = &virtual table offset 0, this adjustment 140737354050048,
obj_ptr = 0x7fffffffaa50}, obj_ref = {obj_ptr = 0x1, is_const_qualified = false, is_volatile_qualified = 170},
data = 1 '\001'}}, <std::unary_function<boost::exception_ptr, void>> = {}, static args = ,
static arity = }, }, m_Exceptions = std::vector of length 0, capacity 0, m_StatusTimer = {px = 0x694270}}
src = {
Buffer = 0x1558bb70 "\",\version\, Size = 231, MustRead = false, Eof = false}
no_state = 140737488333392

#5  0x00007ffff63db854 in icinga::DaemonCommand::Run (this=0x658370, vm=..., ap=std::vector of length 0, capacity 0) at ../../../lib/cli/daemoncommand.cpp:272

configs = std::vector of length 1, capacity 1 = {"/etc/icinga2/icinga2.conf"}
newItems = std::vector of length 229204, capacity 262144 = {{px = 0x764f80}, {px = 0x7779f0}, {px = 0x6a9710}, {px = 0x6aa980}, {px = 0x783e60}, {px = 0x783b00}, {
px = 0x784520}, {px = 0x7841c0}, {px = 0x6abfa0}, {px = 0x69b070}, {px = 0x69aee0}, {px = 0x6a9ac0}, {px = 0x6aa220}, {px = 0x6a2ce0}, {px = 0x6a3090}, {px = 0x6a9e70},
{px = 0x6987e0}, {px = 0x698100}, {px = 0x698430}, {px = 0x699e00}, {px = 0x783440}, {px = 0x7837a0}, {px = 0x6a53d0}, {px = 0x712140}, {px = 0x7124a0}, {px = 0x6ab490}, {
px = 0x699a50}, {px = 0x69f080}, {px = 0x6ab840}, {px = 0x762af0}, {px = 0x6a1310}, {px = 0x6abbf0}, {px = 0x6a8850}, {px = 0x6ac350}, {px = 0x75d7f0}, {px = 0x75dba0},
{px = 0x784880}, {px = 0x788510}, {px = 0x788c70}, {px = 0x7893d0}, {px = 0x787db0}, {px = 0x788160}, {px = 0x7888c0}, {px = 0x789020}, {px = 0x6ab0e0}, {px = 0x6aa5d0}, {
px = 0x6a2580}, {px = 0x6a21d0}, {px = 0x698b90}, {px = 0x75d440}, {px = 0x69da70}, {px = 0x69e010}, {px = 0x6a4dc0}, {px = 0x6a1a70}, {px = 0x6a37f0}, {px = 0x69a950},
{px = 0x6a16c0}, {px = 0x69a1b0}, {px = 0x6a8c00}, {px = 0x6a9360}, {px = 0x6a8fb0}, {px = 0x6a1e20}, {px = 0x6a2930}, {px = 0x6992f0}, {px = 0x6a0f60}, {px = 0x6a3ba0}, {
px = 0x698f40}, {px = 0x6996a0}, {px = 0x6aad30}, {px = 0x6a3440}, {px = 0x765cb0}, {px = 0x697860}, {px = 0x767aa0}, {px = 0x777400}, {px = 0x157b9380}, {
px = 0x15812df0}, {px = 0x157dff00}, {px = 0x157bf710}, {px = 0x157e5020}, {px = 0x157e6da0}, {px = 0x157b1270}, {px = 0x157ff480}, {px = 0x157b7d60}, {
px = 0x15807940}, {px = 0x1581c520}, {px = 0x1579fa30}, {px = 0x157fa710}, {px = 0x157a17b0}, {px = 0x157caf70}, {px = 0x157faac0}, {px = 0x157a3530}, {
px = 0x157fb220}, {px = 0x157fb5d0}, {px = 0x157a43f0}, {px = 0x1579b7d0}, {px = 0x157c1bf0}, {px = 0x157c1fa0}, {px = 0x15829750}, {px = 0x1582bc30}, {
px = 0x1582d600}, {px = 0x157a8650}, {px = 0x157daa30}, {px = 0x1582caf0}, {px = 0x15787010}, {px = 0x1579ca40}, {px = 0x1579cdf0}, {px = 0x157f6c10}, {
px = 0x1579d1a0}, {px = 0x1579d550}, {px = 0x1579d900}, {px = 0x1582e870}, {px = 0x1579e060}, {px = 0x1581a3f0}, {px = 0x1579e410}, {px = 0x157873c0}, {
px = 0x1582fae0}, {px = 0x1579eb70}, {px = 0x1579ef20}, {px = 0x1578ed70}, {px = 0x15790390}, {px = 0x158314b0}, {px = 0x1579f2d0}, {px = 0x157945f0}, {
px = 0x157b0b10}, {px = 0x1579f680}, {px = 0x157ba5f0}, {px = 0x15832ad0}, {px = 0x1579fde0}, {px = 0x157cabc0}, {px = 0x157a0190}, {px = 0x157d6f30}, {
px = 0x157a0540}, {px = 0x157a08f0}, {px = 0x157a0ca0}, {px = 0x157a1050}, {px = 0x157a1400}, {px = 0x15796300}, {px = 0x15782a00}, {px = 0x157838c0}, {
px = 0x157a1b60}, {px = 0x157a9160}, {px = 0x157a1f10}, {px = 0x157ab640}, {px = 0x157a22c0}, {px = 0x157843d0}, {px = 0x157b0ec0}, {px = 0x157b19d0}, {
px = 0x158254f0}, {px = 0x157b3b00}, {px = 0x157da2d0}, {px = 0x157da680}, {px = 0x157e6640}, {px = 0x157e69f0}, {px = 0x15825c50}, {px = 0x157e3a00}, {
px = 0x157e7150}, {px = 0x157e7500}, {px = 0x157e4c70}, {px = 0x157befb0}, {px = 0x157e78b0}, {px = 0x157e7c60}, {px = 0x157e8010}, {px = 0x157e83c0}, {
px = 0x157e8770}, {px = 0x157e5b30}, {px = 0x157e8b20}, {px = 0x157e8ed0}, {px = 0x157e9630}, {px = 0x15824280}, {px = 0x15826760}, {px = 0x157ef260}, {
px = 0x15803330}, {px = 0x157ef9c0}, {px = 0x157e9d90}, {px = 0x157f0120}, {px = 0x157f04d0}, {px = 0x157ea140}, {px = 0x157ea4f0}, {px = 0x157eac50}, {
px = 0x157eb000}, {px = 0x157bad50}, {px = 0x157eb3b0}, {px = 0x157eb760}, {px = 0x157ebb10}, {px = 0x157ebec0}, {px = 0x157ec270}, {px = 0x157ec620}, {
px = 0x157f3fd0}, {px = 0x157ec9d0}, {px = 0x157ecd80}, {px = 0x157fd350}, {px = 0x1579bb80}, {px = 0x157a8db0}, {px = 0x157a9510}, {px = 0x157fd700}, {
px = 0x157fdab0}, {px = 0x157a98c0}, {px = 0x157fde60}, {px = 0x157fe210}, {px = 0x157fe5c0}, {px = 0x157fe970}, {px = 0x157ff0d0}, {px = 0x157bb100}, {
px = 0x157ab290}, {px = 0x157ab9f0}, {px = 0x15830d50}, {px = 0x157ac8b0}, {px = 0x157aded0}, {px = 0x158096c0}, {px = 0x157a7790}, {px = 0x15807590}, {
px = 0x157dbca0}, {px = 0x1581a7a0}, {px = 0x157bb4b0}, {px = 0x158249e0}, {px = 0x157b49c0}, {px = 0x157b9730}, {px = 0x157c0980}, {px = 0x158233c0}, {
px = 0x157cee20}, {px = 0x157dcb60}, {px = 0x1578deb0}, {px = 0x157b24e0}, {px = 0x157822a0}, {px = 0x158284e0}, {px = 0x157c0220}, {px = 0x157c05d0}, {
px = 0x1580f6a0}, {px = 0x157eeb00}, {px = 0x15791250}, {px = 0x157d0440}, {px = 0x15781b40}, {px = 0x15828c40}, {px = 0x15803e40}, {px = 0x15795440}, {
px = 0x157bbc10}, {px = 0x158041f0}, {px = 0x157dc050}, {px = 0x158147c0}, {px = 0x15799a50}, {px = 0x157b4610}, {px = 0x15813900}, {px = 0x158066d0}, {
px = 0x157c70c0}, {px = 0x157af8a0}, {px = 0x157c6960}, {px = 0x157bbfc0}, {px = 0x158101b0}, {px = 0x15816540}, {px = 0x157c7f80}, {px = 0x157c91f0}, {
px = 0x15813cb0}, {px = 0x157fcfa0}, {px = 0x15811b80}, {px = 0x15790ea0}, {px = 0x15831c10}, {px = 0x1581edb0}, {px = 0x1582a9c0}, {px = 0x157ce310}, {
px = 0x157a5a10}, {px = 0x1580d570}, {px = 0x15812690}, {px = 0x1582bfe0}, {px = 0x157c7470}, {px = 0x157ae9e0}, {px = 0x157aa3d0}, {px = 0x157d7a40}, {
px = 0x15814f20}, {px = 0x157bc720}, {px = 0x15798f40}, {px = 0x15801d10}, {px = 0x15808450}, {px = 0x157c7bd0}, {px = 0x15810cc0}, {px = 0x157a9c70}, {
px = 0x15784020}, {px = 0x157b4260}, {px = 0x1579dcb0}, {px = 0x157c40d0}, {px = 0x157c3970}, {px = 0x157bb860}, {px = 0x157fc840}, {px = 0x157c86e0}, {
px = 0x157996a0}, {px = 0x157a6c80}, {px = 0x157e23e0}, {px = 0x157adb20}, {px = 0x157bcad0}, {px = 0x157e5780}, {px = 0x15820b30}, {px = 0x1580e080}, {
px = 0x15800340}, {px = 0x157dec90}, {px = 0x15794240}, {px = 0x157e32a0}, {px = 0x158279d0}, {px = 0x158006f0}, {px = 0x1582fe90}, {px = 0x15831860}, {
px = 0x157ef610}, {px = 0x157e1c80}, {px = 0x157a6170}, {px = 0x1579c2e0}, {px = 0x1578e610}, {px = 0x157992f0}, {px = 0x15791d60}, {px = 0x157f3870}, {
px = 0x158177b0}, {px = 0x1578e260}, {px = 0x15832e80}, {px = 0x157f6fc0}, {px = 0x1579a1b0}, {px = 0x1581c8d0}, {px = 0x157e2ef0}, {px = 0x1581bdc0}, {
px = 0x1578a000}, {px = 0x15814b70}, {px = 0x157f64b0}, {px = 0x157abda0}, {px = 0x158293a0}, {px = 0x157d37e0}, {px = 0x157a38e0}, {px = 0x157bd5e0}, {
px = 0x15785290}, {px = 0x157c2350}, {px = 0x15822150}, {px = 0x157a73e0}, {px = 0x15821640}, {px = 0x157a4b50}, {px = 0x1580e430}, {px = 0x158305f0}, {
px = 0x157d3b90}, {px = 0x157a4040}, {px = 0x157b7250}, {px = 0x15786150}, {px = 0x15793e90}, {px = 0x157e0660}, {px = 0x158131a0}, {px = 0x157ae630}, {
px = 0x1581ab50}, {px = 0x15801960}, {px = 0x157ed890}, {px = 0x157d3f40}, {px = 0x157b2ff0}, {px = 0x157fbd30}, {px = 0x157d97c0}, {px = 0x1581c170}, {
px = 0x157f7720}, {px = 0x157f0880}, {px = 0x1580ce10}, {px = 0x1580d1c0}, {px = 0x15808800}, {px = 0x157b84c0}, {px = 0x157d42f0}, {px = 0x1580d920}, {
px = 0x15810910}, {px = 0x15802f80}, {px = 0x157b03b0}, {px = 0x157c95a0}, {px = 0x157dade0}, {px = 0x157f5d50}, {px = 0x157f7ad0}, {px = 0x158020c0}, {
px = 0x157e1170}, {px = 0x157d46a0}, {px = 0x157f3c20}, {px = 0x157bec00}, {px = 0x157df3f0}, {px = 0x157b5880}, {px = 0x157fb980}, {px = 0x1580bf50}, {
px = 0x157f8230}, {px = 0x157b5c30}, {px = 0x157d81a0}, {px = 0x1581f160}, {px = 0x157b2c40}, {px = 0x157d9060}, {px = 0x157aab30}, {px = 0x157d9f20}, {
px = 0x157c4480}, {px = 0x157b9ae0}, {px = 0x157f6860}, {px = 0x15786500}, {px = 0x157ffbe0}, {px = 0x1578d750}, {px = 0x157d2cd0}, {px = 0x157d3430}, {
px = 0x1579b420}, {px = 0x15784780}, {px = 0x157d0ba0}, {px = 0x157dd2c0}, {px = 0x157b33a0}, {px = 0x157a82a0}, {px = 0x157d1300}, {px = 0x1581ea00}, {
px = 0x1582b120}, {px = 0x15820020}, {px = 0x1578b270}, {px = 0x157a6520}, {px = 0x15800aa0}, {px = 0x157bf360}, {px = 0x15814410}, {px = 0x157e9280}, {
px = 0x157f1ea0}, {px = 0x157aed90}, {px = 0x15825140}, {px = 0x157cdbb0}, {px = 0x157c6200}, {px = 0x1578d3a0}, {px = 0x1582c390}, {px = 0x15822c60}, {
px = 0x157df7a0}, {px = 0x157b6ea0}, {px = 0x15817b60}, {px = 0x157d1e10}, {px = 0x157f85e0}, {px = 0x157f7e80}, {px = 0x157898a0}, {px = 0x157a7ef0}, {
px = 0x157d4a50}, {px = 0x1582b880}, {px = 0x15827620}, {px = 0x15828ff0}, {px = 0x1582b4d0}, {px = 0x157d9b70}, {px = 0x15824d90}, {px = 0x157f0fe0}, {
px = 0x15830240}, {px = 0x15805810}, {px = 0x15827270}, {px = 0x1580dcd0}, {px = 0x15788d90}, {px = 0x157efd70}, {px = 0x158036e0}, {px = 0x15802820}, {
px = 0x15809e20}, {px = 0x1580a1d0}, {px = 0x1580a580}, {px = 0x1580a930}, {px = 0x1580ace0}, {px = 0x1580b090}, {px = 0x1580b440}, {px = 0x1580b7f0}, {
px = 0x1580bba0}, {px = 0x15799e00}, {px = 0x15828890}, {px = 0x15829b00}, {px = 0x15829eb0}, {px = 0x157a2dd0}, {px = 0x157a3180}, {px = 0x1578c4e0}, {
px = 0x1578c890}, {px = 0x15818dd0}, {px = 0x157f2600}, {px = 0x158198e0}, {px = 0x157f29b0}, {px = 0x157f2d60}, {px = 0x157f3110}, {px = 0x157f34c0}, {
px = 0x157ca810}, {px = 0x15820780}, {px = 0x157f4380}, {px = 0x158071e0}, {px = 0x157f4730}, {px = 0x157f4ae0}, {px = 0x157f4e90}, {px = 0x15823010}, {
px = 0x157f5240}, {px = 0x157f55f0}, {px = 0x15804950}, {px = 0x15804d00}, {px = 0x158050b0}, {px = 0x157c0d30}, {px = 0x157c10e0}, {px = 0x1581cc80}, {
px = 0x1579a560}, {px = 0x157c1840}, {px = 0x158219f0}, {px = 0x15822500}, {px = 0x15784ee0}, {px = 0x157a2670}, {px = 0x15785da0}, {px = 0x157a2a20}, {
px = 0x15816ca0}, {px = 0x157b8870}, {px = 0x157b8c20}, {px = 0x158182c0}, {px = 0x157b8fd0}, {px = 0x15819c90}, {px = 0x157cf1d0}, {px = 0x157cf930}, {
px = 0x157cfce0}, {px = 0x157d0090}, {px = 0x157ac150}, {px = 0x157ac500}, {px = 0x157d21c0}, {px = 0x157acc60}, {px = 0x157ad3c0}, {px = 0x157ad770}, {
px = 0x1581d030}, {px = 0x15782db0}, {px = 0x1581d3e0}, {px = 0x1581d790}, {px = 0x1581db40}, {px = 0x1581e2a0}, {px = 0x1581e650}, {px = 0x1580c6b0}, {
px = 0x15792c20}, {px = 0x15785640}, {px = 0x157dcf10}, {px = 0x157d0f50}, {px = 0x1578aec0}, {px = 0x15794d50}, {px = 0x1578a760}, {px = 0x157c8a90}, {
px = 0x157cd0a0}, {px = 0x157a8a00}, {px = 0x1579acc0}, {px = 0x157d9410}, {px = 0x15786c60}, {px = 0x157e48c0}, {px = 0x15783c70}, {px = 0x15810560}, {
px = 0x158168f0}, {px = 0x15806320}, {px = 0x15818a20}, {px = 0x157d2920}, {px = 0x15795f50}, {px = 0x15789c50}, {px = 0x1578f4d0}, {px = 0x157889e0}, {
px = 0x1580eb90}, {px = 0x1580ca60}, {px = 0x15819530}, {px = 0x1581f8c0}, {px = 0x157c65b0}, {px = 0x15797cd0}, {px = 0x15813550}, {px = 0x1582dd60}, {
px = 0x157c5e50}, {px = 0x1581a040}, {px = 0x15815680}, {px = 0x15815a30}, {px = 0x15809a70}, {px = 0x157bce80}, {px = 0x1582ec20}, {px = 0x157bd230}, {
px = 0x157b1d80}, {px = 0x157a5dc0}, {px = 0x15805bc0}, {px = 0x15801200}, {px = 0x1582d250}, {px = 0x157ff830}, {px = 0x15806e30}, {px = 0x157ed4e0}, {
px = 0x157d5cc0}, {px = 0x15796e10}, {px = 0x157e4160}, {px = 0x157aaee0}, {px = 0x157b79b0}, {px = 0x157c4f90}, {px = 0x1579e7c0}, {px = 0x157e3650}, {
px = 0x15797920}, {px = 0x157d72e0}, {px = 0x15809310}, {px = 0x15792fd0}, {px = 0x157e53d0}, {px = 0x1578fc30}, {px = 0x157fae70}, {px = 0x157c3d20}, {
px = 0x157f1af0}, {px = 0x15802470}, {px = 0x157fed20}, {px = 0x15797570}, {px = 0x1580e7e0}, {px = 0x157894f0}, {px = 0x1580fa50}, {px = 0x15783510}, {
px = 0x157b6af0}, {px = 0x157d6070}, {px = 0x15811420}, {px = 0x157a68d0}, {px = 0x157fc0e0}, {px = 0x15798b90}, {px = 0x15826000}, {px = 0x1578ffe0}, {
px = 0x157f6100}, {px = 0x157a3c90}, {px = 0x158080a0}, {px = 0x157c5340}, {px = 0x157a5660}, {px = 0x15814060}, {px = 0x157fc490}, {px = 0x157ba9a0}, {
px = 0x157ad010}, {px = 0x157b3eb0}, {px = 0x15832370}, {px = 0x158015b0}, {px = 0x157dda20}, {px = 0x1578ab10}, {px = 0x15817f10}, {px = 0x157f59a0}, {
px = 0x158152d0}, {px = 0x157d5560}, {px = 0x15782650}, {px = 0x157e99e0}, {px = 0x15826b10}, {px = 0x157d6b80}, {px = 0x15807cf0}, {px = 0x157ed130}, {
px = 0x15793730}, {px = 0x157b2130}, {px = 0x157dddd0}, {px = 0x157919b0}, {px = 0x15811f30}, {px = 0x1582d9b0}, {px = 0x157ce6c0}, {px = 0x157f8990}, {
px = 0x157d8550}, {px = 0x15787b20}, {px = 0x157f7370}, {px = 0x15795ba0}, {px = 0x15821290}, {px = 0x157b1620}, {px = 0x15812a40}, {px = 0x15788280}, {
px = 0x157868b0}, {px = 0x157d2570}, {px = 0x157cf580}, {px = 0x157af140}, {px = 0x157a7030}, {px = 0x158203d0}, {px = 0x15826ec0}, {px = 0x157dc400}, {
px = 0x157c35c0}, {px = 0x157c5aa0}, {px = 0x1581b2b0}, {px = 0x1579b070}, {px = 0x157edc40}, {px = 0x15796a60}, {px = 0x157c8e40}, {px = 0x15798430}, {
px = 0x15823770}, {px = 0x15787ed0}, {px = 0x157a4f00}, {px = 0x15828130}, {px = 0x157c4be0}, {px = 0x157b6390}, {px = 0x157e0dc0}, {px = 0x157f8d40}, {
px = 0x157d4e00}, {px = 0x15815de0}, {px = 0x157afc50}, {px = 0x15789140}, {px = 0x157d67d0}, {px = 0x1582f380}, {px = 0x15831100}, {px = 0x158309a0}, {
px = 0x157971c0}, {px = 0x1578a3b0}, {px = 0x157ba240}, {px = 0x157c56f0}, {px = 0x157d16b0}, {px = 0x157d5910}, {px = 0x157949a0}, {px = 0x1582a610}, {
px = 0x158045a0}, {px = 0x15818670}, {px = 0x157dc7b0}, {px = 0x157859f0}, {px = 0x15831fc0}, {px = 0x15821da0}, {px = 0x157c1490}, {px = 0x1582e110}, {
px = 0x1579a910}, {px = 0x15811070}, {px = 0x157d7df0}, {px = 0x1578cc40}, {px = 0x157a47a0}, {px = 0x157c9d00}, {px = 0x157de180}, {px = 0x15832720}, {
px = 0x157de8e0}, {px = 0x157a52b0}, {px = 0x157b9e90}, {px = 0x157e18d0}, {px = 0x1578b620}, {px = 0x157b54d0}, {px = 0x157cdf60}, {px = 0x157d6420}, {
px = 0x157aa020}, {px = 0x157e2790}, {px = 0x157b4d70}, {px = 0x15787770}, {px = 0x157dd670}, {px = 0x157db190}, {px = 0x15781ef0}, {px = 0x157b0000}, {
px = 0x157fff90}, {px = 0x15788630}, {px = 0x157db8f0}, {px = 0x157a7b40}, {px = 0x1579c690}, {px = 0x1578e9c0}, {px = 0x1578bd80}, {px = 0x1578db00}, {
px = 0x157957f0}, {px = 0x15793380}, {px = 0x157ae280}, {px = 0x15784b30}, {px = 0x157af4f0}, {px = 0x158263b0}, {px = 0x157b5fe0}, {px = 0x1582e4c0}, {
px = 0x1581af00}, {px = 0x157ca460}, {px = 0x15808f60}, {px = 0x157bc370}, {px = 0x157c6d10}, {px = 0x1581ba10}, {px = 0x157cc940}, {px = 0x15783160}, {
px = 0x1578f880}, {px = 0x157d07f0}, {px = 0x157e6290}, {px = 0x157edff0}, {px = 0x157cba80}, {px = 0x157f0c30}, {px = 0x15803a90}, {px = 0x15817050}, {
px = 0x1581f510}, {px = 0x157c7820}, {px = 0x157e0a10}, {px = 0x1581fc70}, {px = 0x157b0760}, {px = 0x157d8cb0}, {px = 0x1579bf30}, {px = 0x1580c300}, {
px = 0x157cb320}, {px = 0x157cbe30}, {px = 0x157cc1e0}, {px = 0x1580ef40}, {px = 0x157e2030}, {px = 0x1582f730}, {px = 0x158117d0}, {px = 0x157987e0}, {
px = 0x157e02b0}, {px = 0x157e5ee0}, {px = 0x157b7600}, {px = 0x157e3db0}, {px = 0x157c2ab0}, {px = 0x1578f120}, {px = 0x157cd450}, {px = 0x157b8110}, {
px = 0x157e2b40}, {px = 0x158122e0}, {px = 0x157ee750}, {px = 0x157de530}, {px = 0x157c3210}, {px = 0x157e4510}, {px = 0x1580f2f0}, {px = 0x15805460}, {
px = 0x15823b20}, {px = 0x157f94a0}, {px = 0x157dfb50}, {px = 0x1582c740}, {px = 0x157f90f0}, {px = 0x15791600}, {px = 0x157ea8a0}, {px = 0x15827d80}, {
px = 0x157c9950}, {px = 0x15805f70}, {px = 0x157cc590}, {px = 0x157f2250}, {px = 0x157c4830}, {px = 0x15790740}, {px = 0x15817400}, {px = 0x157924c0}, {
px = 0x157cb6d0}, {px = 0x157f1390}, {px = 0x1582a260}, {px = 0x15806a80}, {px = 0x157c2e60}, {px = 0x157cd800}, {px = 0x157f9c00}, {px = 0x157d1a60}, {
px = 0x158258a0}, {px = 0x1582cea0}, {px = 0x15802bd0}, {px = 0x15816190}, {px = 0x157f9850}, {px = 0x157d3080}, {px = 0x157cccf0}, {px = 0x157fa360}, {
px = 0x15800e50}, {px = 0x157fcbf0}, {px = 0x1581def0}, {px = 0x157ee3a0}, {px = 0x157ca0b0}, {px = 0x157d7690}, {px = 0x1582efd0}, {px = 0x157b5120}, {
px = 0x15819180}, {px = 0x157df040}, {px = 0x157b2890}, {px = 0x15792870}, {px = 0x157bfe70}, {px = 0x15793ae0}, {px = 0x157aa780}, {px = 0x157db540}, {
px = 0x157f9fb0}, {px = 0x1581b660}, {px = 0x1585a510}, {px = 0x1585d500}, {px = 0x1584c070}, {px = 0x15858790}, {px = 0x1584b910}, {px = 0x158430a0}, {
px = 0x15836220}, {px = 0x15853dd0}, {px = 0x15851540}, {px = 0x1585f9e0}, {px = 0x1584ff20}, {px = 0x15846090}, {px = 0x1584d2e0}, {px = 0x15854c90}, {
px = 0x1585f630}, {px = 0x15862620}, {px = 0x1585cda0}, {px = 0x15835710}, {px = 0x15854180}, {px = 0x158643a0}, {px = 0x158344a0}, {px = 0x1586b5f0}, {
px = 0x1584ae00}, {px = 0x1585bee0}, {px = 0x15855b50}, {px = 0x15861760}, {px = 0x158400b0}, {px = 0x15864eb0}, {px = 0x15849430}, {px = 0x1583fd00}, {
px = 0x1583e6e0}, {px = 0x15851ca0}, {px = 0x1583f950}, {px = 0x15871220}, {px = 0x15870360}, {px = 0x15861ec0}, {px = 0x15876340}, {px = 0x15867ea0}, {
px = 0x15838ab0}, {px = 0x15843bb0}, {px = 0x15846ba0}, {px = 0x1584ddf0}, {px = 0x1584a6a0}, {px = 0x15859db0}, {px = 0x15850a30}, {px = 0x1584cb80}, {
px = 0x15833230}, {px = 0x158527b0}, {px = 0x15865610}, {px = 0x1584b1b0}, {px = 0x1586ae90}, {px = 0x15852400}, {px = 0x15835360}, {px = 0x158365d0}, {
px = 0x1583a830}, {px = 0x15847300}, {px = 0x1583be50}, {px = 0x1583ea90}, {px = 0x15835e70}, {px = 0x15847e10}, {px = 0x158562b0}, {px = 0x15858b40}, {
px = 0x1583d0c0}, {px = 0x15857520}, {px = 0x15841320}, {px = 0x1585b3d0}, {px = 0x15856660}, {px = 0x158557a0}, {px = 0x158340f0}, {px = 0x1583b6f0}, {
px = 0x15861000}, {px = 0x158335e0}, {px = 0x15839d20}, {px = 0x15837bf0}, {px = 0x15872bf0}, {px = 0x15849f40}, {px = 0x1584bcc0}, {px = 0x1586ffb0}, {
px = 0x15862270}, {px = 0x15844a70}, {px = 0x15858030}, {px = 0x1583a480}, {px = 0x1584c420}, {px = 0x15840bc0}, {px = 0x1584ecb0}, {px = 0x1583abe0}, {
px = 0x15840460}, {px = 0x15841e30}, {px = 0x158604f0}, {px = 0x15836d30}, {px = 0x15833990}, {px = 0x15840810}, {px = 0x15869870}, {px = 0x1585e010}, {
px = 0x1586ed40}, {px = 0x15841a80}, {px = 0x1583baa0}, {px = 0x1585c640}, {px = 0x1586f850}, {px = 0x1586f4a0}, {px = 0x15859650}, {px = 0x1584b560}, {
px = 0x1586dad0}, {px = 0x158446c0}, {px = 0x15874970}, {px = 0x1584c7d0}, {px = 0x15863c40}, {px = 0x15856dc0}, {px = 0x15869110}, {px = 0x15845580}, {
px = 0x1584e550}, {px = 0x15837490}, {px = 0x15846440}, {px = 0x1585e3c0}, {px = 0x158694c0}, {px = 0x15863ff0}, {px = 0x15875be0}, {px = 0x15867740}, {
px = 0x158481c0}, {px = 0x158664d0}, {px = 0x15864b00}, {px = 0x1583b340}, {px = 0x158476b0}, {px = 0x158467f0}, {px = 0x158750d0}, {px = 0x15866120}, {
px = 0x1583a0d0}, {px = 0x1586b9a0}, {px = 0x15871d30}, {px = 0x1586f0f0}, {px = 0x158416d0}, {px = 0x15840f70}, {px = 0x15843800}, {px = 0x15857170}, {
px = 0x15849b90}, {px = 0x15872490}, {px = 0x1586a730}, {px = 0x1584cf30}, {px = 0x1584e900}, {px = 0x15844310}, {px = 0x1583d820}, {px = 0x15853670}, {
px = 0x1586cc10}, {px = 0x15839210}, {px = 0x158689b0}, {px = 0x158634e0}, {px = 0x15864750}, {px = 0x15870ac0}, {px = 0x1586de80}, {px = 0x15873700}, {
px = 0x15865260}, {px = 0x15869fd0}, {px = 0x1583c5b0}, {px = 0x15865d70}, {px = 0x158395c0}, {px = 0x15848920}, {px = 0x1586c860}, {px = 0x15873350}, {
px = 0x1584da40}, {px = 0x15845930}, {px = 0x158659c0}, {px = 0x15873ab0}, {px = 0x1585e770}, {px = 0x15850680}, {px = 0x1583e330}, {px = 0x15837fa0}, {
px = 0x1583cd10}, {px = 0x1583ee40}, {px = 0x15834850}, {px = 0x15839970}, {px = 0x15834c00}, {px = 0x15838700}, {px = 0x158370e0}, {px = 0x15835ac0}, {
px = 0x1583af90}, {px = 0x15856a10}, {px = 0x15851190}, {px = 0x15858ef0}, {px = 0x15845ce0}, {px = 0x1584e1a0}, {px = 0x15852050}, {px = 0x1586c4b0}, {
px = 0x15873e60}, {px = 0x1583f1f0}, {px = 0x15849080}, {px = 0x15836980}, {px = 0x15868d60}, {px = 0x1584d690}, {px = 0x158421e0}, {px = 0x15871980}, {
px = 0x15872fa0}, {px = 0x1586cfc0}, {px = 0x158553f0}, {px = 0x15869c20}, {px = 0x15837840}, {px = 0x1585dc60}, {px = 0x15874d20}, {px = 0x158548e0}, {
px = 0x158502d0}, {px = 0x15853a20}, {px = 0x15875480}, {px = 0x158497e0}, {px = 0x158532c0}, {px = 0x15860c50}, {px = 0x15868250}, {px = 0x1585c9f0}, {
px = 0x158583e0}, {px = 0x15859a00}, {px = 0x15850de0}, {px = 0x15875f90}, {px = 0x1586b240}...}
sa = {_sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {_val = {140737237576814, 140737488335520, 140737339556311, 6827816, 140737351925658, 0,
140737237577774, 140737292672080, 140737341980452, 6847752, 140737351925658, 0, 140737237579284, 0, 140737341980435, 6848152}}, sa_flags = -136429670,
sa_restorer = 0x0}

#6  0x000000000042ccda in Main () at ../../icinga-app/icinga.cpp:497

args = std::vector of length 0, capacity 0
logLevel = icinga::LogInformation
autocomplete = false
cmdname = {static NPos = 18446744073709551615, m_Data = "daemon"}
command = {px = 0x658370}
initconfig = {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/init.conf"}
argc = 3
autoindex = 0
visibleDesc = {static m_default_line_length = 80, m_caption = "Global options", m_line_length = 80, m_min_description_length = 40,
m_options = std::vector of length 14, capacity 16 = {{px = 0x663d00, pn = {pi_ = 0x6641b0}}, {px = 0x664230, pn = {pi_ = 0x664330}}, {px = 0x664350, pn = {
pi_ = 0x6643b0}}, {px = 0x6643d0, pn = {pi_ = 0x664500}}, {px = 0x664520, pn = {pi_ = 0x664630}}, {px = 0x6646e0, pn = {pi_ = 0x6647d0}}, {px = 0x6647f0, pn = {
pi_ = 0x6648f0}}, {px = 0x664910, pn = {pi_ = 0x664a70}}, {px = 0x664ab0, pn = {pi_ = 0x664b80}}, {px = 0x665040, pn = {pi_ = 0x6651a0}}, {px = 0x665220, pn = {
pi_ = 0x665330}}, {px = 0x665350, pn = {pi_ = 0x665490}}, {px = 0x665500, pn = {pi_ = 0x665590}}, {px = 0x6655d0, pn = {pi_ = 0x6656e0}}},
belong_to_group = std::vector of length 14, capacity 64 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1}, groups = std::vector of length 1, capacity 1 = {{
px = 0x665900, pn = {pi_ = 0x6659f0}}}}
positionalDesc = {m_names = std::vector of length 0, capacity 0, m_trailing = "arg"}
vm =
argv = 0x7fffffffead8
logLevel = icinga::LogInformation
hiddenDesc = {static m_default_line_length = 80, m_caption = "Hidden options", m_line_length = 80, m_min_description_length = 40,
m_options = std::vector of length 3, capacity 4 = {{px = 0x664b10, pn = {pi_ = 0x664d50}}, {px = 0x664db0, pn = {pi_ = 0x664e80}}, {px = 0x665800, pn = {
pi_ = 0x6658e0}}}, belong_to_group = std::vector of length 3, capacity 64 = {0, 0, 0}, groups = std::vector of length 0, capacity 0}
rc = 1

#7  0x000000000042e216 in main (argc=3, argv=0x7fffffffead8) at ../../icinga-app/icinga.cpp:763

rl = {rlim_cur = 16384, rlim_max = 16384}
rc = 4363808

Attachments

• long.conf mfriedrich - 2016-01-21 16:22:30 +00:00

Changesets

2016-02-23 11:08:07 +00:00 by mfriedrich 91dd96c

Fix unique creation of temp files

refs #10948

2016-02-23 11:30:59 +00:00 by mfriedrich 4cb5b97

Fix unique creation of temp files

fixes #10948

2016-02-23 11:34:38 +00:00 by mfriedrich a837198

Fix unique creation of temp files

fixes #10948

---

Relations:

• relates #10948
• relates #10948
• relates #10948

[icinga-migration]

Updated by rhillmann on 2016-01-14 14:56:29 +00:00

i can confirm this issue.
Icinga 2.4.1 is unusable at the moment. This is why we downgraded to 2.3.11...

[icinga-migration]

Updated by gbeutner on 2016-01-21 08:13:35 +00:00

• Category set to Configuration
• Assigned to set to mfriedrich

dnsmichi: Can you have a look at this please?

[icinga-migration]

Updated by mfriedrich on 2016-01-21 16:25:21 +00:00

• File added long.conf
• Status changed from New to Feedback
• Assigned to changed from mfriedrich to vytenis

I'm not able to reproduce this with a single config item built from the given examples. Seems this only happens in larger environments themselves.

@Vytenis

• Which size is your state file?
• How many objects are involved?
• Any chance to get this config for reproducing the problem? (might want to email it to michael.friedrich(at)netways.de)

[icinga-migration]

Updated by vytenis on 2016-01-21 16:35:38 +00:00

This happens with repeated configuration updates (once per hour, after 12 hours-3days or so) via API.
Config is autogenerated and follows the same patterns however corruption happens on all hosts in cluster (that monitor different hosts). However, all clusters are rather large.
Statefile is at around 120MB~. Object count:

information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 1 ApiUser.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 1 Zone.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 2 LivestatusListeners.
information/ConfigItem: Instantiated 68 CheckCommands.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 147808 Dependencies.
information/ConfigItem: Instantiated 1025 Hosts.
information/ConfigItem: Instantiated 103 HostGroups.
information/ConfigItem: Instantiated 802 UserGroups.
information/ConfigItem: Instantiated 883 Users.
information/ConfigItem: Instantiated 76496 Services.
information/ConfigItem: Instantiated 436 ServiceGroups.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.

I can email the configuration later, after I anonymize it a bit.

[icinga-migration]

Updated by mfriedrich on 2016-01-26 18:04:58 +00:00

• Status changed from Feedback to Assigned
• Assigned to changed from vytenis to mfriedrich
• Priority changed from Normal to High

Thanks. I'm eagerly awaiting your mail :)

[icinga-migration]

Updated by vytenis on 2016-01-26 18:17:37 +00:00

Sorry, did not have the bandwidth lately to do it, will send it tomorrow.

[icinga-migration]

Updated by mfriedrich on 2016-01-26 18:49:14 +00:00

Awesome, thanks :)

[icinga-migration]

Updated by digrman on 2016-01-26 20:31:28 +00:00

I am also seeing this behavior. I just sent dnsmichi a tar file.

I would note that I also found some errant host conf files that had file names that started with "" and "-".

e.g.
-i-221fcb94.ec2.somecompany.net.conf
--i-7f26edc9.somecompany.net.conf

So my issue might not the same but might have a similar fix if parsing "-" is an issue in object names.

Thanks in advance for any help!

[icinga-migration]

Updated by digrman on 2016-01-26 20:48:59 +00:00

TO add: those files with - and -- come from an automated puppet process and puppetdb connection. Had a problem with host names in puppetdb that truncated and left the "-"s.

Thanks,

Lance

[icinga-migration]

Updated by vytenis on 2016-01-27 23:33:19 +00:00

Michael - I've emailed you our base configuration and two variations of our "core" configuration file that defines all hosts/services/dependencies.

[icinga-migration]

Updated by mfriedrich on 2016-01-28 09:13:00 +00:00

• Target Version set to 2.4.2

[icinga-migration]

Updated by vytenis on 2016-01-29 15:41:14 +00:00

It may be related to JSON serializer:
I can also get invalid JSON on API, even a restart does not fix it!

curl --user x:x -k https://localhost:4123/v1/status | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4440    0  4440    0     0  19451      0 --:--:-- --:--:-- --:--:-- 19473
parse error: ':' not as part of an object at line 1, column 3764

Corruption looks very similar to statefile corruption (strings are cutoff, newlines are inserted), emailed you the data

[icinga-migration]

Updated by mfriedrich on 2016-02-02 08:41:41 +00:00

critical/cli: Failed to restore state file: Error: Invalid NetString (missing ,)

Maybe related to #11014. Requires further tests with the current git master.

[icinga-migration]

Updated by digrman on 2016-02-02 15:11:26 +00:00

FWIW, I can confirm the issue is still occurring for us after removing the errant host conf files that had file names that started with "" and "-". So that was not the issue. Please let me know if I can provide any further details.

[icinga-migration]

Updated by mfriedrich on 2016-02-05 14:09:41 +00:00

I wasn't able to clear my todo list for reproducing the issue yet, sorry about that. We discussed that issue this week though and since we also have another large customer setup with a huge state file, we thought about a full filesystem causing incomplete (state) files on disk. Does that theory apply to your systems as well?

[icinga-migration]

Updated by vytenis on 2016-02-05 14:39:07 +00:00

No, the disk usage is less than 10% - however I saw some JSON corruption in API responses with git master last week that looked very similar to statefile corruption. Can it be related?

[icinga-migration]

Updated by mfriedrich on 2016-02-05 14:48:40 +00:00

We saw them as well, but Gunnar fixed those in the latest git master.

[icinga-migration]

Updated by mfriedrich on 2016-02-05 14:52:23 +00:00

Can you - by chance - please test this using the latest git master? I cannot dig up the corresponding commit, but it happened on Tuesday iirc.

[icinga-migration]

Updated by vytenis on 2016-02-05 15:28:16 +00:00

Updated to latest git master on a test cluster, will let you know if it does not crash and burn in 4-6 days

[icinga-migration]

Updated by vytenis on 2016-02-10 17:15:51 +00:00

The corruption still happened after 20-25 reload cycles (in less than 24 hours):

Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: [2016-02-10 17:13:05 +0000] critical/cli: Failed to restore state file: Error: Invalid NetString (missing :)
Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: (0) libbase.so: void boost::throw_exception >(boost::exception_detail::error_info_injector const&) (+0x104) [0x7f29c637ef14]
Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: (1) libbase.so: void boost::exception_detail::throw_exception_(std::invalid_argument const&, char const*, char const*, int) (+0x67) [0x7f29c637efc7]
Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: (2) libbase.so: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr const&, icinga::String*, icinga::StreamReadContext&, bool) (+0xdd) [0x7f29c6331e0d]
Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: (3) libbase.so: icinga::ConfigObject::RestoreObjects(icinga::String const&, int) (+0x20d) [0x7f29c6356cad]
Feb 10 17:13:05 icinga03-sjc1 icinga2[38199]: (4) libcli.so: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector > const&) const (+0x687) [0x7f29c6d4d787]

[icinga-migration]

Updated by teqqy on 2016-02-11 16:39:38 +00:00

It seems it corresponds with an often reload of the icinga process. i reload my about 15 times today and got the same error (same stable Version).

[icinga-migration]

Updated by PowellEB on 2016-02-11 16:57:46 +00:00

We are seeing the same issue, but we are not using the API for configuration updates/regenerations
(API enabled 02/08/2016 when we moved to v2.4.1-176-g4049cc0)

We are seeing a crash @ approx 30 hours. same errors in logs.
We have disabled the API feature, and will see if we can go another 30 hours with no crash.

[icinga-migration]

Updated by digrman on 2016-02-11 21:04:38 +00:00

We are having the statefile issue with the same log entry complaining on restart as well.
Puppet restarts Icinga2 with some frequency as new systems come online as they are available in Puppetdb and new config files are generated. Problem seems to occur roughly ever 30-50 hours or so for us.

icinga2-common-2.4.1-1.el7.centos.x86_64
icinga2-bin-2.4.1-1.el7.centos.x86_64
icinga2-2.4.1-1.el7.centos.x86_64
icinga2-ido-mysql-2.4.1-1.el7.centos.x86_64

Is there any other information we could collect to assist? I will see about getting GDB working.

Most recent crashes:
rw------. 1 icinga icinga 48 Feb 8 04:11 report.1454904697.2338359
rw------. 1 icinga icinga 48 Feb 10 15:11 report.1455117095.7040811
Failed to launch GDB: No such file or directory

[icinga-migration]

Updated by mfriedrich on 2016-02-14 14:51:49 +00:00

• Status changed from Assigned to New
• Assigned to deleted mfriedrich
• Target Version deleted 2.4.2

Just a short heads up - unfortunately other projects and tasks require my attention at the time of writing. So if you are eagerly waiting for patch, I'd advise you to either dive into the code yourself or find one capable of. I'll therefore remove the target version and myself as assignee, sorry.

[icinga-migration]

Updated by ziaunys on 2016-02-15 02:17:17 +00:00

We just saw this issue as well. Our entire Icinga2 infrastructure was down and it wouldn't start until I removed the state file. It seems to have happened randomly because no one has made any changes since it's the weekend for us. This seems like a fairly critical issue. Let me know if there's anything I can do to help troubleshoot or expedite a patch.

[icinga-migration]

Updated by PowellEB on 2016-02-15 16:23:46 +00:00

Since disabling the API feature 02/11/2016 0700 EST, we have not had a crash todate.
We are having issues now with standard "systemctl reload icinga2", so using restart
temporarily...but no state file issues so far.

[icinga-migration]

Updated by mfriedrich on 2016-02-23 11:08:23 +00:00

It is not related to long object names, nor the size. Most likely there are multiple (dead) processes writing the same temporary cache file and thus one renames it while another process is writing to it. The git master provides a patch for creating exclusive and unique temp files. Please test that.

[icinga-migration]

Updated by mfriedrich on 2016-02-23 11:33:58 +00:00

• Subject changed from Icinga statefile corruption with very long object names to Icinga state file corruption with temporary file creation
• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Target Version set to 2.4.2

[icinga-migration]

Updated by mfriedrich on 2016-02-23 11:35:02 +00:00

• Backport? changed from Not yet backported to Already backported

[icinga-migration]

Updated by mfriedrich on 2016-02-23 11:35:03 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset 4cb5b97.

[icinga-migration]

Updated by digrman on 2016-02-23 14:47:01 +00:00

This is great news!
Thanks so much for working on this.

[icinga-migration]

Updated by mfriedrich on 2016-02-24 09:58:30 +00:00

• Relates set to 11212

[icinga-migration]

Updated by mfriedrich on 2016-02-24 09:58:56 +00:00

• Relates set to 11211

[icinga-migration]

Updated by mfriedrich on 2016-02-25 00:35:15 +00:00

• Relates set to 7456