[dev.icinga.com #1931] how to handle buffer overflows in ido2db

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/1931

Created by mfriedrich on 2011-09-25 10:18:36 +00:00

Assignee: (none)
Status: Closed (closed on 2012-05-05 11:34:43 +00:00)
Target Version: (none)
Last Update: 2014-12-08 14:46:22 +00:00 (in Redmine)

---

following up from #1259 it should be discussed, how a full buffer in idomod should be handled by the core.

• drop out after X retries (like it's currently implemented, default is 5, see idomod.h)
  • add a syslog entry (todo)
• retry as long as a buffer push is possible - this will block the main process

Changesets

2011-09-27 18:31:10 +00:00 by mfriedrich 9dfb19f

idoutils: move thread start + origcfg dump to new NEBTYPE_PROCESS_INITSTART and retained back to normal #1934, make logging threadsafe #1962, correct overflow log #1931

this changes the thread not to be started in event loop (new neb type making it happen sooner).
the original config gets dumped immediately, but after daemonizing too, which NEBTYPE_PROCESS_INITSTART
indicates for now. the retained config dump stays where it happened, the target in timeline is perfectly
fine when using a circular buffer to cache these things.
while the consumer queue thread is reading and dumping the configs to the database, the core can start
the event loop, pushing status updates and current checks to the queue as well.

the main disadvantage on a non-blocking neb module will be the buffer holding the data and needing
the exact same time as the core was blocked to finally dump the whole config stuff. so this remains still
in testing mode, and needs possible enhancements, basically on the sleep time of the threads and so on.

the debug logging not being threadsafe was being fixed with the previous commit, holding a file pointer
and writing simultanously to it is not really a good idea. the main logging functions used a core internal
function which would invoke a neb callback to self() if something gets written to the icinga log file.
this has been removed for now, the syslog will be used like done in ido2db, further logging must be done
side by side with the demanded ido2db logging and their syslog facilities.
previously, the newly added warning on the buffer overflow when the retry was hit, resulted in a segfault
therefore this fix was done urgent.

refs #1931
refs #1934
refs #1962

---

Relations:

• relates #1259
• relates #1931
• precedes #1931

[icinga-migration]

Updated by mfriedrich on 2011-09-27 18:30:14 +00:00

the syslog entry would now be added properly, using #1962 making it threadsafe.

[icinga-migration]

Updated by mfriedrich on 2011-09-27 18:30:24 +00:00

• Category set to 70
• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Priority changed from Normal to Low

[icinga-migration]

Updated by mfriedrich on 2011-09-28 10:07:48 +00:00

• Due Date deleted 2011

[icinga-migration]

Updated by mfriedrich on 2011-10-09 18:49:36 +00:00

the config dump must not be interrupted, so the blocking outage for config dump on a full buffer is intended be taken like the usual procedure. afterwards, for historical and status data, the full blocking should be able to run into timeout, just like the error writing to datasink option would provide in the first place. meaning to say that the xxx items to flush will drop out after y retries and not being within the config dump.

so to speak, implement the following

• on config dump start, set a global variable to indicate that
• on config dump end, reset that global variable
• while writing to the buffer, make it blocking while config dump
• if not within config dumps, set a timeout and/or retry max limit
• if dropping out due to a slow database connection, this should be logged and the user to be warned
• looping endless waiting for the buffer to be writable again will cause higher load on the system and a blocking core, race condition

a better attempt will be the change of data inserts on the ido2db part - see #1934

[icinga-migration]

Updated by mfriedrich on 2011-10-20 12:27:46 +00:00

• Subject changed from how to handle buffer overflows in idomod to how to handle buffer overflows in ido2db
• Category changed from 70 to 79

can only happen in ido2db.

[icinga-migration]

Updated by mfriedrich on 2011-10-24 17:20:20 +00:00

• Status changed from Assigned to Feedback
• Assigned to deleted mfriedrich
• Target Version deleted 1.6

this follows #1934 and a change in datadumping with idoutils.

[icinga-migration]

Updated by mfriedrich on 2012-05-05 11:34:43 +00:00

• Status changed from Feedback to Closed

wontfix.

[icinga-migration]

Updated by mfriedrich on 2014-12-08 14:46:22 +00:00

• Project changed from 18 to Core, Classic UI, IDOUtils
• Category changed from 79 to IDOUtils