Skip to content
This repository has been archived by the owner on Jan 15, 2019. It is now read-only.

[dev.icinga.com #866] non-blocking icinga core while config dump at startup to ido2db #401

Closed
icinga-migration opened this issue Oct 8, 2010 · 10 comments

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/866

Created by kloppi on 2010-10-08 18:50:23 +00:00

Assignee: (none)
Status: Closed (closed on 2011-09-25 00:11:28 +00:00)
Target Version: (none)
Last Update: 2014-12-08 14:34:41 +00:00 (in Redmine)


Hi,

when i start my icinga, it tooks over 5 minutes to start till the first check startsup when the idomod is integrated. It looks like the ido suggest the core to wait, until all sql operations are done and executed in fifo mode. To Speedup a simple option would be nice to cache all sql statements from the icinga process either in memory or in a file. Another process took all datas and put them in database. A Difference between icinga-web and icinga-cgi would be the effect but i think the difference would be minimized after some runtime because the quene would be processed permanently.

Changesets

2011-02-24 15:31:36 +00:00 by mfriedrich 0846bbc12d02f8d6667c657798ecf4296cf2c63f

idoutils: non-blocking icinga core while config dump at startup to ido2db #866 reduce housekeeping cycle (1h interval) #1157

check the issues for more information.
https://dev.icinga.org/issues/866
https://dev.icinga.org/issues/1157

refs #866
refs #1157

2011-04-28 10:52:39 +00:00 by mfriedrich 39fa0ac

idoutils: non-blocking icinga core while config dump at startup to ido2db #866 fix race condition when issueing multiple reloads results in hanging IDO2DB processes #1410

check the issues for more information.
https://dev.icinga.org/issues/866
https://dev.icinga.org/issues/1410

refs #866
refs #1410

2011-09-25 00:12:03 +00:00 by mfriedrich a34a899

* idoutils: idomod: change output_buffer to asynchrounous circular buffer between core and data sink, removing core blocking #1259

** main thread writes circular buffer, protected by a mutex
** client thread is started after event loop is starting, reading from circular buffer
** the config dump starts when the event loop starts, minimizing the overall startup delay
** on event loop start, idomod will now tell how many items to flush - please adjust this value in idomod.cfg accordinly
** default minimal output_buffer_items is now 50000 (see idomod.cfg)
** core blocking is removed, even if the database backend is slow / ido2db not responding

for more please read https://dev.icinga.org/issues/1259

refs #1259
refs #866

Relations:

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2010-10-10 11:34:10 +00:00

main problem in this regard is the config dump during startup (see idomod.cfg). I've been thinking (a lot and very often) about a second channel for that, pushing the configs aside. next to that, ido2db should be able to write everything incoming into a message queue like zeromq provides, and then having a worker running the database insert/update.

problem would be a new dependency, so i was thinking of "forking" idoutils into a renamed module, and do a complete rewrite with nonblocking sockets. or even based on mod_gearman. also changing the db layout like merlin has it.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-02-24 13:45:13 +00:00

  • Subject changed from Non-Blocking icinga core while SQL communication is dating up the database to Non-Blocking icinga core while SQL communication is updating the database

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-02-24 14:12:25 +00:00

i've been playing around with a worker thread on the dbuf read from the socket, using a mutex to keep buffers in sync.

the main problem with this attempt is - a full configdump with 200 hosts and 4000 services + rest takes <1 second. the dbuf holds ~600k lines then, which need to be worked on. so ido2db takes a lot of cpu available (2 cores, one is "dedicated" to ido2db process) to be working on handling the data (like normal).

so the consensus is ... no more blocking core, but more cpu usage.

having that run on a single core is a considered fail.

the housekeeping thread interferes a bit with the config dump. since the thread must get to now the instance_name (from the config dump!) in order to do housekeeping on the correct instance, i've now against postponed it 300 secs, and made this the default (less values not possible).

furthermore the housekeeping cycle has been extended to 3600 seconds instead of 60 seconds. normally people would set 50 to 60 minutes the lowest on data housekeeping, so why checking this every single minute? also reduces the delete queries being fired onto the database.

overall, except the cpu usage (and hacking the pthread scheduler on 'nice' priority seems to be not working) it's something i consider for further testing and productive deployments too as it will be huge enhancement, dropping long grown problems.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-02-24 15:17:19 +00:00

  • Subject changed from Non-Blocking icinga core while SQL communication is updating the database to non-blocking icinga core while config dump at startup to ido2db
  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Target Version set to 1.4

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-04-27 17:10:58 +00:00

  • Target Version changed from 1.4 to 1.5

i gotta resolve/debug several bugs on a circular buffer implementation in ido2db, and this can't be done for 1.4 right on without violating the 2-weeks-freeze policy - so postponing to 1.5

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-04-28 11:32:47 +00:00

  • Status changed from Assigned to Feedback
  • Priority changed from Low to Normal
  • Target Version changed from 1.5 to 1.4
  • Done % changed from 0 to 90

icinga 1.4 will feature a first implementation attempt, while resolving #1410 - goes hand in hand.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-05-02 12:14:38 +00:00

  • Target Version changed from 1.4 to 1.5
  • Done % changed from 90 to 50

postponed. needs a rewritten solution.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-05-27 16:22:09 +00:00

  • Assigned to deleted mfriedrich
  • Target Version deleted 1.5

i'll leave that until someone got a better idea.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2011-09-25 00:11:28 +00:00

  • Status changed from Feedback to Closed
  • Done % changed from 50 to 0

please check #1259 for future reference and/or solutions on both - buffering and config dump blockings.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2014-12-08 14:34:41 +00:00

  • Project changed from 18 to Core, Classic UI, IDOUtils
  • Category set to IDOUtils

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant