[dev.icinga.com #1259] implement circular buffer for idoutils #563
Comments
|
Updated by mfriedrich on 2011-04-27 16:57:58 +00:00
second attempt was not working either, so postponing that stuff for 1.5 |
|
Updated by mfriedrich on 2011-05-27 16:33:33 +00:00
|
|
Updated by mfriedrich on 2011-09-20 06:45:18 +00:00
idea for a message queue. the actual problem with that - if the kernel can't handle more, the msg is lost and the kernel complains meaning there's no actuall buffering on disk. furthermore this solution attemps to fork another child for doing the asynchronous processing. this would be dangerous for the already housekeeping thread to be forked then too - there should be only one thread running by the main process for data processing. another idea would be 0mq as new dependency, removing the deepdown msg queue layer and keeping an eye on the buffering. |
|
Updated by mfriedrich on 2011-09-25 00:09:17 +00:00
another idea would be a circular buffer within idomod - as the synchronous buffer is already there. the overall construct on the datasink and reconnecting, flushing etc is to be seen deprecated when placing main producer and thread consumer on that buffer, secured by a mutex lock. 1/ main problem - the core daemonizes by default, and started threads through idomod_init will die silently. solution - register a callback on PROCESS_DATA and catch up on NEBTYPE_PROCESS_EVENTLOOPSTART where we start the queue thread for consuming data config screen1
screen2
the basic difference is now that idomod does not connect immediately to the socket, but on eventloop start (and then flushing the buffer onto the socket, while asynchronously adding more data to it. from the syslog - TIMEPERIODTRANSITION is an output by mklivestatus indicating that the event loop has started. so 16 seconds startup and a ready-to-start-checking core on a vmware dualcore with 2gb ram ;-) (this includes data_processing_options = dropping timedevents, it's even less so 11 seconds all over. it could be less, if the daemonizing would happen differently. the problem with that is of course - a full buffer requires a lot of work of ido2db to handle that data properly. so the final data in the database will take it's time. still, one more problem. wouldn't be idoutils, if not. starting the queue thread after the config dump will cause problems. so the overall config dump (which is nothing else than looping through memory loaded lists and/or reading from file if demanded) won't be started initially when the config is loaded, but finally when the queue thread is started, as "post processing callback". the seperation which config will be processed, happens within the called funtions. benefit once more - moving the config dump after the event loop actually removes more blocking on startup, the delay is now ~2sec. the data in the database is now verified ok :) known bug if not all retained states are dumped to the database, a check needs to pass the actual state into the database (this is being seen on the first startup only, then the status tables remain filled). this is a problem with dumping retained states to the database. will be pushed to mfriedrich/ido + dev/ido
|
|
Updated by mfriedrich on 2011-09-25 00:09:31 +00:00
|
|
Updated by mfriedrich on 2011-09-27 21:49:33 +00:00 by reports from testers, this needs to be enhanced somehow. now the core does not block anymore, but the socket will because ido2db / rdbms is not fast enough. so there would be an end-to-end buffer needed, one for buffering on the core part if the connection dies, and one on the ido2db side, buffering data from the socket and the database. if having 2 asynchronous buffers, the lock (and wait) can be established again, and the core blocking would be minor, having 2 locations to increase performance. but beware, threading and circular buffers will make debugging harder. |
|
Updated by mfriedrich on 2011-09-27 21:49:58 +00:00
|
|
Updated by mfriedrich on 2011-09-27 23:28:43 +00:00 the circular buffer requires to add new config options to ido2db.cfg (defaults will be 50k items and /tmp/ido2db.tmp). looking good. but buffering also shows that the config dump itsself takes ages. probably don't drop config on restart/reload, remove checks on is_active and fake the web status data on the old status then. |
|
Updated by mfriedrich on 2011-10-24 17:19:35 +00:00
see #1934 for a deeper anaylsis on the data dumping and a proof of concept. it just does NOT work with this design. multiple workers or circular buffers won't make it happen when there are so many things to be done on an insert or update. and when you are using singlethreaded db libraries. => proof of concept reflects = rejected. |
|
Updated by mfriedrich on 2014-12-08 14:34:51 +00:00
|
This issue has been migrated from Redmine: https://dev.icinga.com/issues/1259
Created by mfriedrich on 2011-03-02 08:52:04 +00:00
Assignee: (none)
Status: Rejected (closed on 2011-10-24 17:19:35 +00:00)
Target Version: (none)
Last Update: 2014-12-08 14:34:51 +00:00 (in Redmine)
the current attempt on locking and writing and reading the dbuf range, and doing memmove is a quick&dirty hack, but not a desired solution for a final release (see #866).
it comes to mind that existing solutions from the external commands should be used, as the algorithm is basically the same.
socket -> reader -> circular_buffer <- process_data -> rdbms
see the following examples. it might make sense to use the direct implementation from icinga itsself, but i think we cannot prevent some code copies into ido2db.c or better, a new utils.c over there too.
include/icinga.h
base/commands.c
the one for processing the data in the buffer in the while loop.
maybe allow the same setting as core -
1> check as often as possible.base/events.c
the socketreader
base/utils.c
basically adapt the main initialisation to create and clear the circular buffer.
int init_command_file_worker_thread(void){
int shutdown_command_file_worker_thread(void){
void cleanup_command_file_worker_thread(void *arg){
this is not a worker thread in ido2db, but the main thread processing the data from the socket (in the core case a named pipe)
submitting the data is basically an insert into the circular buffer.
===
so handling the data from the socket the same way as the command pipe (only s/cmdfile/socket/) will do the trick, and not increase cpu load as it's now in #866
Attachments
Changesets
2011-09-25 00:12:03 +00:00 by mfriedrich a34a899
2011-09-27 15:56:09 +00:00 by mfriedrich 412e220
2011-09-27 23:30:19 +00:00 by mfriedrich b419560
Relations:
The text was updated successfully, but these errors were encountered: