[dev.icinga.com #162] add 'module' as object config, allowing cfg_dir usage loading multiple modules without touching broker_module in icinga.cfg

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/162

Created by mfriedrich on 2009-10-28 10:27:16 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2011-05-04 14:21:46 +00:00)
Target Version: 1.4
Last Update: 2011-05-04 14:21:46 +00:00 (in Redmine)

---

During install with ./configure --enable-idoutils configure should generate an unquoted entry for the concurrent broker module in icinga.cfg

Just for starters it would be a good idea to do that when installing the initial config with icinga.cfg

Opinions on that?

Changesets

2011-04-29 23:13:00 +00:00 by mfriedrich ac3bc80

core: add 'module' as object config, allowing cfg_dir usage loading multiple modules without touching broker_module in icinga.cfg #162

define module{
        module_name     idomod
        module_type     neb
        path            /usr/local/icinga/bin/idomod.o
        args            config_file=/usr/local/icinga/etc/idomod.cfg
}

this is *fully* optional to the default way of adding event broker modules.

it just shows an initial proof of concept which steps are needed to define
new objects and their basic usage.

and by adding this, every packager can ignore icinga.cfg:broker_module
by just installing idoutils.cfg into etc/modules/ or similar, just including
the cfg_dir and allowing this object config to be read, and the event broker
modules to be loaded.

pls read the issue for further information on that
https://dev.icinga.org/issues/162

refs #162

2011-04-30 12:25:41 +00:00 by mfriedrich 073d6e3

idoutils: install sample (commented) config in modules/idoutils.cfg using new 'module' object config #162

refs #162

2011-05-01 12:15:23 +00:00 by mfriedrich a3d83b6

update icinga.spec for usage of module definitions with idoutils #162

refs #162

2011-05-02 07:13:17 +00:00 by mfriedrich 7c1a00f

idoutils: fix initial creation of etc/modules if not created in make install-idoutils

refs #162

2011-05-02 11:57:06 +00:00 by mfriedrich 603fb89

classic ui: add module object definition for viewing with config.cgi including csv+json output #162

refs #162

---

Relations:

• relates #162

[icinga-migration]

Updated by Meier on 2009-10-28 19:21:54 +00:00

Like and dislike it. Its more automatic which is good on the other hand packagers have to revert it. e.g. the RPM is build with idoutils but splits them out in an optional package. Maybe the main config can also use includes so we can use config snippets for that kind of stuff.

[icinga-migration]

Updated by mfriedrich on 2009-11-24 14:33:05 +00:00

• Priority changed from Normal to Low

[icinga-migration]

Updated by Meier on 2010-01-05 14:21:23 +00:00

With more and more usefull brokers coming up I reall think it would be great if broker-configuration would be loadable from seperate files in a directory (the common *.d directory approach). That is much cleaner then automated config file edits.

[icinga-migration]

Updated by mfriedrich on 2010-01-10 01:33:31 +00:00

• Assigned to set to mfriedrich

sounds good. i would split that into nebmodules directory and other stuff like ido2db.cfg

• modules.d (idomod.cfg)
• extern.d (ido2db.cfg)

did a deeper look how the core handles this for basic config_dirs

1. vim xdata/xodtemplate.c

/* process all files in a specific config directory */
int xodtemplate_process_config_dir(char *dirname, int options){

by reading the filelist and then passing the correct config file based on nebmodule.o => nebmodule.cfg over to e.g. idomod

As the default dir reading just matches define {...} it would be good to get own syntax within the config file for that:

cfg_module_dir=sysconfdir/modules.d

to not to interfere with the core cfg_dir parsing.

In base/icinga.c it will then be referenced (with several functions calls in between) from

                        /* read in all object config data */
                        if(result==OK)
                                result=read_all_object_data(config_file);

Basically it would be better to implement that directly into the nebmods.

/* add a new module to module list */
int neb_add_module(char *filename,char *args,int should_be_loaded)

awaits arguments read from icinga.cfg currently.

here is the only call to that:

                else if(!strcmp(variable,"broker_module")){
                        modptr=strtok(value," \n");
                        argptr=strtok(NULL,"\n");
#ifdef USE_EVENT_BROKER 
                        neb_add_module(modptr,argptr,TRUE);
#endif
                        }

so it should be possible to hack neb_add_module to take modules.d/ if argptr is empty - just to stay compatible to old configs.

the arguments are being passed here:

/* load all modules */
int neb_load_all_modules(void){
        nebmodule *temp_module=NULL;
        int result=OK;

        for(temp_module=neb_module_list;temp_module;temp_module=temp_module->next){
                result=neb_load_module(temp_module);
                }

        return OK;
        }


/* load a particular module */
int neb_load_module(nebmodule *mod)

[...]

        /* run the module's init function */
        initfunc=mod->init_func;
        result=(*initfunc)(NEBMODULE_NORMAL_LOAD,mod->args,mod->module_handle);

so it would be good to decide upon module name then which arg will be passed. extraction from the module name (full path) should be easy and so would be the loading from modules.d

i'll give that a try.

and to add - idomod should also through an error if it cannot read it's config file!

[icinga-migration]

Updated by mfriedrich on 2010-01-31 19:34:21 +00:00

mh i'll postpone that a bit. it will be possible by reading the directory and then parsing the config automatically.

i'll propose a config switch with "module_enabled=1" otherwise icinga won't parse the config and load the module.
also a switch where the module can be found (should be set during configure!) e.g. "module_path=/usr/local/icinga/bin/broker/idomod.so"

i also would like to see then the following structure:

etc/module.d - for cfgs to read automatically
bin/broker - for nebmodule libs

icinga.cfg only needs the cfg dir to parse everything there.

after reading the config it must be checked of course if the binary is availaible.

if a neb module like mk_livestatus won't provide a cfg, a basic config is required. the module_path is optional, if not set, it will take the default like nebmodule.o

[icinga-migration]

Updated by mfriedrich on 2010-03-17 09:15:45 +00:00

• Priority changed from Low to Normal

  /usr/lib/icinga/module/ ... idomod.so, mklivestatus.so
  /etc/icinga/module.d/ ... idomod.cfg, mklivestatus.cfg

  idomod.cfg -> + module_enabled=1|0

  icinga.cfg -> + cfg_module_dir=/etc/icinga/module.d/

should be set via configure for packaging too!

[icinga-migration]

Updated by mfriedrich on 2010-06-01 21:16:54 +00:00

• Target Version set to 1.0.2

another idea, which comes initially from shinken [1]

remove idomod.cfg and create a module-idomod.cfg directly on the core in objects/

This will be parsed autotmatically (or we add module/ and include that to icinga.cfg)

The definition looks the same like service/hostdefinitions:

define module {
name idomod
path libdir/idomod.o
instance_name default
output_type unixsocket
output localstatedir/ido.sock
tcp_port 5668
use_ssl 0
output_buffer_items 5000
...
}

--enable-idoutils will enable the make install-idoutils and there we need to patch that into in copying the fresh modules config.

The core config parser reads all those values into a generic module config struct.
there might be slight adaptions for idomod, livestatus, merlin. This struct will be held in memory after reading the config through the core (icinga restart/reload).

This means that the module configuration is set and reset in memory (coming from the cfg on disk). When Icinga Core reloads the configuration, it can also reload the config on the module (at this stage we need a mutex on those variables, or if the config reload is sequential, it's fine).

Currently, reloading the core does NOT reload the module configuration - a restart is required. If there would be the opportunity to keep the config struct in memory and let the module read from that, it will be fine.
Just regard - neb_init or sth must be called when reloading the configuration on the core and having a module detected, in case the idomod variables are not shared externally.

1/ keep the struct in memory
2/ The other way would be extending idomod to read such module definition directly.

getting from 1/ into

• extending idomod to also accept external variables and/or reload calls.

• extend the data_processing_options into more readable values to be set. the calculated ones should be kept, but instead sth else, easier would be nice.

the old possibility including must be kept, but the new one should be introduced with our own module - idomod.

[1] http://shinken.git.sourceforge.net/git/gitweb.cgi?p=shinken/shinken;a=blob\_plain;f=src/etc/shinken-specific.cfg;hb=456c501d1c32d6c2705090f8e69e71330a637ad8

[icinga-migration]

Updated by mfriedrich on 2010-06-14 10:43:00 +00:00

• Target Version changed from 1.0.2 to 1.0.3

[icinga-migration]

Updated by mfriedrich on 2010-07-26 10:07:50 +00:00

• Target Version changed from 1.0.3 to 1.2 (Stable)

postponed.

[icinga-migration]

Updated by mfriedrich on 2010-09-16 11:56:32 +00:00

• Target Version changed from 1.2 (Stable) to 62

release 1.2 is near, so postponing that again.

[icinga-migration]

Updated by mfriedrich on 2010-10-07 08:20:44 +00:00

• Subject changed from --enable-idoutils shoud enable idomod in icinga.cfg to include config like modules directly from a conf.d dir

http://omdistro.org/projects/omd/repository/revisions/master/show/packages/nagios/patches

Mathias did a nice patch on that, but the main problem remains that the core reads the config several times on different locations, so do the cgis. this needs to be fixed in order to support everything.

[icinga-migration]

Updated by mfriedrich on 2010-10-10 12:13:22 +00:00

• Target Version changed from 62 to 1.3

[icinga-migration]

Updated by mfriedrich on 2011-01-04 17:12:02 +00:00

• Status changed from New to Assigned

1.3 will be unstable, so a good chance to introduce that as opt-in.

[icinga-migration]

Updated by mfriedrich on 2011-01-05 15:58:06 +00:00

• Status changed from Assigned to Closed
• Assigned to deleted mfriedrich
• Target Version deleted 1.3

i'll close that for now as it's unclear what will happen - omd patch ahs been dropped too.

if needed, i'll do a complete rewrite then, making it useful for main configs, but also event broker modules and such.

[icinga-migration]

Updated by mfriedrich on 2011-04-29 23:09:40 +00:00

• Subject changed from include config like modules directly from a conf.d dir to add 'module' as object config, allowing cfg_dir usage loading multiple modules without touching broker_module in icinga.cfg
• Status changed from Closed to Assigned
• Assigned to set to mfriedrich
• Target Version set to 1.4

in the last few days i did a lot of make fullinstall and always lost my icinga.cfg:broker_module

so i sat down and thought about initiating one final attempt on that. given what shinken proposed for module configs - https://github.com/naparuba/shinken/blob/master/etc/shinken-specific.cfg

i decided to create a new definition for 'module' allowing

• module_name
• module_type
• path
• args

module_name and path are mandatory to get this module registered.

define module{
        module_name     idomod
        module_type     neb
        path            /usr/local/icinga/bin/idomod.o
        args            config_file=/usr/local/icinga/etc/idomod.cfg
}

this gets read by xdata/xodtemplate.c as normal object.

this is mainly the same implementation as reading commands (basically). using the same skiplists, and such, and finally bringing this into common/objects.c where add_module makes sure that the modules are in memory list for further usage.

there is only one major problem with that - the object config gets read after the neb modules are initialized.

but i remembered that there is the option to read specific object config, in that case READ_MODULES - which gives me the opportunity to tell icinga to just read the modules config in advance.

because i can't touch the current neb_add_module function or the list handling, because the old mechanism on loading nebmodules must work like it should (icinga.cfg:broker_module) i decided to just loop through the module list after reading them, and then calling the neb_add_module function with module~~path and module~~>args in order to add those neb modules to the normal lists where they get loaded.

and they do ...

test - one idomod module

Checking modules...
        Checked 1 modules.

objects.cache

define module {
        module_name     idomod
        module_type     neb
        path    /usr/local/icinga/bin/idomod.o
        args    config_file=/usr/local/icinga/etc/idomod.cfg
        }

startup

Apr 30 00:35:20 nbfr icinga: Icinga 1.4.0 starting... (PID=31022)
Apr 30 00:35:20 nbfr icinga: Local time is Sat Apr 30 00:35:20 CEST 2011
Apr 30 00:35:20 nbfr icinga: LOG VERSION: 2.0
Apr 30 00:35:20 nbfr icinga: idomod: IDOMOD 1.4.0 (05-11-2011) Copyright (c) 2005-2008 Ethan Galstad (nagios@nagios.org), Copyright (c) 2009-2011 Icinga Development Team (http://www.icinga.org))
Apr 30 00:35:20 nbfr icinga: idomod: Could not open data sink!  I'll keep trying, but some output may get lost...
Apr 30 00:35:20 nbfr icinga: Event broker module '/usr/local/icinga/bin/idomod.o' initialized successfully.
Apr 30 00:35:20 nbfr icinga: Finished daemonizing... (New PID=31023)

test - insert duplicates modules.

Warning: Duplicate definition found for module 'idomod' (config file '/usr/local/icinga/etc/modules/ido.cfg', starting on line 8)

test - empty module path

Error: Module name or path is NULL

test - load one module object and one icinga.cfg:broker_module

Apr 30 00:52:15 nbfr icinga: Icinga 1.4.0 starting... (PID=31510)
Apr 30 00:52:15 nbfr icinga: Local time is Sat Apr 30 00:52:15 CEST 2011
Apr 30 00:52:15 nbfr icinga: LOG VERSION: 2.0
Apr 30 00:52:15 nbfr icinga: idomod: IDOMOD 1.4.0 (05-11-2011) Copyright (c) 2005-2008 Ethan Galstad (nagios@nagios.org), Copyright (c) 2009-2011 Icinga Development Team (http://www.icinga.org))
Apr 30 00:52:15 nbfr icinga: idomod: Could not open data sink!  I'll keep trying, but some output may get lost...
Apr 30 00:52:15 nbfr icinga: Event broker module '/usr/local/icinga/bin/idomod.o' initialized successfully.
Apr 30 00:52:15 nbfr icinga: idomod: IDOMOD 1.4.0 (05-11-2011) Copyright (c) 2005-2008 Ethan Galstad (nagios@nagios.org), Copyright (c) 2009-2011 Icinga Development Team (http://www.icinga.org))
Apr 30 00:52:15 nbfr icinga: idomod: Still unable to connect to data sink.  0 items lost, 0 queued items to flush.
Apr 30 00:52:15 nbfr icinga: Event broker module '/usr/local/icinga/bin/idomod.o' initialized successfully.
Apr 30 00:52:15 nbfr icinga: Finished daemonizing... (New PID=31511)

[icinga-migration]

Updated by mfriedrich on 2011-04-29 23:11:47 +00:00

the sample configs won't be installed, as people should place those themselves into the proper location (and so do packagers if they install idoutils).

this needs a bunch of testing, but it fully optional to existing implementation and does not harm the core in any way. if nothing there, nothing gets loaded additionally.

note: idoutils are not aware of such configs to dump, so this is not reflected in the database itsself then.

[icinga-migration]

Updated by mfriedrich on 2011-04-30 12:39:34 +00:00

i've now decided to install a sample (commented) config when initiating make install-idoutils as this will be easier for packagers to provide their own configs.

[icinga-migration]

Updated by mfriedrich on 2011-05-02 11:56:00 +00:00

• Done % changed from 0 to 90

i've now added support for module object defs in config.cgi - including csv and json export.

depends if idoutils should export that - this would need a new config table, 3 new queries, and one time startup flush of the table.

[icinga-migration]

Updated by mfriedrich on 2011-05-04 14:21:46 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 90 to 100

ready for icinga 1.4, initially and finally resolved.