[dev.icinga.com #4049] bugfixes for socket queue, transactions and libdbi handling

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/4049

Created by mfriedrich on 2013-05-04 14:52:01 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2013-07-20 12:18:43 +00:00)
Target Version: 1.9.2
Last Update: 2014-12-08 14:38:05 +00:00 (in Redmine)

Icinga Version: 1.10.0
OS Version: any

---

there's more for the socket queue and transactions, as well as libdbi bug workarounds which require long term tests.

Changesets

2013-06-22 15:53:13 +00:00 by (unknown) f841c16

update Changelog

refs #4049
refs #4210

---

Relations:

• relates #4210

[icinga-migration]

Updated by jschanz on 2013-05-07 11:42:37 +00:00

Please discuss about a solution for the behaviour of the is_active state in icinga_objects.
Socket queue works fine, but after an Icinga reload or restart all objects will be set to "is_active = 0". Due to the fast restart of the Icinga-Core, this is a huge problem for service checks like "check_bp_status.pl" from the Business Process Addon. This plugin checks if an service is active or valid (is_active=1). Because of the fast restart of the core, all objects will be "0" until the complete config dump is written to the database. This tooks in our environment about 3 minutes. During this time the check result of the above check is UNKNOWN and will be alerted.
Same behaviour affects all applications which are using the is_active flag.
Is it possible to invalidate all old objects after the config is written to the database?

Fast restart like done with the socket queue is pretty cool. So we get no Whoops from Icinga anymore :-) Thanks ...

[icinga-migration]

Updated by gbeutner on 2013-05-07 15:07:56 +00:00

How exactly is this related to this issue?

[icinga-migration]

Updated by jschanz on 2013-05-08 06:34:13 +00:00

This affects also Icinga-Web. Until the config dump was written to the database, no state will be visibile in Icinga-Web as Icinga-Web also takes care of the is_active flag (0 / 1) in icinga_objects table.

[icinga-migration]

Updated by mfriedrich on 2013-05-08 07:15:20 +00:00

split the issue please. this issue was solely created for the socket queue and database backend drivers. when it comes to database queries, this is a different topic.

[icinga-migration]

Updated by mfriedrich on 2013-05-29 09:08:33 +00:00

• Subject changed from socket queue refactoring to bugfixes for socket queue, transactions and libdbi handling
• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Target Version set to 1.9.2

[icinga-migration]

Updated by mfriedrich on 2013-05-31 20:10:41 +00:00

62 seconds for the config dump processed with 4k services, mysql 5.1, untuned. vm with 4 cores, 2gb ram.

May 31 22:05:02 sol icinga: Icinga 1.9.0 starting... (PID=24639)
May 31 22:05:02 sol icinga: Local time is Fri May 31 22:05:02 CEST 2013
May 31 22:05:02 sol icinga: LOG VERSION: 2.0
May 31 22:05:02 sol icinga: idomod: IDOMOD 1.9.0 (05-07-2013) Copyright(c) 2005-2008 Ethan Galstad, Copyright(c) 2009-2013 Icinga Development Team (https://www.icinga.org)
May 31 22:05:02 sol ido2db: Client connected, data available.
May 31 22:05:02 sol icinga: idomod: Successfully connected to data sink.  11 queued items to flush.
May 31 22:05:02 sol icinga: idomod: Successfully flushed 11 queued items to data sink.
May 31 22:05:02 sol ido2db: Handling client connection...
May 31 22:05:02 sol icinga: Event broker module 'IDOMOD' version '1.9.0' from '/usr/lib/idomod.so' initialized successfully.
May 31 22:05:02 sol ido2db: Successfully connected to mysql database
May 31 22:05:03 sol icinga: Finished daemonizing... (New PID=24643)
May 31 22:05:03 sol icinga: SERVICE FLAPPING ALERT: test_host_049;test_random_17;STARTED; Service appears to have started flapping (21.1% change >= 20.0% threshold)
May 31 22:05:03 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=412, right=0; iostats=15701
May 31 22:05:04 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1925237, right=0; iostats=1940526
May 31 22:05:05 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3567166, right=0; iostats=3582455
May 31 22:05:05 sol icinga: Event loop started...
May 31 22:05:09 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3568430, right=0; iostats=3583719
May 31 22:05:10 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3568476, right=0; iostats=3583765
May 31 22:05:11 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3568522, right=0; iostats=3583811
May 31 22:05:12 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3441719, right=0; iostats=3583984
May 31 22:05:14 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3441719, right=0; iostats=3583984
May 31 22:05:15 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3350553, right=0; iostats=3603922
May 31 22:05:16 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3239622, right=0; iostats=3604095
May 31 22:05:17 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3239622, right=0; iostats=3604095
May 31 22:05:19 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=3128691, right=0; iostats=3604268
May 31 22:05:21 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2907427, right=0; iostats=3605212
May 31 22:05:23 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2796323, right=0; iostats=3605212
May 31 22:05:24 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2796496, right=0; iostats=3605385
May 31 22:05:25 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2692499, right=0; iostats=3612492
May 31 22:05:26 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2565695, right=0; iostats=3612664
May 31 22:05:28 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2454591, right=0; iostats=3612664
May 31 22:05:29 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2454591, right=0; iostats=3612664
May 31 22:05:30 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2232648, right=0; iostats=3612929
May 31 22:05:32 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2121544, right=0; iostats=3612929
May 31 22:05:33 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2010440, right=0; iostats=3612929
May 31 22:05:34 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2010440, right=0; iostats=3612929
May 31 22:05:34 sol icinga: error executing command '/etc/icinga/tests/forker': No such file or directory. Make sure that the file actually exists (in PATH, if set) and is executable!
May 31 22:05:35 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=2013567, right=0; iostats=3616056
May 31 22:05:36 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1791705, right=0; iostats=3616402
May 31 22:05:38 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1680601, right=0; iostats=3616402
May 31 22:05:39 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1680601, right=0; iostats=3616402
May 31 22:05:41 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1569589, right=0; iostats=3616494
May 31 22:05:42 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1347554, right=0; iostats=3616667
May 31 22:05:43 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1236450, right=0; iostats=3616667
May 31 22:05:44 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1236450, right=0; iostats=3616667
May 31 22:05:45 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1249956, right=0; iostats=3630173
May 31 22:05:47 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=1027965, right=0; iostats=3630390
May 31 22:05:48 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=917034, right=0; iostats=3630563
May 31 22:05:49 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=917034, right=0; iostats=3630563
May 31 22:05:50 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=806874, right=0; iostats=3631507
May 31 22:05:52 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=695770, right=0; iostats=3631507
May 31 22:05:53 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=695770, right=0; iostats=3631507
May 31 22:05:54 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=584839, right=0; iostats=3631680
May 31 22:05:55 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=592759, right=0; iostats=3639600
May 31 22:05:56 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=370724, right=0; iostats=3639773
May 31 22:05:58 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=259620, right=0; iostats=3639773
May 31 22:05:59 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=259620, right=0; iostats=3639773
May 31 22:06:00 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=148781, right=0; iostats=3640038
May 31 22:06:02 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=37677, right=0; iostats=3640038
May 31 22:06:04 sol ido2db: IDO2DB proxy stats (p=0x201faa0): left=256, right=0; iostats=3640294
May 31 22:06:04 sol ido2db: Config dump completed

[icinga-migration]

Updated by mfriedrich on 2013-05-31 20:14:19 +00:00

TODO

• proceed changes with more config objects from https://git.icinga.org/?p=icinga-core.git;a=commit;h=150fe2eb9c10b6c484087713e8099610faecd785

[icinga-migration]

Updated by mfriedrich on 2013-06-03 18:56:40 +00:00

https://git.icinga.org/?p=icinga-core.git;a=shortlog;h=refs/heads/fix/socketqueue-4049 will be merged to support/1.9 (and next once 1.9.2 is out)

[icinga-migration]

Updated by mfriedrich on 2013-06-22 15:43:52 +00:00

again the untuned debian squeeze box, rebased branch against current next dev tree.

Jun 22 17:41:03 sol icinga: Icinga 1.9.1 starting... (PID=26397)
Jun 22 17:41:07 sol icinga: Event loop started...
Jun 22 17:42:03 sol ido2db: Config dump completed

[icinga-migration]

Updated by mfriedrich on 2013-06-28 13:50:38 +00:00

testers feedback: http://www.monitoring-portal.org/wbb/index.php?page=Thread&postID=190800#post190800

[icinga-migration]

Updated by mfriedrich on 2013-06-28 13:50:52 +00:00

• Status changed from Assigned to 8
• Done % changed from 0 to 100

[icinga-migration]

Updated by mfriedrich on 2013-07-20 12:18:43 +00:00

• Status changed from 8 to Resolved

[icinga-migration]

Updated by mfriedrich on 2014-12-08 14:38:05 +00:00

• Project changed from 18 to Core, Classic UI, IDOUtils
• Category changed from 79 to IDOUtils
• Icinga Version changed from 1 to 1
• OS Version set to any