[dev.icinga.com #11103] Problem with hostgroup_members table cleanup

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11103

Created by wstiern on 2016-02-06 03:11:36 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-02-08 10:10:04 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:48 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 1

---

At some point this series of errors is thrown in the icinga2.log:

  state_type = '1',  status_update_time = FROM_UNIXTIME(1454727237) WHERE service_object_id = 4858;INSERT INTO icinga_statehistory (check_source, current_check_attempt, endpoint_object_id, instance_id, last_hard_state, last_state, long_output, max_check_attempts, object_id, output, state, state_change, state_time, state_time_usec, state_type) VALUES ('icinga.pulse.prod', '1', 161, 1, '3', '3', '', '3', 4858, '', '3', '1', FROM_UNIXTIME(1454727237), '987210', '1');INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (161, FROM_UNIXTIME(1454727238), '19208', 1, 'SERVICE ALERT: lga-rtb14;rtb-node-dataservices-exceptions-0;UNKNOWN;HARD;1;', FROM_UNIXTIME(1454727238), '16384', 4697);UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'graphite',  check_source = 'icinga.pulse.prod',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '3',  endpoint_object_id = 161,  event_handler = '',  event_handler_enabled = '1',  execution_time = '15.483777046203613',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1454727238),  last_hard_state = '3',  last_hard_state_change = FROM_UNIXTIME(1454727238),  last_state_change = FROM_UNIXTIME(1454727207),  latency = '0',  long_output = '',  max_check_attempts = '3',  next_check = FROM_UNIXTIME(1454727822),  normal_check_interval = '10',  notifications_enabled = '1',  original_attributes = 'null',  output = '',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = '',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '2.5',  scheduled_downtime_depth = '0',  service_object_id = 4697,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1454727238) WHERE service_object_id = 4697;INSERT INTO icinga_statehistory (check_source, current_check_attempt, endpoint_object_id, instance_id, last_hard_state, last_state, long_output, max_check_attempts, object_id, output, state, state_change, state_time, state_time_usec, state_type) VALUES ('icinga.pulse.prod', '1', 161, 1, '3', '3', '', '3', 4697, '', '3', '1', FROM_UNIXTIME(1454727238), '18850', '1')"
[2016-02-05 21:54:01 -0500] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2016-02-05 21:54:01 -0500] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')

When this happens, the Icingaweb2 UI loses contact with the daemon ("Backend Icinga is not available") and doesn't recover until I stop icinga2, nuke /var/lib/icinga2/icinga2.state, and start icinga2 again. I've wiped and rebuilt both IDO and Icingaweb DBs many times. Please let me know what other information I can provide.

Attachments

• icinga2.error wstiern - 2016-02-06 03:12:51 +00:00 - Excerpt from error log during DB failure.

Changesets

2016-02-08 10:05:27 +00:00 by mfriedrich 99dbb24

DB IDO: Fix problem with hostgroup_members cleanup

fixes #11103

2016-02-22 19:23:18 +00:00 by mfriedrich 5b30c9e

DB IDO: PArtially revert change for *group_members

Otherwise the index breaks.

refs #11103

2016-02-23 08:45:10 +00:00 by mfriedrich 2b64364

DB IDO: Fix problem with hostgroup_members cleanup

fixes #11103

2016-02-23 08:45:10 +00:00 by mfriedrich 2ec36d2

DB IDO: PArtially revert change for *group_members

Otherwise the index breaks.

refs #11103

---

Relations:

• relates #10855
• relates #8714
• relates #11103

[icinga-migration]

Updated by wstiern on 2016-02-06 03:13:02 +00:00

• File added icinga2.error

Totally botched the error text. Here's a file.

[icinga-migration]

Updated by tobiasvdk on 2016-02-06 20:22:22 +00:00

• Relates set to 11017

[icinga-migration]

Updated by mfriedrich on 2016-02-08 09:50:23 +00:00

• Relates deleted 11017

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:00:47 +00:00

• Relates set to 10855

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:01:56 +00:00

• Subject changed from State File Corruption to Problem with hostgroup_members table cleanup
• Category set to DB IDO

The delete/insert queries are not put into a query group required by the changes in #8714 and #10855.

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:02:04 +00:00

• Relates set to 8714

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:06:03 +00:00

• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Target Version set to 2.4.2

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:07:23 +00:00

Please re-test using the latest snapshot packages (icinga2 --version == v2.4.1-178-g99dbb24).

[icinga-migration]

Updated by mfriedrich on 2016-02-08 10:10:04 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset 99dbb24.

[icinga-migration]

Updated by wstiern on 2016-02-08 15:13:01 +00:00

dnsmichi wrote:

Please re-test using the latest snapshot packages (icinga2 --version == v2.4.1-178-g99dbb24).

Pardon my ignorance here, but I can't find a RPM that matches that version in the repo. There do appear to be daily snapshots, should I grab one from today?

[icinga-migration]

Updated by mfriedrich on 2016-02-08 15:23:16 +00:00

Which distribution are you on?

[icinga-migration]

Updated by wstiern on 2016-02-08 15:54:36 +00:00

dnsmichi wrote:

Which distribution are you on?

CentOS 7. I've just added the snapshot branch of the filesystem as another Yum repo. If there's a specific datestamped version I should use let me know, otherwise I'll test with whatever's current. Thanks for your help! :)

[icinga-migration]

Updated by mfriedrich on 2016-02-09 14:58:14 +00:00

el7 packages are built on each git commit so you should be fine using the current revision. Once you've installed the package, run icinga2 --version and verify its working (post your results here then).

[icinga-migration]

Updated by mfriedrich on 2016-02-22 19:22:43 +00:00

• Relates set to 11133

[icinga-migration]

Updated by gbeutner on 2016-02-23 09:58:49 +00:00

• Backport? changed from Not yet backported to Already backported