Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #10829] IDO breaks when writing to icinga_programstatus with latest snapshots #3774

Closed
icinga-migration opened this issue Dec 11, 2015 · 9 comments
Closed

[dev.icinga.com #10829] IDO breaks when writing to icinga_programstatus with latest snapshots #3774

icinga-migration opened this issue Dec 11, 2015 · 9 comments
Labels
area/db-ido Database output blocker Blocks a release or needs immediate attention bug Something isn't working
Milestone
2.4.2

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10829

Created by tgelf on 2015-12-11 16:25:25 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-12-15 11:04:50 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:31 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 0

Hi,

I'm seeing this with the latest snapshots:

critical/TcpSocket: Invalid socket: Connection refused
critical/IdoMysqlConnection: Error "Duplicate entry '1' for key 'instance_id'" when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'master1', '1', '1', 1, '1', FROM_UNIXTIME(1449850710), '1', '1', '1', '19928', '1', FROM_UNIXTIME(1449847581), 'v2.4.1-64-gc5b13ff', FROM_UNIXTIME(1449850710));DELETE FROM icinga_programstatus WHERE instance_id = 1;INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '6');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '6');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '4');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '4');DELETE FROM icinga_runtimevariables WHERE instance_id = 1;COMMIT;BEGIN"
critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!

Could this be related to recent insert/update fixes?

Cheers,
Thomas

Attachments

Changesets

2015-12-15 10:58:50 +00:00 by mfriedrich da3d210

Fix priority ordering for IDO queries

fixes #10829
refs #8714

2015-12-16 12:57:29 +00:00 by mfriedrich 6387bec

Fix duplicate key violation for downtimes

refs #10829

2016-02-23 08:09:06 +00:00 by mfriedrich 2bc1d32

Fix priority ordering for IDO queries

fixes #10829
refs #8714

2016-02-23 08:15:48 +00:00 by mfriedrich 187a61d

Fix duplicate key violation for downtimes

refs #10829

Relations:
@icinga-migration
Copy link
Author

Updated by tgelf on 2015-12-11 16:46:01 +00:00

  • Priority changed from High to Normal

Lowering prio, does not happen all the times... strange. Any ideas? Could it happen after unclean disconnects or similar?

Cheers,
Thomas

@icinga-migration
Copy link
Author

Updated by tgelf on 2015-12-14 09:08:59 +00:00

Well, doesn't happen all the times - but again and again. I think I found the reason for this:

critical/IdoMysqlConnection: Error "Duplicate entry '1' for key 'instance_id'" when executing query "
INSERT INTO icinga_programstatus (active_host_checks_enabled, ...) VALUES
('1', '1', '1', 'master1', '1', '1', 1, '1', FROM_UNIXTIME(1449926784), '1', '1', '1', '31722', '1',
FROM_UNIXTIME(1449926494), 'v2.4.1-69-g8055f05', FROM_UNIXTIME(1449926784));
DELETE FROM icinga_programstatus WHERE instance_id = 1;
INSERT INTO icinga_runtimevariables..."

As we can see it inserts new programstatus before deleting the old one. This happens in the very same query execution.

Best,
Thomas

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-14 09:22:08 +00:00

  • Relates set to 8714

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-14 17:04:14 +00:00

  • Priority changed from Normal to High

    [2015-12-14 18:03:15 +0100] information/ConfigItem: Instantiated 1 Downtime.
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Activated all objects.
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Triggering Start signal for config items
    [2015-12-14 18:03:15 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Activated all objects.
    [2015-12-14 18:03:16 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_scheduleddowntime"
    DETAIL: Key (instance_id, object_id, entry_time, internal_downtime_id)=(1, 178, 2015-12-14 18:03:15+01, 39) already exists.
    " when executing query "INSERT INTO icinga_scheduleddowntime (author_name, comment_data, downtime_type, duration, endpoint_object_id, entry_time, instance_id, internal_downtime_id, is_fixed, is_in_effect, name, object_id, scheduled_end_time, scheduled_start_time, was_started) VALUES (E'icingaadmin', E'Scheduled downtime for backup', E'1', E'0', 1, TO_TIMESTAMP(1450112595), 1, E'39', E'1', E'0', E'mbmif.int.netways.de!load!mbmif.int.netways.de-1450112595-0', 178, TO_TIMESTAMP(1450144800), TO_TIMESTAMP(1450141200), E'0')"
    [2015-12-14 18:03:16 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for checkable 'mbmif.int.netways.de!procs'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for checkable 'mbmif.int.netways.de!procs'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk /'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for checkable 'mbmif.int.netways.de!ido-mysql'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk /'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk'
    [2015-12-14 18:03:23 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk'
    [2015-12-14 18:03:23 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for checkable 'mbmif.int.netways.de!ido-mysql'
    [2015-12-14 18:03:25 +0100] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
    [2015-12-14 18:03:35 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_programstatus"
    DETAIL: Key (instance_id)=(1) already exists.
    " when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES (E'1', E'1', E'1', E'mbmif.int.netways.de', E'1', E'1', 1, E'1', TO_TIMESTAMP(1450112615), E'1', E'1', E'1', E'17506', E'1', TO_TIMESTAMP(1450112594), E'v2.4.1-75-gf297ccd', TO_TIMESTAMP(1450112615))"
    [2015-12-14 18:03:35 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!
    [2015-12-14 18:03:35 +0100] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
    [2015-12-14 18:03:45 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_programstatus"
    DETAIL: Key (instance_id)=(1) already exists.
    " when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES (E'1', E'1', E'1', E'mbmif.int.netways.de', E'1', E'1', 1, E'1', TO_TIMESTAMP(1450112625), E'1', E'1', E'1', E'17506', E'1', TO_TIMESTAMP(1450112594), E'v2.4.1-75-gf297ccd', TO_TIMESTAMP(1450112625))"
    [2015-12-14 18:03:45 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-14 17:04:44 +00:00

  • Target Version set to 2.4.2

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-14 17:06:24 +00:00

  • File added debug.log.gz

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-15 10:18:57 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Include in Changelog changed from 1 to 0

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-15 11:04:50 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset da3d210.

@icinga-migration
Copy link
Author

Updated by gbeutner on 2016-02-23 09:58:31 +00:00

  • Backport? changed from Not yet backported to Already backported

@icinga-migration icinga-migration added blocker Blocks a release or needs immediate attention bug Something isn't working area/db-ido Database output labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.4.2 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/db-ido Database output blocker Blocks a release or needs immediate attention bug Something isn't working
Projects
None yet
Milestone
2.4.2
Development

No branches or pull requests
1 participant
@icinga-migration