You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Created by aiwku1277 on 2016-02-16 16:34:15 +00:00
Assignee: (none)
Status: Closed (closed on 2016-08-04 15:33:19 +00:00)
Target Version: (none)
Last Update: 2016-08-08 11:17:22 +00:00 (in Redmine)
Icinga Version: 2.4.1
Backport?: Not yet backported
Include in Changelog: 1
I am running two Icinga instances located in master zone on Centos 6.7. Frequently, Icinga will send the first reminder alert only moments after the initial alert. The alerts are coming from the same host and logs show Icinga writing that the first alerts were sent to the db before sending the second (thus the second email is a reminder notification). It doesn’t appear to be occurring with a specific host/service or for a particular interval. Sometimes the reminder is sent only a minute after the first notification, sometimes it’s several minutes. It only appears to be occurring with the first reminder notification.
Note: I did recently add an "if" statement to my notification config. Issue occurred prior to that addition and that feature works as expected (except for this reminder issue).
#####Notification Config######
apply Notification "mail-support-host" to Host {
import "mail-host-notification"
user_groups = ["support"]
assign where host.vars.owner == "support"
}
apply Notification "mail-support-service" to Service {
import "mail-service-notification"
user_groups = ["support"]
if (match ("swap*", service.name)) {
interval = 1h
}
assign where host.vars.owner == "support"
ignore where service.name == "openmanage"
}
#####Notification Template######
template Notification "mail-host-notification" {
command = "mail-host-notification"
states = [ Down ]
types = [ Problem, Custom,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
period = "24x7"
interval = 30m
}
template Notification "mail-service-notification" {
command = "mail-service-notification"
states = [ Critical, Unknown ]
types = [ Problem, Custom,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
period = "24x7"
interval = 30m
}
#####Host and Service Template######
template Host "generic-host" {
max_check_attempts = 3
check_interval = 5m
retry_interval = 1m
check_command = "hostalive"
}
template Service "generic-service" {
max_check_attempts = 3
check_interval = 5m
retry_interval = 1m
}
#####Log of Recent Occurence######
[2016-02-15 02:32:01 -0500] notice/Process: Running command '/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30': PID 9332
[2016-02-15 02:32:01 -0500] debug/CheckerComponent: Check finished for object 'myhost1!swap'
[2016-02-15 02:32:03 -0500] notice/Process: PID 9332 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30') terminated with exit code 2
[2016-02-15 02:32:03 -0500] debug/Checkable: No origin or local origin for object 'myhost1!swap', setting icingaa as check_source.
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:32:03 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:32:03 -0500] debug/DbEvents: add checkable check history for 'myhost1!swap'
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:32:03 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:32:03 -0500] debug/DbObject: Endpoint node: 'icingaa' status update for 'myhost1!swap'
[2016-02-15 02:32:04 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521523), '334273', 1, 'SERVICE ALERT: myhost1;swap;CRITICAL;SOFT;3;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521523), '65536', 912)
[2016-02-15 02:32:47 -0500] debug/DbEvents: Updating reachability for checkable 'myhost1!swap': reachable.
[2016-02-15 02:32:52 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:32:52 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:01 -0500] debug/CheckerComponent: Executing check for 'myhost1!swap'
[2016-02-15 02:33:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:01 -0500] notice/Process: Running command '/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30': PID 9433
[2016-02-15 02:33:01 -0500] debug/CheckerComponent: Check finished for object 'myhost1!swap'
[2016-02-15 02:33:02 -0500] notice/Process: PID 9433 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30') terminated with exit code 2
[2016-02-15 02:33:02 -0500] debug/Checkable: No origin or local origin for object 'myhost1!swap', setting icingaa as check_source.
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add checkable check history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:02 -0500] debug/DbObject: Endpoint node: 'icingaa' status update for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add state change history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] notice/Checkable: State Change: Checkable myhost1!swap hard state change from CRITICAL to CRITICAL detected.
[2016-02-15 02:33:02 -0500] information/Checkable: Checking for configured notifications for object 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/Checkable: Checkable 'myhost1!swap' has 1 notification(s).
[2016-02-15 02:33:02 -0500] notice/Notification: Attempting to send notifications for notification object 'myhost1!swap!mail-support-service'.
[2016-02-15 02:33:02 -0500] debug/Notification: Type 'Problem', TypeFilter OK, Warning, Unknown, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: State 'Critical', StateFilter Unknown, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=12)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:02 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user1'
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:02 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user2'
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:02 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user3'
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:02 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user4'
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:02 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:02 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user5'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add notification history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user1'.
[2016-02-15 02:33:02 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user2'.
[2016-02-15 02:33:02 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user3'.
[2016-02-15 02:33:02 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user4'.
[2016-02-15 02:33:02 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user5'.
[2016-02-15 02:33:02 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '939426', 1, 'SERVICE ALERT: myhost1;swap;CRITICAL;HARD;1;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '65536', 912)
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:02 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:02 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:03 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '942856', 1, 'SERVICE NOTIFICATION: user1;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '524288', 912)
[2016-02-15 02:33:03 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '943273', 1, 'SERVICE NOTIFICATION: user3;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '524288', 912)
[2016-02-15 02:33:03 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '946856', 1, 'SERVICE NOTIFICATION: user5;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '524288', 912)
[2016-02-15 02:33:03 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '948872', 1, 'SERVICE NOTIFICATION: user2;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '524288', 912)
[2016-02-15 02:33:03 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521582), '949467', 1, 'SERVICE NOTIFICATION: user4;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521582), '524288', 912)
[2016-02-15 02:33:17 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:33:17 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:33:17 -0500] notice/NotificationComponent: Attempting to send reminder notification for object 'myhost1!swap'
[2016-02-15 02:33:17 -0500] notice/Notification: Attempting to send notifications for notification object 'myhost1!swap!mail-support-service'.
[2016-02-15 02:33:17 -0500] debug/Notification: Type 'Problem', TypeFilter OK, Warning, Unknown, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: State 'Critical', StateFilter Unknown, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=12)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:17 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user1'
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:17 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user2'
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:17 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user3'
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:17 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user4'
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, Type 'Problem', TypeFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FType=32, TypeFilter=47)
[2016-02-15 02:33:17 -0500] debug/Notification: User notification, State 'Critical', StateFilter OK, Warning, Unknown, Up, Down, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart and FlappingEnd (FState=4, StateFilter=-1)
[2016-02-15 02:33:17 -0500] information/Notification: Sending notification 'myhost1!swap!mail-support-service' for user 'user5'
[2016-02-15 02:33:17 -0500] debug/DbEvents: add notification history for 'myhost1!swap'
[2016-02-15 02:33:17 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user1'.
[2016-02-15 02:33:17 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user2'.
[2016-02-15 02:33:17 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user3'.
[2016-02-15 02:33:17 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user4'.
[2016-02-15 02:33:17 -0500] debug/DbEvents: add contact notification history for service 'myhost1!swap' and user 'user5'.
[2016-02-15 02:33:17 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:17 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:17 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:17 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:17 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521597), '980988', 1, 'SERVICE NOTIFICATION: user1;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521597), '524288', 912)
[2016-02-15 02:33:17 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521597), '987013', 1, 'SERVICE NOTIFICATION: user2;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521597), '524288', 912)
[2016-02-15 02:33:18 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:18 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:18 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:18 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:18 -0500] debug/DbEvents: add log entry history for 'myhost1!swap'
[2016-02-15 02:33:18 -0500] information/Notification: Completed sending notification 'myhost1!swap!mail-support-service' for checkable 'myhost1!swap'
[2016-02-15 02:33:18 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521598), '2877', 1, 'SERVICE NOTIFICATION: user4;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521598), '524288', 912)
[2016-02-15 02:33:18 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521598), '9017', 1, 'SERVICE NOTIFICATION: user3;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521598), '524288', 912)
[2016-02-15 02:33:18 -0500] debug/IdoMysqlConnection: Query: INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (1, FROM_UNIXTIME(1455521598), '21847', 1, 'SERVICE NOTIFICATION: user5;myhost1;swap;PROBLEM (CRITICAL);by_ssh_swap;SWAP CRITICAL - 1% free (3 MB out of 8189 MB) ', FROM_UNIXTIME(1455521598), '524288', 912)
[2016-02-15 02:34:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:01 -0500] debug/CheckerComponent: Executing check for 'myhost1!swap'
[2016-02-15 02:34:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:01 -0500] notice/Process: Running command '/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30': PID 9697
[2016-02-15 02:34:01 -0500] debug/CheckerComponent: Check finished for object 'myhost1!swap'
[2016-02-15 02:34:02 -0500] notice/Process: PID 9697 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30') terminated with exit code 2
[2016-02-15 02:34:02 -0500] debug/Checkable: No origin or local origin for object 'myhost1!swap', setting icingaa as check_source.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] debug/DbEvents: add checkable check history for 'myhost1!swap'
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] debug/DbObject: Endpoint node: 'icingaa' status update for 'myhost1!swap'
[2016-02-15 02:34:02 -0500] notice/Process: PID 9697 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30') terminated with exit code 2
[2016-02-15 02:34:02 -0500] debug/Checkable: No origin or local origin for object 'myhost1!swap', setting icingaa as check_source.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] debug/DbEvents: add checkable check history for 'myhost1!swap'
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:34:02 -0500] debug/DbObject: Endpoint node: 'icingaa' status update for 'myhost1!swap'
[2016-02-15 02:34:34 -0500] debug/DbEvents: Updating reachability for checkable 'myhost1!swap': reachable.
[2016-02-15 02:34:37 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:34:37 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:35:10 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:35:10 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:36:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:01 -0500] debug/CheckerComponent: Executing check for 'myhost1!swap'
[2016-02-15 02:36:01 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:36:01 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:01 -0500] notice/Process: Running command '/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30': PID 9894
[2016-02-15 02:36:01 -0500] debug/CheckerComponent: Check finished for object 'myhost1!swap'
[2016-02-15 02:36:02 -0500] notice/Process: PID 9894 ('/usr/lib64/nagios/plugins/check_by_ssh' '-C' '/usr/lib64/nagios/plugins/check_swap -w 20% -c 10%' '-H' 'xx.xx.xx.xx' '-l' 'root' '-t' '30') terminated with exit code 2
[2016-02-15 02:36:02 -0500] debug/Checkable: No origin or local origin for object 'myhost1!swap', setting icingaa as check_source.
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:02 -0500] debug/DbEvents: add checkable check history for 'myhost1!swap'
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:36:02 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:36:02 -0500] debug/DbObject: Endpoint node: 'icingaa' status update for 'myhost1!swap'
[2016-02-15 02:37:47 -0500] debug/DbEvents: Updating reachability for checkable 'myhost1!swap': reachable.
[2016-02-15 02:37:52 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:37:52 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
[2016-02-15 02:39:34 -0500] debug/DbEvents: Updating reachability for checkable 'myhost1!swap': reachable.
[2016-02-15 02:39:37 -0500] notice/Dependency: Dependency 'myhost1!swap!Host-Service Checks' passed: Parent host 'myhost1' matches state filter.
[2016-02-15 02:39:37 -0500] notice/Dependency: Dependency 'myhost1!swap!ssh Checks' passed: Parent service 'myhost1!ssh' matches state filter.
This issue has been migrated from Redmine: https://dev.icinga.com/issues/11170
Created by aiwku1277 on 2016-02-16 16:34:15 +00:00
Assignee: (none)
Status: Closed (closed on 2016-08-04 15:33:19 +00:00)
Target Version: (none)
Last Update: 2016-08-08 11:17:22 +00:00 (in Redmine)
I am running two Icinga instances located in master zone on Centos 6.7. Frequently, Icinga will send the first reminder alert only moments after the initial alert. The alerts are coming from the same host and logs show Icinga writing that the first alerts were sent to the db before sending the second (thus the second email is a reminder notification). It doesn’t appear to be occurring with a specific host/service or for a particular interval. Sometimes the reminder is sent only a minute after the first notification, sometimes it’s several minutes. It only appears to be occurring with the first reminder notification.
Note: I did recently add an "if" statement to my notification config. Issue occurred prior to that addition and that feature works as expected (except for this reminder issue).
Relations:
The text was updated successfully, but these errors were encountered: