Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #10363] Notification times w/ empty begin/end specifications prevent sending notifications #3486

Closed
icinga-migration opened this issue Oct 14, 2015 · 7 comments
Closed

[dev.icinga.com #10363] Notification times w/ empty begin/end specifications prevent sending notifications #3486

icinga-migration opened this issue Oct 14, 2015 · 7 comments
Labels
area/notifications Notification events bug Something isn't working
Milestone
2.5.0

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10363

Created by nichols356 on 2015-10-14 18:32:15 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-08-04 15:30:05 +00:00)
Target Version: 2.5.0
Last Update: 2016-08-08 11:17:18 +00:00 (in Redmine)

Icinga Version: 2.3.10
Backport?: Not yet backported
Include in Changelog: 1

I apologize in advance if I am doing this the wrong way. I believe I have run into a bug the way I have implemented notifications and wanted to share it with the icinga2 dev team in hopes that this may improve the product.

My customers need several tiers of notifications with different people notified at different tiers and "apply Notification" got a bit out of hand with the many variances from one host to another. My solution was to define all required notification variables within the host itself, using templates for frequently used notification tiers and doing one-offs for the outliers that needed something special. This has the added benefit of being able to see what notification variables are being applied to a specific host from within icingaweb2. It could be that I'm just doing this completely wrong, but I would think the way I'm doing it should work.

My problem is that it appears to be applying all the variables as I want them to, but notifications aren't happening when they are expected. Using a test host I have established that the following is what happens:

  • Host goes down, PROBLEM notification is not delivered.
  • Host comes up, RECOVERY notification is not delivered.

Alternatively...

  • Host goes down, PROBLEM notification is not delivered.
  • Send CUSTOM notification, works as expected.
  • Host comes up, RECOVERY is sent as expected.

I would think that the CUSTOM notification wouldn't work if notifications were completely broken, so I don't know why PROBLEM never sends, and RECOVERY will only send if you send a CUSTOM before it.

It looks like this in my configuration files:

object Host "appdev-test-prod" {
    address="x.x.x.x"
    import "debug-host"
    vars.notify_infra = {
        email = [ "appdev-test" ]
        interval = 1m
        delay = 0m
        window =    "24x7"
    }
}
template Host "debug-host" {
    max_check_attempts = 1
    check_interval = 30s
    retry_interval = 30s
    check_command = "hostalive"
    enable_flapping = 0
}
object TimePeriod "24x7" {
    import "legacy-timeperiod"

    display_name = "24 hours a day 7 days a week"
    ranges = {
        "monday" = "00:00-24:00"
        "tuesday" = "00:00-24:00"
        "wednesday" = "00:00-24:00"
        "thursday" = "00:00-24:00"
        "friday" = "00:00-24:00"
        "saturday" = "00:00-24:00"
        "sunday" = "00:00-24:00"
    }
}
apply Notification "host_notify_infra" to Host {
    import "email-host-notification"
    import "vars_notify_infra"

    user_groups = host.vars.notify_infra.email

    assign where host.vars.notify_infra.email
}
template Notification "email-host-notification" {
    import "host-notification-states-types"
    command = "mail-host-command"
}
template Notification "host-notification-states-types" {
    states = [ Up, Down ]
            // Possible states:
            // UP, DOWN
    types = [ Problem, Acknowledgement, Recovery, Custom ]
            // Possible types:
            // Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved
}
template Notification "vars_notify_infra" {
    interval = host.vars.notify_infra.interval
    times.begin = host.vars.notify_infra.delay
    times.end = host.vars.notify_infra.cutoff
    period = host.vars.notify_infra.window
}
object User "nichols.356" {
    import "generic-user"
    groups = [ "appdev-test" ]
    email = "nichols.356@osu.edu"
}
object UserGroup "appdev-test" {}

I think I got everything pertaining to the .conf files...

Here is what the output of "icinga2 object list" looks like

Object 'appdev-test-prod' of type 'Host':
  % declared in '/etc/icinga2/conf.d/custom/hosts/appdev/webservers.conf', lines 602:1-602:30
  * __name = "appdev-test-prod"
  * action_url = ""
  * address = "x.x.x.x"
    % = modified in '/etc/icinga2/conf.d/custom/hosts/appdev/webservers.conf', lines 604:2-604:25
  * address6 = ""
  * check_command = "hostalive"
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 29:3-29:29
  * check_interval = 30
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 27:3-27:22
  * check_period = ""
  * command_endpoint = ""
  * display_name = "appdev-test-prod"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 30:3-30:21
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 30
  * groups = [ ]
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 1
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 26:3-26:24
  * name = "appdev-test-prod"
  * notes = ""
  * notes_url = ""
  * retry_interval = 30
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 28:3-28:22
  * templates = [ "appdev-test-prod", "debug-host" ]
    % = modified in '/etc/icinga2/conf.d/custom/hosts/appdev/webservers.conf', lines 602:1-602:30
    % = modified in '/etc/icinga2/conf.d/custom/templates.conf', lines 25:1-25:26
  * type = "Host"
  * vars
    * notify_infra
      % = modified in '/etc/icinga2/conf.d/custom/hosts/appdev/webservers.conf', lines 616:3-621:4
      * delay = 0
      * email = [ "appdev-test" ]
      * interval = 60
      * window = "24x7"
  * volatile = false
  * zone = ""

Object 'appdev-test-prod!host_notify_infra' of type 'Notification':
  % declared in '/etc/icinga2/conf.d/custom/notifications/notifications-new.conf', lines 28:1-28:46
  * __name = "appdev-test-prod!host_notify_infra"
  * command = "mail-host-command"
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 24:2-24:30
  * command_endpoint = ""
  * host_name = "appdev-test-prod"
    % = modified in '/etc/icinga2/conf.d/custom/notifications/notifications-new.conf', lines 28:1-28:46
  * interval = 60
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 58:2-58:48
  * name = "host_notify_infra"
  * period = "24x7"
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 61:9-61:53
  * service_name = ""
  * states = [ 16, 32 ]
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 5:2-5:22
  * templates = [ "host_notify_infra", "email-host-notification", "host-notification-states-types", "vars_notify_infra" ]
    % = modified in '/etc/icinga2/conf.d/custom/notifications/notifications-new.conf', lines 28:1-28:46
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 22:1-22:47
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 4:1-4:54
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 57:1-57:41
  * times
    * begin = 0
      % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 59:9-59:52
    * end = null
      % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 60:9-60:53
  * type = "Notification"
  * types = [ 32, 16, 64, 8 ]
    % = modified in '/etc/icinga2/conf.d/custom/notifications/templates-new.conf', lines 8:2-8:56
  * user_groups = [ "appdev-test" ]
    % = modified in '/etc/icinga2/conf.d/custom/notifications/notifications-new.conf', lines 32:2-32:43
  * users = null
  * vars = null
  * zone = ""

icinga2.log (please also look at the attached screenshot with notations of the events)

[2015-10-14 13:01:23 -0400] information/Checkable: Checking for configured notifications for object 'appdev-test-prod'
[2015-10-14 13:01:57 -0400] information/Checkable: Checking for configured notifications for object 'appdev-test-prod'
[2015-10-14 13:02:53 -0400] information/Checkable: Checking for configured notifications for object 'appdev-test-prod'
[2015-10-14 13:03:15 -0400] information/ExternalCommandListener: Executing external command: [1444842195] SEND_CUSTOM_HOST_NOTIFICATION;appdev-test-prod;0;nichols.356;custom notification
[2015-10-14 13:03:15 -0400] information/Checkable: Checking for configured notifications for object 'appdev-test-prod'
[2015-10-14 13:03:15 -0400] information/Notification: Sending notification 'appdev-test-prod!host_notify_infra' for user 'nichols.356'
[2015-10-14 13:03:15 -0400] information/Notification: Completed sending notification 'appdev-test-prod!host_notify_infra' for checkable 'appdev-test-prod'
[2015-10-14 13:05:33 -0400] information/Checkable: Checking for configured notifications for object 'appdev-test-prod'
[2015-10-14 13:05:33 -0400] information/Notification: Sending notification 'appdev-test-prod!host_notify_infra' for user 'nichols.356'
[2015-10-14 13:05:33 -0400] information/Notification: Completed sending notification 'appdev-test-prod!host_notify_infra' for checkable 'appdev-test-prod'
[2015-10-14 13:05:44 -0400] information/DynamicObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

I hope this provides enough information to you for debugging purposes. If there is anything else I should provide, please let me know.

Attachments

Changesets

2016-08-04 15:27:35 +00:00 by mfriedrich 43a177f

Fix that notifications with empty times.{begin,end} are not sent

fixes #10363
@icinga-migration
Copy link
Author

Updated by nichols356 on 2015-10-26 20:17:56 +00:00

  • File added relevant_debug.log

I just some time to recreate this on a fresh install. It appears I left out a few parts of my config. Oops. I disabled the hosts.conf file and added the following chain of configs into a "test.conf" file to simplify debugging.

--version

[root@icinga-test conf.d]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.3.11)

Copyright (c) 2012-2015 Icinga Development Team (https://www.icinga.org)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /var/run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /var/run/icinga2/icinga2.pid
  Application type: icinga/IcingaApplication

System information:
  Operating system: Linux
  Operating system version: 2.6.32-573.7.1.el6.x86_64
  Architecture: x86_64
  Distribution: CentOS release 6.7 (Final)

test.conf

object Host "appdev-test-prod" {
    address="140.254.38.125"
    import "debug-host"
    vars.notify_infra = {
        email = [ "appdev-test" ]
        interval = 1m
        delay = 0m
        window =    "24x7"
    }
}
template Host "debug-host" {
    max_check_attempts = 1
    check_interval = 30s
    retry_interval = 30s
    check_command = "hostalive"
    enable_flapping = 0
}
apply Notification "host_notify_infra" to Host {
    import "email-host-notification"
    import "vars_notify_infra"

    user_groups = host.vars.notify_infra.email

    assign where host.vars.notify_infra.email
}
template Notification "email-host-notification" {
    import "host-notification-states-types"
    command = "mail-host-command"
}
template Notification "host-notification-states-types" {
    states = [ Up, Down ]
            // Possible states:
            // UP, DOWN
    types = [ Problem, Acknowledgement, Recovery, Custom ]
            // Possible types:
            // Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved
}
template Notification "vars_notify_infra" {
    interval = host.vars.notify_infra.interval
    times.begin = host.vars.notify_infra.delay
    times.end = host.vars.notify_infra.cutoff
    period = host.vars.notify_infra.window
}
object User "nichols.356" {
    import "generic-user"
    groups = [ "appdev-test" ]
    email = "nichols.356@osu.edu"
}
object UserGroup "appdev-test" {}

object NotificationCommand "mail-host-command" {
        import "plugin-notification-command"
        import "host-command-template"
        command = [ SysconfDir + "/icinga2/scripts/mail-host-notification.sh" ]
        env +=  {
                USEREMAIL = "$user.email$"
                }
}

template NotificationCommand "host-command-template" {
//      import "plugin-notificaiton-command"    ## it does not like importing plugin-notification-command in a template, so I must directly import it to each NotificationCommand object
        env =   {
                NOTIFICATIONTYPE = "$notification.type$"
                HOSTALIAS = "$host.display_name$"
                HOSTADDRESS = "$address$"
                HOSTSTATE = "$host.state$"
                LONGDATETIME = "$icinga.long_date_time$"
                SHORTDATETIME = "$icinga.short_date_time$"
                HOSTOUTPUT = "$host.output$"
                NOTIFICATIONAUTHORNAME = "$notification.author$"
                NOTIFICATIONCOMMENT = "$notification.comment$"
                HOSTDISPLAYNAME = "$host.display_name$"
//              USEREMAIL = "$user.email$"
                }
}

again, the same happens while recreating this issue:

I did get a debug log and wrote down the specific times of interest. I trimmed it to start just before I pulled the plug and stop right after i received the recovery notification, so it should contain all relevant information.

Here is the timeline:

16:00:50 unplugged - no notification
16:02:10 plugged back in - no notification

16:02:58 unplugged - no notification
16:04:15 custom notification sent
16:04:30 custom notification received!
16:04:55 replugged
16:05:25 recovery notification received!

@icinga-migration
Copy link
Author

Updated by nichols356 on 2015-10-27 18:20:23 +00:00

More progress on this. It appears that if I comment out the line

    times.end = host.vars.notify_infra.cutoff

in my config above, it resolves the issue. Also, this can be resolved by setting host.vars.notify_infra.cutoff to something like 60m.

If times.end doesn't exist, notification works.
If times.end is null, notification doesn't work.
If times.end is set to a value, notification works.

I would expect that if times.end is null, it should behave as if times.end doesn't exist.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-02-24 22:44:30 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-03-04 15:35:53 +00:00

  • Parent Id set to 11311

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-08-04 15:27:22 +00:00

  • Subject changed from notification not triggered to Notification times w/ empty begin/end specifications prevent sending notifications

  • Target Version set to 2.5.0

    object Host "10363-host" {
    check_command = "dummy"
    check_interval = 10m
    retry_interval = 10m
    max_check_attempts = 2
    }

    object User "10363-user" {
    email = "michael.friedrich@netways.de"
    }

    apply Notification "10363-notification" to Host {
    import "mail-host-notification"

    times.begin = 0
times.end = null

users = [ "10363-user" ]

assign where host.name == "10363-host"

    }

    [2016-08-04 15:45:09 +0200] notice/Checkable: State Change: Checkable 10363-host hard state change from DOWN to DOWN detected.
    [2016-08-04 15:45:09 +0200] information/Checkable: Checking for configured notifications for object '10363-host'
    [2016-08-04 15:45:09 +0200] debug/Checkable: Checkable '10363-host' has 1 notification(s).
    [2016-08-04 15:45:09 +0200] notice/Notification: Attempting to send notifications for notification object '10363-host!10363-notification'.
    [2016-08-04 15:45:09 +0200] notice/Notification: Not sending notifications for notification object '10363-host!10363-notification': after escalation range

While at it, also fixed printing the values in the debug log messages.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-08-04 15:30:06 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 43a177f.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-08-08 11:17:19 +00:00

  • Parent Id deleted 11311

@icinga-migration icinga-migration added bug Something isn't working area/notifications Notification events labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.5.0 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/notifications Notification events bug Something isn't working
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
1 participant
@icinga-migration