[dev.icinga.com #905] add a command to disable notifications program-wide with expire time as scheduled event; classic ui & idoutils reflected

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/905

Created by rbrinkmo on 2010-10-15 13:46:19 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2012-08-30 14:51:28 +00:00)
Target Version: 1.8
Last Update: 2012-08-30 14:51:28 +00:00 (in Redmine)

---

.

Attachments

• icinga_classicui_905_notifications_disabled_expire_time_command_form.png mfriedrich - 2012-07-30 16:32:42 +00:00
• icinga_classicui_905_notifications_disabled_expire_time.png mfriedrich - 2012-07-30 16:32:42 +00:00
• icinga_classicui_906_notifications_disabled_expire_time_not_set.png mfriedrich - 2012-07-30 16:32:42 +00:00

Changesets

2012-07-30 20:19:26 +00:00 by mfriedrich 5b4e773

core: add a command to disable notifications program-wide with expire time as scheduled event; add commands to cmd.cgi, idoutils programstatus table #905

this adds a new command

DISABLE_NOTIFICATIONS_EXPIRE_TIME;<schedule_time>;<expire_time>

which schedules a new event for expiring disabled notifications ==
re-enable all notifications program-wide.
you can override that expire with ENABLE_NOTIFICATIONS command, not
having an expiry after that. the event for expiry will ignore future
timestamps > event->run_time, so the expiration only takes place on the
correct timestamp.

extinfo.cgi Process Info view adds a new row with "Disable Notifications
Expire Time", showing a readable date time format when notifications are
disabled, and expire time was set.

for cgi.cfg, the default duration can be set as well (default is 1 day).

default_expiring_disabled_notifications_duration=86400

on the event broker side of life, this will be made available within the
programstatus data and updates (which normally happen quite often).
idomod and ido2db have that added to their data structs as well, the
rdbms will get a new column in the programstatus table (cut due to
30char limit of oracle):

disable_notif_expire_time

if one wants to set a start_time, this is currently not
possible/implemented.

other than that, extensive testing is required.

refs #905

2012-08-25 13:22:26 +00:00 by mfriedrich 54b8eff

fix tests, broken by previous commits

refs #905
refs #2878
refs #2725

2012-08-25 23:47:53 +00:00 by mfriedrich 4468b25

idoutils: fix faulty timestamp from #905 on oracle

refs #905

---

Relations:

• relates #905
• relates #905

[icinga-migration]

Updated by ricardo on 2010-11-05 10:48:32 +00:00

I would like to see that in Icinga.

And implementation is pretty similar to Acknowledges with expire time, execpt that we add a new Command + new timed event.

[icinga-migration]

Updated by mfriedrich on 2010-11-05 11:27:17 +00:00

it still needs a change in the NEB API, and as proposed in #770 our own IEB putting data then into IDO or similar.

if this feature should be implemented, imho an IEB is a must before. thoughts?

[icinga-migration]

Updated by ricardo on 2010-11-05 11:36:20 +00:00

I don't think that this will break the NEB as well.

cause it will be 2 seperated commands.

The ones witch have comments allready, stay as they are. The one's with new, forced comments get submitted as usual and an additional comment command (sounds funny) will be submitted on top of it.

[icinga-migration]

Updated by mfriedrich on 2011-03-24 21:30:43 +00:00

• Status changed from New to Assigned
• Assigned to set to ricardo
• Target Version set to 1.5

i would consider it for icinga 1.5 or 1.6

[icinga-migration]

Updated by mfriedrich on 2011-06-26 15:56:16 +00:00

the question which comes to mind is - this should only be a timely limited command replacement for disable notifications on hosts/services, but by adding another end_time then.

wouldn't it be more sufficient to schedule a downtime, supressing the notifications either way? having set the duration (flexible) and start/endtime (fixed) this would allow basically the same goal to be reached, don't you think?

[icinga-migration]

Updated by mfriedrich on 2011-07-12 18:48:02 +00:00

• Target Version changed from 1.5 to 1.6

[icinga-migration]

Updated by rbrinkmo on 2011-07-26 15:38:05 +00:00

A scheduled downtime has a different meaning in our SLA reporting. It is like a maintenance (the host/service don't have to be available). That's the reason why we need a different function just to disable the notification for a defined time for those hosts and services which should be available from the view of SLA Reporting.

It would be great if this information also would be available in the idodb.

[icinga-migration]

Updated by mfriedrich on 2011-09-22 18:27:54 +00:00

do you think it would be possible for you to implement that as well next to #770 @ricardo ? if so i would take care about the idoutils part and the docs / web thingies as well.

[icinga-migration]

Updated by mfriedrich on 2011-10-25 09:06:58 +00:00

• Target Version changed from 1.6 to 1.7

scheduled for later on.

[icinga-migration]

Updated by mfriedrich on 2012-01-27 20:39:44 +00:00

• Status changed from Assigned to Feedback
• Target Version changed from 1.7 to 1.8

[icinga-migration]

Updated by mfriedrich on 2012-07-30 15:53:22 +00:00

• Subject changed from adding a command and function for timely limited deactivation of Notifications to add a command to disable notifications with expire time as scheduled event; classic ui & idoutils reflected
• Status changed from Feedback to Assigned
• Assigned to changed from ricardo to mfriedrich

i don't think this should be done on a host or service basis, but globally for all notifications, as we can re-use inner core functionality on this.

• enable_notifications as variable indicator
• enable_all_notifications() as reset when the time expired
• disable_all_notifications() when the command happens, plus scheduling a new expire event

this requires the following changes

core

• new event: EVENT_EXPIRE_DISABLED_NOTIFICATIONS
• new command: DISABLE_NOTIFICATIONS_EXPIRE_TIME;<scheduled_time>;<end_time>
  (scheduled_time is ignored, but must be set as DISABLE_NOTIFICATIONS requires this as well!)
• log a message when the disabled notifications will expire, inkl readable timestamp
• log a message on expire event, that notifications are now re-enabled
• retention.dat writes disable_notifications_expire_time
• retention.dat reads disable_notifications_expire_time, and schedules a new event, if the time did not yet expire (same as ack with expiry time), and notifications are still disabled
• status.dat reads/writes disable_notifications_expire_time
• disable_notifications_expire_time is classified as program_status data

cgis

• disable_notifications_expire_time will be read from status.dat
• on DISABLE_NOTIFICATIONS command form, the expire time can be ticked, and js shows the expire time textbox to be filled. it then sends a different command, namely DISABLE_NOTIFICATIONS_EXPIRE_TIME.
• the default cgi.cfg option is default_expiring_disabled_notifications_duration=86400

idoutils

• idomod passes the variable to ido2db, as protoapi type: IDO_DATA_DISABLED_NOTIFICATIONS_EXPIRE_TIME (also IDO_MAX_DATA_TYPES +1)

• ido2db will write that to the program_status table, expanding the data fields

• short name (oracle's 30 char limit) for all: disable_notif_expire_time

  mysql> select * from icinga_programstatus;
  +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------+------------------------------+-----------------------------------+---------------------------+
  | programstatus_id | instance_id | status_update_time | program_start_time | program_end_time | is_currently_running | process_id | daemon_mode | last_command_check | last_log_rotation | notifications_enabled | active_service_checks_enabled | passive_service_checks_enabled | active_host_checks_enabled | passive_host_checks_enabled | event_handlers_enabled | flap_detection_enabled | failure_prediction_enabled | process_performance_data | obsess_over_hosts | obsess_over_services | modified_host_attributes | modified_service_attributes | global_host_event_handler | global_service_event_handler | disable_notifications_expire_time | disable_notif_expire_time |
  +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------+------------------------------+-----------------------------------+---------------------------+
  | 1 | 1 | 2012-07-30 17:51:11 | 2012-07-30 17:47:52 | 0000-00-00 00:00:00 | 1 | 26310 | 1 | 2012-07-30 17:51:11 | 0000-00-00 00:00:00 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | | | 0000-00-00 00:00:00 | 2012-07-30 16:45:09 |
  +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------

what's not possible

• set a start_time/end_time window. such a command must be called now() lasting til end_time (one could still schedule a cronjob pushing that command at night where maintenance happens)

• add a comment to that command won't be necessary. the core will log start/stop event to syslog, as well as the disable_notifications_expire_time will be available on status.dat / idoutils for gui selections. to gather an insight who did it, enable cmd.cgi logging in the first place.

  tail -f /var/log/messages | grep -i Disable

  Jul 30 16:40:20 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343659220;1343659509
  Jul 30 16:40:20 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 16:45:09 2012 (1343659509)
  Jul 30 16:45:10 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.

might require more tests and bugfixes for now

[icinga-migration]

Updated by mfriedrich on 2012-07-30 16:32:42 +00:00

• File added icinga_classicui_905_notifications_disabled_expire_time_command_form.png
• File added icinga_classicui_905_notifications_disabled_expire_time.png
• File added icinga_classicui_906_notifications_disabled_expire_time_not_set.png

[icinga-migration]

Updated by mfriedrich on 2012-07-30 16:36:25 +00:00

the ENABLE_NOTIFICATIONS command should possibly clear

• disable_notifications_expire_time
• the scheduled event by that given time

question remains - how to get the timed_event for that given type and time in order to safely delete it? the event itsself should only happen if the time is not cleared before.

[icinga-migration]

Updated by mfriedrich on 2012-07-30 19:30:22 +00:00

• the expiry will only happen, if disable_notifications_expire_time > 0 (others indicate a forced override by an external ressource)
• ENABLE_NOTIFICATIONS will case a forced expiry, setting disable_notifications_expire_time = 0
• if the normal expiry happens, the call to enable_all_notifications must have disable_notifications_expire_time=0 set already to prevent further logging

there might be a race condition on

• disabling notifications, expire time, schedule expire event
• forced expire by enable notifications
• leave scheduled event in the queue
• disable again with expire time, setting a new expire time

this can be resolved on checking if the expire_time > event_time->run_time, only letting valid expiry times but not cleaning those in the future.

see the test below, which shows 2 events, but only the last one actually happens.

Jul 30 21:10:46 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343675446;1343676025
Jul 30 21:10:46 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:20:25 2012 (1343676025).
Jul 30 21:10:56 icinga-dev icinga: Disabled Notifications forced expire, re-enabled notifications.
Jul 30 21:11:22 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343675482;1343676305
Jul 30 21:11:22 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:25:05 2012 (1343676305).
Jul 30 21:25:06 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.

showing that the single one works as well with those adaptions.

Jul 30 21:26:31 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343676391;1343676496
Jul 30 21:26:31 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:28:16 2012 (1343676496).
Jul 30 21:28:17 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.

[icinga-migration]

Updated by mfriedrich on 2012-07-30 19:54:03 +00:00

• Subject changed from add a command to disable notifications with expire time as scheduled event; classic ui & idoutils reflected to add a command to disable notifications program-wide with expire time as scheduled event; classic ui & idoutils reflected

[icinga-migration]

Updated by mfriedrich on 2012-07-30 19:59:11 +00:00

hm, i see this was not the original feature request. misread that. follow the host/service notification expiry and discussion on #2919 (with #2473 and #2474 i do not think that this will be necessary, if you can see that someone disabled notifications on a host or service, even a reset button is there already).

[icinga-migration]

Updated by mfriedrich on 2012-08-01 02:33:56 +00:00

• Status changed from Assigned to Feedback
• Done % changed from 0 to 90

[icinga-migration]

Updated by mfriedrich on 2012-08-30 14:51:28 +00:00

• Status changed from Feedback to Resolved
• Done % changed from 90 to 100