Previous versions of Lithium provided Report and Cleared notifications for when an Incident was raised.
Whilst this ensured the notifications were received, there were two scenarios not handled well: recurring (flapping) incidents and change in incident severity.
In the case of a flapping/recurring incident, repeated Report, Cleared, Report, Cleared, Report, etc alerts would be generated each time the condition occurred. Alternatively, when an incident moved from one severity to another (e.g. Warning to Impaired) a separate Cleared and Report alert was generated to clear the original condition and raise the new condition.
In Lithium 5.0, Incident notifications have been extended to include New, Recurring, Transition and Cleared.
The goal of these changes has been to significantly reduce the amount of alerts generated by Lithium to facilitate useful Email Alerts as well as to support use of Push Notification technology.
New Incident
When an Incident is raised for the first time, a New Incident alert is generated by way of an Action Script. The standard Action Script that ships with Lithium will send an email to a configured recipient list to raise the alarm.
Recurring Incident
If the Incident clears (returns to normal) but occurs again within 4 hours then it is deemed to be a Recurring Incident.
When a Recurring Incident is raised, a new type of command (report_recurring) is sent to the configured Actions to generate alerts specific to recurring incidents.
The Email Alert Action Script is configured to send an email alert for the Recurring Incident on the first recurrence only. That is, you will receive one email to alert you to the recurring incident and no further alerts until the Incident remains cleared and inactive for 4 hours.
Transition
If the Metric that an Incident refers to changes status from one non-normal status to another (e.g Impaired to Critical) then it is treated as an Incident Transition.
Rather than generating separate Cleared and Report alerts for the old and new condition, a single new command (transition) is sent to the configured Actions to raise the alert than an Incident has changed status but has not returned to Normal.
The Email Alert Action Script is configured to send out an Email Alert when an Incident reaches a new highest or lowest severity only. This ensures that the volume of email alerts is kept under control if an Incident is rapidly changing state.
Cleared
The first time an Incident is raised and then returned to normal (cleared), a standard Cleared event is sent to the configured Action Scripts.
However, a new clear_recurring command is used when a Recurring Incident has been inactive for 4 hours and is hence deemed to have cleared without any further recurrence.