Lithium is a suite of network, server and storage monitoring applications with support for industry standard SNMP and proprietary monitoring protocols including those used by Apple’s Xserve, Xsan and Xserve RAID devices.

Scroll down for step-by-step instructions...

Thresholds and Alerts Tutorial

Lithium ships with a wide range of default thresholds referred to as Triggers. Triggers specify conditions that result in an Incident or alert being raised by Lithium. When an Incident is raised it is displayed in Console and typically an Action fires to send out an email, iPhone push notification or other user-defined tasks.

The conditions or thresholds defined by a Trigger can be edited and enabed or disabled on a per-Trigger basis or a powerful rule-based system can be used to make wide-ranging changes across multiple devices for common object types.

Turn Off Unwanted Alerts

When a Device is added to Lithium, the default Triggers will be applied. This can result in some unwanted Incidents being reported for conditions that are not actually indicative of a fault. The most common example is a LAN Switch with ports that may be up or down depending on whether a workstation is connected to them. Lithium will report an Incident for the port being down, but this is not actually a real problem.

Unwanted Incidents can be turned off by selecting the Device in Console and clicking on the Incident Flag icon in the bottom right of the window and selecting 'Review Active Triggers'. Alternatively, with the Device selected in Console you can also click on 'Device -> Review Active Triggers' in the Menu Bar. The 'Review Active Triggers' sheet will be displayed showing a list of all active fault conditions for this Device. Select which of the Incident you want to permanently disable for this Device and then click 'Disable Selected'. Those fault conditions will then clear and will not recur for this Device.

Adjust Thresholds and Delays

Each individual Trigger threshold, condition and delay can be adjusted or disabled through Lithium Console. This process is referred to as Trigger Tuning. Trigger Tuning is done on a per-Object basis; for example for one particular network interface or one particular storage volume.

Click on the Object you want to adjust the Triggers for and then Command-Click on the Object and select 'Trigger Tuning -> Adjust Triggers'. Alternatively, with the Object selected you can also click on 'Monitored Entity -> Adjust Triggers' from the Menu Bar. Either option will display the Trigger Tuning sheet for adjusting the Triggers specific to that object.

The upper pane of the Trigger Tuning sheet lists the Trigger Sets that are applied to this Object. A Trigger Set is a group of Triggers that are all applied to the same Metric. For example, the Operational State Trigger Set applied to a Network Interface Object defines multiple Triggers that are all applied to the Operational State Metric for that interface. These Triggers will raise an Incident for different values of the Operational State Metric including 'Down', 'Not Present', etc. The lower pane shows more information about the selected Trigger Set including the individual Trigger condition, value and delay settings.

The Condition, Values and Duration (Delay) can be edited directly for each trigger by clicking on and editing the value through the Trigger Tuning sheet. The Duration is the length of time in seconds that the condition describes must be present before an Incident is raised by Lithium. This allows you to set a hold-down delay for transient events where Lithium is only required to raise an alter if the condition is present for a continuous length of time.

Blanket and Granular Threshold Adjustment

Lithium uses a rule-based system for controlling whether or not a Trigger Set is applied to a particular object as well as what the condition, value and duration should be for each Trigger as it is applied to each individual Object. The rule-based system allows you to make blanket or far-reaching Trigger adjustments that may affect multiple Objects or even multiple Devices at once. The rule-based system works on a most-specific-rule matching algorithm so you can setup a blanket "Do Not Apply" rule but then override that for particular Objects with a more-specific "Apply" rule for a given Trigger Set.

The rule-based side of Trigger Tuning can be accessed directly through the 'Application Rules' and 'Value Rules' tabs in the lower pane of the Trigger Tuning sheet. The rules shown in either tab are specific to the Trigger Set that is selected in the upper pane. An 'Application Rule' controls whether or not the selected Trigger Set is applied to an Object, where as a 'Value Rule' controls the Value(s) and Duration applied to the Trigger when it is used with a particular Object. Each Rule can be defined as applying to a specific Site, Device or Object or to a wild-card value that will match any of those fields. This is how far-reaching blanket rules vs. more specific granular rules are defined.

To illustrate the use of Trigger Tuning rules, consider the example given earlier of a LAN Switch being monitored by Lithium. This switch will have one up-link port that must be up at all times as well as many other ports that may be up or down at any given time. Obviously, you want Lithium to report a fault if the uplink port is down. However, you don't want to be flooded with alerts for the workstation ports when someone connects or disconnects from the LAN. This can be setup by creating two 'Application Rules' for the 'Operational State' Trigger Set belonging to one of the Network Interface Objects on the switch."

For this example, select a Device in Lithium that has Network Interfaces monitored -- a LAN switch would be perfect for this example. Click on the 'Network Interfaces' Container and then select any of the Network Interface Objects. Control-Click on the selected Object (i.e a specific network interface) and select "Trigger Tuning -> Adjust Triggers" from the pop-up menu. In the Trigger Tuning sheet displayed, select the 'Operational State' trigger set in the top pane and then select the 'Application Rules' tab. Click on the 'Add Rule' button in the bottom-right of the sheet to display the 'Add New Application Rule' sheet.

The first rule we'll add is the blanket "Do Not Apply" rule that will allow to all interfaces. In the sheet, change the 'Object' Criteria to '*ALL*' and ensure that the Action is set to 'Do NOT Apply Trigger Set'. Then click 'Add' to add the new rule. By adding that rule, you have told Lithium to NOT apply the 'Operational State' Trigger Set to any Network Interfaces on this Device. That is, no Incidents will be raised for the Operational Status of any Network Interface. Now we'll add the more specific rule for the uplink interface that should have the Operational State monitored.

Click on 'Close' to close the Trigger Tuning sheet for that Object and then select another Network Interface object, i.e another Network Interface on the Device. Control-Click on that Object and select "Adjust Triggers -> Trigger Tuning" from the pop-up menu. The Trigger Tuning sheet will appear for that Object. Ensure the 'Operational State' Trigger Set is selected then click on the 'Application Rules' tab. Click on 'Add Rule' and this time, ensure that the Object criteria is set to the specific interface, change the Action to 'Apply Trigger Set' and click 'Add'. This rule is more specific in that it does not contain any wild card matches and hence it will over-ride the less specific rule that was applied to all Interfaces. By adding this second rule you have told Lithium that the 'Operational State' Trigger Set is to be applied just to this specific Network Interface Object.

Actions

Actions are user-configured tasks Lithium performs when an Incident raised, i.e when a fault condition is present or threshold exceeded. Actions use Action Scripts to perform tasks such as Email Alerts or to dispatch an iPhone Push Notification. Action Scripts can be written in any language executable on the host running Lithium Core. Lithium ships with built-in Action scripts for Email Alerts and iPhone Push Notifications.

Email Alerts

During the initial setup of Lithium Core you can add a default Email Alert Action. To add an additional Email Alert action, click on the '+' button in the bottom-left of the Console Window and select "New Action for Customer...", the desired Lithium Customer and click on 'Email Alert Action'. Alternatively, you can select the Customer (or a member Site or Device) and click on 'Customer -> Add New Action -> Email Alert Action' from the Menu Bar. Either option will display the 'Add New Email Alert' sheet.

The 'Add New Email Alert' sheet allows you to configure a description for the Action and the protocol-specific Action configuration for Email Alerts. This includes the SMTP Mail Server, Sent-From Address and Recipients for the Email Alert. Click on Next allows you to confirm more advanced options for the Action including Day and Time of Day restrictions on when the Action can be executed. Click on 'Save' to add the new Email Alert Action. This will add a new Action using the 'Generic SMTP Email Alert (email_alert.pl)' Action Script with the configuration specified.

iPhone Alerts

Unless un-checked during the initial setup of Lithium Core, a Default iPhone Alert action is created to support the Apple Push Notification Service with Lithium Touch on the iPhone and iPod Touch. This action can be added later by clicking on the '+' button in the bottom-left of the Console Window and select "New Action for Customer...", the desired Lithium Customer and click on 'Other Action'. Alternatively, you can select the Customer (or a member Site or Device) and click on 'Customer -> Add New Action -> Other Action' from the Menu Bar. Either option will display the generic 'Add New Action' sheet.

Enter a Description for the Action and then select the 'iPhone Push Notification Script' and any desired Time/Day restrictions. There is no additional configuration required to set up iPhone Push Notifications from Lithium. Download Lithium Touch from the App Store (free) and connect to your Lithium Core deployment. Push Notifications will be automatically enabled once you have successfully connected to the Lithium Core deployment with your iPhone or iPod Touch.

The type of Push Notification displayed (Audible, Visual, etc) is controlled through the Settings application on your iPhone under 'Notifications'.

Lithium will only generate a text push notification in the event of an Impaired or Critical level Incident occurring or being cleared. Warning level events cause the Lithium Touch application badge number to update only without a full push notification.

Custom Alerts and Actions

Action Scripts

Lithium's built-in Action Scripts for Email Alerts and iPhone Push Notifications are written in Perl. You can modify, re-use and re-distribute them suit your needs. The scripts themselves are executed by Lithium Core with a large amount of data available about the actual Incident passed to the script as command line arguments. The configuration variables for the Action are also made available in a block of XML stored in a temporary folder. This path to this file is also passed to the script as an argument.

Examining the Email Alert (email_alert.pl) Action Script provides a good basis for understanding what information is passed to the script and how the XML interaction between Script and Lithium Core works. More specific questions and discussion about Scripting are welcome at http://support.lithium5.com

Lithium by LithiumCorp Pty Ltd