Loading....

Using Event Correlation to Automatically Raise Downed Server Alert Severity

RightITnow Event Management

Using Event Correlation to Automatically Raise Downed Server Alert Severity

Downed Web Server, but Operators Do Not Notice without Event Correlation

Your network operations center is super busy and your operators are too busy deploying hundreds of virtual systems for a new site to notice that a downed web server alert now has more than 200 associated events. That server has been down for quite a while and it would be nice to get it back up and running before your web-space clients start flooding your Zendesk with batches of associated tickets! Wouldn’t it be nice if you could write a rule that would automatically change the severity of that downed server alert to CRITICAL so your operators would take notice? You can with RightITnow ECM correlation rules!

Define the Downed Web Server Correlation Rule Conditions

Using the RightITnow ECM Correlation Rule Builder, you can instruct the RightITnow ECM correlation engine to match all events with a count greater than 10 whose description contains the words, “Web Server Down.” Once you have collected these events, you can act on the resultant alert.

Set the Downed Web Server Alert Correlation Action

So, what do you want to do with all of these downed web server events? You want to make sure that the resultant alert is assigned a very high severity so your operators will jump in fast to resolve the issue. You can use the Correlation Rule Builder to trigger the Set to Critical action that would set the severity of any qualifying alert to critical, automatically. Now, that alert lights up in bold red on your alert console, crying out for some operator attention and intervention.

As Long as We Have Focus on this Event, Let’s Do More!

You have built a correlation rule that automatically and reliably collects downed server events into an associated downed server alert and sets the priority to CRITIAL. As long as you have focus on this alert, you can do more with the correlation rule than just raise the severity to CRITICAL. You can use the RightITnow ECM Correlation Rule Builder to check if related following alerts exist, for example, is my mail server also down? You can also instruct the rule to perform an action if the condition is NOT true. For example, if you are getting downed server events, but not more than 10, you can trigger the Set to Medium Priority rule to set the alert severity to medium instead of CRITICAL.

Custom Alert Fields

RightITnow Alert Management

Managing IT Ops Alerts with Custom Alert Fields

Creating Custom Alert Fields

You can create custom alert fields that appear on the Alerts Console as column headings, allowing you to add useful information categories to your alerts. You can create 20 custom fields, and 10 of those custom fields may be indexed. Queries on indexed custom fields run faster. Indexed custom fields have a maximum length of 254 characters. You may change an existing custom field from indexed to non-indexed and vice versa without losing the data it currently has.

Importing Incident Fields as Custom Alert Fields

You can map incident fields into RightITnow as custom alert fields, allowing you to present information from your third-party incident system on the single pane of glass Alerts Console. The only fields that can be imported from incident systems are the ones that can only take values from a list. Free text fields cannot be imported. The created custom fields share the same name as the field in the incident system and are of type List. If the field is dependent from another field in the incident system, both fields are imported and the relationship is preserved. Should the values from the custom list fields you have already imported change in the external system, you can refresh the values within RightITnow ECM by clicking Refresh imported fields.

Custom Alert Field Update Behavior

When updating an alert’s custom alert list field that has dependent custom fields, ECM clears dependent custom field values. If an action that updates a custom field with value “A” is fired against an alert that has already value “A” on that custom field, then the alert and database will not be updated and no workflow action executed. When deleting a custom field, ECM will prompt you whether it should clear the existing values of that custom field in the database.

Executing Actions Upon Custom Field Update

You can configure the workflow engine to execute an action or an action group after a custom field has been updated. This can be very helpful for automating tasks depending upon how custom field values change. For example, if the value “cost center” custom field updates from “United States” to “UK,” then you could trigger an email to your Accounting department apprising them of the change in cost center.

Rolling Up Alerts

RightITnow Alert Management

Managing and Organizing High Alert Volume Using Rollup

Rolling up Alerts to Decrease Alerts Console Clutter

Sometimes, the best thing for an IT Ops operator is to simplify the alerts information flowing across the console. By selecting a lead alert and then selecting the alerts to roll up into that lead alert, you can significantly reduce the amount of repetitive alerts on the Console. For example, you may have an operator in charge of addressing all the disk full alerts. Instead of forcing the operator to search for each disk full alert, you can roll all of them up into a lead disk full alert so the operator can find all of them at once.

Rolling up Alerts to Connect Events

You may have an IT Ops issue that is creating lots of dissimilar alerts, and you may want to roll them up into one lead alert for an operator so that operator can see the impact of the IT Ops issue just by accessing the lead alert and drilling down into the alerts rolled up under that lead alert. For example, if you have a disk down and that disk stores print spool and email data, you will see disk down, print spooler, and email alerts flowing across your alerts console. You can roll up those print spooler and email alerts into the disk down alert and point your operator at it so your operator has all the information needed to address the leading and all downstream issues lighting up your console.

Configuring Alert Rollup for Your Environment

Rolling up alerts comes with lots of related features that let you customize the functionality to the needs of your specific environment. You can configure the feature to close rolled up alerts when lead alert is closed, include rolled up alerts in reports and exports, and automatically group alerts on roll up.

Finding those Nested and Lead Alerts

Only the lead alert appears in the Alerts console, but the rolled up alerts do not. You can expand the lead alert to see the alerts that you rolled up into it. When filtering the console, the lead alerts are returned but any rolled up alerts are not. However, you can include a condition in the Filter Builder that searches lead or rolled up alerts by using the Has Nested Alerts and Lead Alert ID conditions. This way, you get the power of rolling up alerts with the ability to find the individual lead and rolled-up alerts.

Achieving Two-Way Integration Between an IT Ops Event Correlation Manager and Incident Systems

RightITnow Integration

Achieving Two-Way Integration Between an IT Ops Event Correlation Manager and Incident Systems

Two-Way Incident System Integration: It All Starts with Connectors

RightITnow ECM provides incident system connectors that allow direct integration and interoperability with ATLASSIAN JIRA, ManageEngine ServiceDesk Plus, BMC Service Desk (ARS 7.5), and Salesforce Service Cloud. ECM also provides a connector to the IT Service Management component of Serena’s Serena Business Manager (SBM). The connector creates an incident based on a RightITnow ECM alert and reflects the changes made on the incident to the alert. Additional incident management connectors include ServiceNow® and Zendesk.

Automatically Creating Incident Tickets Pre-Populated with Alert Info

ECM’s Create Incident action and associated Insert Incident menu item automatically create a third-party incident ticket in an Incident Management system such as ServiceNow®, and this incident is pre-populated with the alert info from which you inserted the incident. For example, imagine that you are reviewing an alert in ECM’s Alert Console and you spot one for which you would like to create a ServiceNow® incident ticket. All you need to do is right-click the alert and select the Insert Incident command to create a corresponding incident within ServiceNow® that contains all relevant alert information from ECM. Similarly, you can use the Update Incident command to update the incident ID attached to an alert with an existing incident ID, and even automatically execute an action after the update. You can choose to append or overwrite the info in the incident system.

Updating ECM Alert Custom Fields with Incident System Info

You can create custom alert fields in ECM that correspond to fields in external incident systems. For example, you could create a custom alert field, incidentStatus that corresponds to the ServiceNow® incident status field, and then using the Update Alert Custom Field action, you can update the status of the ServiceNow® incident as displayed in the ECM Alert’s custom field, yielding two-way integration between ECM and the third-party incident system.

Acting On Underlying Alerts Using Incident Creation Workflow Rules

ECM provides sophisticated workflow rules you can use to configure ECM to take action on alerts when creating an incident in a third-party incident management system from within ECM. Incident creation workflow rules include automatically assigning alerts to the logged in user when creating an incident based on the alert, and running an action or actions after creating an incident or updating an alert with an incident. You can also configure the workflow to take specific actions whenever incident value status changes.

Integrating an IT Ops Event Correlation Manager with Third-Party Apps and Tools

RightITnow Integration

Integrating an IT Ops Event Correlation Manager with Third-Party Apps and Tools

Event, Alert, Entity and Maintenance Management with the ECM REST API

You can use the ECM REST API to harness all of the power of ECM within your own IT Ops management app. You can use the ECM REST API to manage and manipulate events, alerts, entities, and entity maintenance. Event and Alert functionality includes getting alerts, filters and breached SLAs; getting a list of events that have occurred between a specified time interval; changing alert priority, alert severity and alert ownership; unassigning, acknowledging, annotating, invoke an action on, and closing alerts; and creating incidents for specified alerts. Entity maintenance functionality includes getting, setting, and removing maintenance windows. Entity management features include creating, reading, updating and deleting entities in a multitude of ways and focuses. See the IT Operations REST API post at http://www.rightitnow.com/operations-management/it-operations-rest-api/ for complete details.

Publishing ECM RSS Feeds to Third-Party Applications

The paged alert filters that you create and save from the Alerts Console are available as RSS feeds. Access the feed by clicking the RSS icon in the alerts console filter pane. RightITnow ECM returns feeds in RSS 2.0 format. You can control the look of the RSS Feeds by configuring the RSS feed template. The feed’s channel elements are derived from the filter’s specifications and each item in the feed represents an alert.

Publishing Alerts to Third-Party Applications

The ECM Alert Publisher connector allows you to export alerts to an external JMS queue or topic, so that these alerts can be processed by your external system. You need to setup the queue or topic, and ECM will publish alerts to it. You can configure publishing to occur automatically whenever an alert is created or any of its fields updated. You can also define a filter so that only alerts that match the filter are published. Apart from auto-publishing of alerts, an ECM action can be used to manually trigger publishing of alerts from the alerts console or through correlation rules and the alert workflow. Alerts are published as a JSON formatted string and include all the alert fields that are available in the ECM user interface, as well as some additional fields used internally by ECM. Only one Alert Publisher connector can run at a time.

IT Ops Data Crunching via CSV

Aside from ECM’s more sophisticated integration points, including built-in and custom connectors, RSS feeds, and the ECM REST API; ECM can export audit log, entity, and alerts data to good old reliable CSV (Comma Separated Values) files that you can import into any of your analytics, reporting, spreadsheet, or other applications that import CSV files. This affords you direct access to the raw, pure data within ECM for analysis and manipulation in your favorite and most familiar applications.

IT Ops Integration and Interoperability via Event Connectors

RightITnow Integration

Solarwinds API from RightITnow

Achieving IT Ops integration and Interoperability with Event Connectors

Native Event Connectors that Bring It all To a Single Pane of Glass

You can use ECM’s built-in, easily configurable event connectors to collect and act on events from many external sources. For example, ECM can collect events from Amazon Web Services (AWS) CloudWatch and can poll AWS every minute for changes to alarm states and raise an event in RightITnow ECM when it encounters one. You can run a separate CloudWatch connector for every AWS region you want to monitor. The region is configurable in the settings pane by selecting the appropriate endpoint. You can also collect and process events from InfoVista, JDBC, ManageEngine Applications Manager, Microsoft SCOM 2007 R2 and 2012, Nagios, and SolarWinds Orion NPM (Network Performance Monitor) and SAM (SolarWinds Server & Application Monitor). The SolarWinds connector discovers component names when polling, but ECM enriches a component only when it encounters an alert related to that component. Additional event connectors include VMware, and Zenoss.

Collect Right from the Source with SNMP Traps and Syslog Connectors

Sometimes, you want to collect events directly from network devices, such as servers, printers, hubs, switches, and routers, rather than collecting them third-hand from an intermediary application. ECM’s SNMP Trap connector allows you to do just that, collecting events directly from all your entities in their purest form for processing by ECM. And what about logs? Logs are an IT operator’s vital roadmap to resolving IT Ops issues, and ECM’s Syslog connector ensures that the logs you collect are most useful by collecting logs that enable separation of the generating software, the storing system, and the analyzing software.

Roll Your Own Event Connector with Our Event SDK

You can use ECM built-in event connectors out of the box, or you can create your own custom event connector using ECM’s Event SDK. The ECM Event SDK is comprised of a number of individual APIs for Java and Python, and a more generalized SOAP API. In RightITnow ECM, an event is the most basic object of meaning and can be used to create alerts or trigger actions in the system. Events allow extensive control over RightITnow ECM. You can use the ECM SOAP interface to send events. This interface provides a fairly quick and easy method of forwarding events from other systems. When expected volumes of events are high or enterprise features are needed, such as failover support, consider using the Java API with the ActiveMQ connector.

De-duplicating and Processing All of These Events

So, you have collected all of these events from disparate systems into ECM. Now what? Let ECM do its magic by de-duplicating the events into existing alerts and then correlating the events with actions that can resolve IT Ops issues before you ever see them on your Alerts Console.

Advanced Entity Management

RightITnow System Configurability

Entity Management from RightITnow

Streamlining Your Entity Topology with Advanced IT Operations Entity Management

Merging Entities to Reduce Redundancy in Your Entity Grid

You can merge multiple entities into one selected entity to reduce redundancy in your entity grid. When ECM merges entities, it deletes all merged entities except the designate entity, adds the names and aliases of merged entities as aliases of the designate entity, adds the IP and MAC addresses of the merged entities to those of the designate entity, assigns the designate entity to the selected owner (if any), assigns the designate entity to the selected groups (if any), adds the maintenance windows of the merged entities to those of the designate entity, associates the alerts of the merged entities are now associated with the designate entity, and assigns child entities of the merged entities as children of the designate entity.

Configuring Entity Classes and Types

You can configure entity classes and types to be displayed and assigned to entities in the Entity console. For example, you may wish to have a class named, “virtual,” and a type named, “VMware,” that you can assign to entities to help you manage and classify them. ECM allows you to classify your entities however makes the most sense to your organization and builds these customizations into the product in various places for use by your operators.

Editing the Entity and Entity Group Hierarchy

You can select entities in the entity grid and set them as the children of another entity, and you can also add entity groups to other entity groups, and remove entity groups from entity groups. When editing the entity hierarchy, you must select non-polled entities to add to a parent entity, for example, entities not imported from Zenoss, SolarWinds, or VMware. You can also select several sibling child entities (non-polled) and remove them from their parent entity (undo the hierarchy) by choosing Remove from parent entity from the context menu.

Insight into Your Entities

The entity network map is a very useful tool for visualizing your entity topology, and it also displays the entity hierarchy within an entity group. This is all presented in a familiar and effective hierarchical tree view. Each entity has built-in fields associated with it, and you can create custom entity fields and these fields, built-in and custom, are available to you in the RSS feed template, so when you access an RSS feed created by you in ECM, that feed will report on your built-in and custom entity fields.

IT Operations Rest API

RightITnow Integration

Customizing and Performing IT Operations Management with a Rich and Powerful REST API

Retrieving Events and Alerts with the ECM REST API

The REST API is designed to integrate ECM with your IT applications .You can use the ECM REST API to get alerts by a filter and by alert ID. You can also get all filters and all breached SLAs for a user, allowing you to program into your application all of the power of the ECM Alerts Console. You can even Retrieve the list of events that have occurred between a specified time interval.

Acting on Alerts with the ECM REST API

You can use the ECM REST API to change alert priority, alert severity and alert ownership. You can also unassign, acknowledge, annotate, invoke an action on, and close alerts. A related ECM REST API call can create incidents for specified alerts.

Manage Maintenance Windows Using the ECM REST API

You can use the ECM REST API to perform the following tasks related to maintenance windows: fetch all maintenance windows for entities, fetch all maintenance windows for entity groups, fetch maintenance window set for an entity, fetch maintenance window set for an entity group, set maintenance window on an entity, remove maintenance window set on entity, set maintenance window on an entity group, and remove maintenance window from group.

Manage Entities Using the ECM REST API

The ECM REST API offers full entity management. You can use the API to Fetch all Entity Groups, Fetch Entities for a Group, Fetch Entities owned by user, Fetch Entities by filter, Get Entity Hierarchy (by ID), Get Entity Hierarchy by Name, Get Entity Hierarchy by Group, Add Entities By Name to Group, Add Entities By ID to Group, Remove Entities By ID from Group, Remove Entities By Name from Group, Get Entity Group Hierarchy (by ID), Get Entity Group Hierarchy (by Name), Add Entity Group to Parent Group (By ID), Add, Entity Group to Parent Group (By Name), Remove Entity Group from Parent Group (By ID), Remove Entity Group from Parent Group (By Name), Rename Entity, Create Entity Group Type, Update Entity Custom Fields, and Retrieve Entities for given Custom Fields.

Entity Owners and Maintenance Windows

RightITnow System Configurability

Streamlining IT Operations Management with Entity Owners and Maintenance Windows

Enrich Entities with Owners to Automatically Assign Associated Alerts

You can assign owners to entities. RightITnow ECM will assign any new alerts originating from this entity to the entity owner, eliminating the need to assign these alerts manually. For example, if you have an email guru in your organization, then you can assign this guru ownership of all of your email related entities so that anytime there is an email issue in your IT Ops environment, your guru would automatically be assigned to address those issues. This is super efficient and a huge timesaver.

Automatically Create Maintenance Windows

You can use the Create Maintenance Window action in correlation rules to automatically create a maintenance window of the specified length in hours for an alert’s entity or for the entity group of the alert’s entity. This way, you can automatically put an entity into maintenance when ECM encounters alert conditions you specify. For example, if an ECM encounters an alert with a disk full message, then you can configure ECM to automatically place all such associated entities into maintenance, and if you also assigned an owner to those entities as described in the previous section, then the owner would automatically be assigned these alerts.

Deep Maintenance Window Features to Enhance IT Ops Management

RightITnow ECM offers deep maintenance window features. You can use an action to create maintenance windows automatically as described in the previous section, or you can create them manually. You can push ECM maintenance windows to Zenoss Device Groups. When you create a maintenance window for an entity group imported from Zenoss, ECM also creates the maintenance window within Zenoss. You can schedule deployment and undeployment of Close Maintenance rules. Entity groups can inherit maintenance windows for more efficient bulk processing, saving you from creating the same maintenance window recursively down the hierarchy. An entity can have multiple maintenance windows and you can also deduplicate events during maintenance windows. By default, ECM deduplicates events received from an entity that is in maintenance only against alerts that are in maintenance. If the event would deduplicate against an alert that is not in maintenance and there are no alerts in maintenance, then a new alert, with different deduplication criteria, is created. However, you can configure ECM to allow events to deduplicate against alerts that are not in maintenance.

Surfacing Entity Ownership and Maintenance Windows to a Single Pane of Glass

You can add the Entity Owner entity field to the Alerts Console grid to reveal more information about the entity which triggered the alert, and the Overall Maintenance dashboard displet indicates which devices are under maintenance. This visibility, coupled with the ability to assign ownership and create maintenance windows for entities helps keep you on top of your entity topology and underlying issues. See http://www.rightitnow.com/operations-management/configurable-it-operations-alert-menu for more about the Alerts Console and http://www.rightitnow.com/operations-management/it-operations-management-dashboards for dashboard information.

Tagging Alerts

RightITnow Alert Management

Giving Your Alerts Useful Nicknames at Birth

When RightITnow ECM encounters a new IT Ops event, it applies a series of rules to it that helps you deal with it in the most efficient way possible. These rules may deduplicate the event or create an new alert based on the event. One of the rule types is a tag rule that you can use to tag the event and corresponding alert with nicknames (keywords) that you can use to analyze alerts data without changing your data model or topology.

What IT Ops Tag Rules Do

A tag rule updates the Tags column in the Alerts table to the value specified in the rule. When building a tag rule, you name and describe the rule, choose the connector and build conditions like you do for any other type of rule, but you can also select multiple tags to add or remove should the conditions of the rule be met. For example, For example, you could write a rule that if the event message contains the word, “postfix” or “sendmail,” then set the Tags column value to “email:”

Tag Rules from RightITnow
This way, operators can search the Tag column for email issues more easily.

Special Tagging Features to Enhance Business Service Management

You can add a new action, Evaluate Tag Rules, to the Alert Console Context menu so that you can re-evaluate tag rules on alerts directly from the Alert Console Context menu. This allows you to re-tag the alert should another process have touch and changed the alert. You can also add or remove multiple tags at once. When executing the tag rules on incoming alerts, if the rule removes tags, it will only remove tags that were added via the event token “tag” or by another tag rule. This is executed before the alert is stored in the database, so tags that are already in the database (previously added) will not be removed.

Enhanced IT Ops Business Service Management Sans Database and Topology Changes

Tags allow you to analyze and manipulate your IT Ops alert data without changing your data model or topology, which are much more disruptive and costly changes. For example, given the correct corresponding tag rules, you could use the Tags column to quickly find all “Disk Full” or “Server Down” conditions in “Boston” without touching your database or re-arranging your entity topology. This enables you Operators to concentrate on resolving the issues rather than mining the data for the issues.

Back To TopBack To Top