Alert correlation

In Incident Management, related alerts are grouped into an incident.

In this video, you will learn how to correlate your alerts into incidents in Incident Management.

Specifically, you will be able to explain the default correlation settings and how alerts are correlated into incidents.  Also, you will be able to configure new clustering settings in the correlation engine.

Correlation engine is where you manage all alert clustering configurations.

You start out with one out-of-the-box clustering setting, so your alerts will be grouped into incidents without any configuration on your end.

This is the default correlation setting. Scope defines which alerts are going to be evaluated by this correlation definition.  Consider it like an entry filter.  Since right now there’s only one correlation, it will evaluate ALL alerts.  It will evaluate the source field values, and the alerts whose source field values match more than 45% will be clustered into an incident.

The incident created by this correlation will have a dynamically composed description.  The default description shows how many sources are affected, and the top three sources, services, and event classes involved in the incident.

You can choose how many similar alerts are needed to create an incident. The default is one alert, so every alert that arrives will either form a new incident or be added to an existing incident.

The time window for correlation is automatically set.

You can set the correlation time window up to 24 hours.  Let me show you how it works.

When an incident is created, the correlation engine starts a timer.  Let’s say we keep the time window to 15 minutes.

If more qualifying alerts come in, they are added to the incident.

But we don’t want to keep adding alerts to the same incidents forever.  If you keep the incident open for new alert membership for an indefinite amount of time, you’ll end up mixing multiple separate issues.

So, here’s how Incident Management handles the time window.

The key concept here is 50%. 50% of the default time window of 15 minutes is 7 minutes 30 seconds.

From the moment the incident formed, qualifying alerts keep getting added to this incident throughout the 15 minute time window.

But the last half, from 7 minutes 30 second to 15 minutes, is the key to determine when the time window for this incident actually closes.

If no qualifying alert arrives during the second half, then the window closes at the default 15 minute mark.

But if a qualifying alert comes in during the second half, say, at the 14 minute mark,

it triggers Incident Management to extend the correlation window.  For how long?

Again the key is the 50%.  50% of 15 minutes is added from the point this alert arrived. So now the time window is set to close at 21 minutes and 30 seconds.

Suppose another alert arrives within the new correlation window, at 16 minutes. The window extends again from the arrival time.

By how much? Again, 50% of the default window!

Now you know how Incident Management correlates alerts into incidents.  Thanks for watching!

Incident Management’s correlation engine groups related alerts and forms incidents. This happens automatically with no configuration.  But in this video, you’ll learn how to create additional correlation settings to tailor it to your specific needs.

We will use a sample use case to step through the process of surfacing the requirements and configuring the product.

Suppose you are implementing Incident Management to facilitate collaboration between the infrastructure team and the application team.

The infrastructure team is organized by region, and each regional team breaks down into a few service categories.

So, if the server that supports service A in the US suffers, you want this team (AMERICAS - service A) and this team (Application 1 - service A) to come together and investigate.  

Also, since the infrastructure teams are organized by location, you need to differentiate the events originated in different regions.  

So this means just because service A is impacted, you don’t want to bundle alerts from the AMERICAS and EMEA clusters together.

You also noted that in the CMDB some of the values are not spelled consistently, like this for example.  But we want to treat these as the same location.

Based on the analysis so far, here’s how we want to correlate our alerts for this use case.

Now let me show you how to set this up.

Provide a name that makes sense to other administrators, too.

Scope is basically an entry filter.  

In our use case, this correlation only applies to the alerts whose class is application or infrastructure.

And we want to cluster alerts from the same service. Service defaults to 100% similarity because it is a list-based field.

And we want to cluster alerts from the same location, but considering the variation in spelling, we’ll set the similarity value at 70%.

What you put into the description field will be used as the incident description, so you want to make it as helpful as possible.

We are going to use macros to insert dynamic information here.

We’ll set the time window to 15 minutes. To understand the correlation time window, watch the “Alert Correlation Method in Incident Management” video.

Now, let’s also modify the correlation group settings, so the alerts evaluated by the new correlation setting won’t be evaluated by the default correlation, and end up in two separate incidents.

Let's change the order so our new definition is evaluated first.

Now let’s simulate what happens to the alerts with this new correlation definition.

Here’s the scenario.  The server in our San Francisco data center that supports a database query service has overheated and started to fail.  Of course, no one knows that yet.

The first event comes in…

The event goes through enrichment to add value to the service field and gets deduplicated into an alert.

Right now, this is the first event of its kind so it simply becomes an alert, but of course if another event with the same dedupe key value arrives, it will be deduplicated into this alert.

Now it arrives at the correlation engine.

The class field value meets the scope, so it enters the correlation we just configured.

Since this is the very first one, it becomes an incident on its own.

The next event arrives.

It is also impacting the same service as the last event but note that its class value is not Infrastructure or Application.

So, it is filtered by the correlation we set up.  Instead, it goes into the default correlation and becomes a new and separate incident.

Here comes another event. It’s enriched, deduplicated, and...

Now, this meets the correlation criteria we’ve defined, so this alert is bundled together with the first alert and becomes part of the incident 001.

Since we set the data center match percentage to 70 rather than 100%, it accommodates the difference in capitalization.

Now, this incident contains an infrastructure alert and an application alert on the same service in one place, and the applicable infrastructure team and the application team have been notified.

Looking at the timeline together, they quickly identified the issue originated in the hardware.

Without this incident correlating the two alerts together, the application team would not have been able to rule out other potential causes.

Now you know how Incident Management correlates alerts. thanks for watching

We’ll troubleshoot a sample incident to see the power of alert correlation.

Here’s a simple setup that we’ll be using for our demonstration. We have a few different sources that are sending data to Incident Management.

For starters, we’re ingesting metrics using the Metrics API. Incident Management is data agnostic, meaning that these metrics can be of any type, and can originate from any source.

We’re also using Splunk for logs and analytics, AppDynamics for application performance monitoring, and Prometheus for database and infrastructure performance monitoring.

And here’s what we are monitoring.

We have a three-tier, customer-facing application called Billing that relies on databases in Atlanta and New York.

One of the switches in the Atlanta data center experiences some degradation - maybe some packet loss - which impedes the communication between the application server and database.

Since all three of these domains are monitored by different teams using different tools, we’ll see application alerts...

Database alerts...

And network alerts.

As an operator trying to resolve this problem, they often don’t have visibility into what other teams are seeing in their monitoring tools. So it takes time to synthesize your analysis of what you can see, with the insights from others.

That’s where Incident Management comes in. Incident Management identifies related alerts and groups them together, organizes them into relevant incidents, and presents these incidents with rich context, making it easy for us to identify the underlying problem.

Let me show you what I mean.

Here is the Incidents panel, where we’ll start our triage efforts.

And this is the incident you’d get for the scenario we just went over.

Let's go into the Situation room to discover more. The service impacted by this incident is Billing...

And the description has been dynamically populated to tell us the critical information - in this case, the locations affected by the incident.

And this incident contains 11 alerts. No doubt, these came from all four data sources.

Sure enough, we got some application related alerts from AppDynamics, database alerts via Prometheus and Splunk, infrastructure alerts from Prometheus, and network alerts from the Metrics API. Such context helps us decide which teams in our organization we might want to engage.

We are going to own this incident, and assign it to our group.

Now let’s examine the alerts.

We’ll sort these alerts by First Event Time so we can see how the issue started, and how it evolved.

You can see in the Manager column how this incident combines alerts from four different data sources.

And under event count, you can see how Incident Management provides noise reduction by deduplicating events into alerts.

Now let’s look at the individual alerts.

The very first alert came from a switch in the Atlanta data center. It looks like the packet drops went out of bounds.

Only a few seconds after this alert was created, we began receiving additional alerts about database timeouts...

Application page load failures...

And various other issues.

So from this, it seems that the packet drop alert might be the root cause behind this incident.

Let’s do some further investigation by looking at the metrics for this incident. We’ll only examine relevant data from the past hour.

For each of these metrics, the pink line represents the raw values of the metric at any given interval. The light gray band in the background represents the normal operating range of the metric, as calculated by Incident Management.

And these colored dots represent anomalies in the metric that Incident Management has detected and classified, in terms of significance.

As the metric continued to stray out of bounds, the anomaly was classified as Critical...

And was later cleared when the metric went back in bounds.

This metric is the packet drops alert we talked about earlier. Let’s see if it’s really the underlying cause of this incident.

If we compare this graph to the graphs of the other alerts, we can see that all the anomalies started as soon as packet drops were detected...

And more importantly, all the anomalies were cleared as soon as packet drops were cleared.

This is a pretty good indication that our hypothesis is correct. The packet drop issue seems to be causing all the other alerts.

With this figured out, we know exactly who we should talk to about next steps. We’ll notify the team responsible for maintaining our network, and ask them to check on the faulty switch in the Atlanta data center.

Imagine how much time we’ve saved just now by having all the info from separate source systems in one place, and having them grouped together based on their relatedness for you. Now you know how you can have Incident Management correlate alerts for you for a faster mean time to recover.

Thanks for watching!