Data ingestion

After watching this video, you will be able to configure a custom integration to bring data from your monitoring source into Incident Management. You will be able to identify when to choose this over other integration options, ingest data in JSON format into Incident Management, map data from your monitoring software to Incident Management event fields, and set up and test a deduplication key to reduce operational noise.

Incident Management offers a few different ways to ingest source data. You can install a data collector to the system you want to monitor...

...or integrate with specific monitoring tools.

Or you can build your own custom integration. You should choose a custom integration when there is no collector for your environment, and there is no specific integration available for your monitoring tool.

Creating your own integration has several benefits. First, you are not limited to particular monitoring tools. You can set up a custom integration with any monitoring source that can send data in JSON format.

Second, you will use real data to design and test your integration. The Create Your Own Integration feature makes it easy to inspect source data before you map it.

And finally, creating a custom integration is simple. When you configure a webhook in a monitoring tool, you usually have to write code to modify the outgoing data payload. The Create Your Own Integration feature does all the mapping right in the Incident Management user interface, with no code.

After you have ingested data using a custom integration, you can transform and enrich it using the Incident Management Workflow Engine.  Still without writing any code.

Let’s step through setting up a custom integration together. Let’s say we have source data in JSON format that’s structured like this:

We want to ingest the events so we can deduplicate and correlate them in Incident Management.

And we need to map them to the Incident Management event fields.

We also want to map the values for severity. Our source events only have four severity levels: NONE, INFO, WARN, and PROBLEM. We want to map these values to Incident Management severity levels like this.

This is where you can set up a custom data ingestion. We are going to set up a new integration. Our source data in this example events, but note that we support metrics also.

We just created a custom endpoint for our integration.

Grab that, and the API Key for our Incident Management instance which is found here.

We’ve input those values in the source system. And now, Incident Management is receiving raw data from our source. Note that up to 10 of the most recent events are cached. You can click the reload button to refresh the list of data payloads.

We can examine the data for each event here.

Note that some of these events have blank hostnames, and for some the hostnames are populated.

Let’s see how the payload and target fields line up. Incident Management automatically matched the obvious ones.

We don’t have a value for source, which is a required field.  So we need to map something to that field.

Let’s look at the data again. We want to map hostname to source, but we know it’s blank in some cases, and required fields can’t be blank.

Let’s do this. We’ll map hostname to source if it exists, and if not we’ll use ip_address.

We’ll map hostname to source first. Then we’ll add a second mapping, and map ip_address. Incident Management will use the first mapping if a value exists and is valid. Otherwise, it will move on and use the next mapping.

This defines the condition that generates the event.

We’ll map the Type field to check.

Next let’s take care of the severity. Our monitoring service has only four severity levels: NONE, INFO, WARN, and PROBLEM. So we’ll map them to the Incident Management severity levels like this...

With that we took care of all the required fields. Let’s see what other fields we might want to bring in. Let’s say we want to keep region. And let’s add a custom tag for ip_address.

The last step before we save our integration is to configure the deduplication key. Here’s how deduplication works. Incident Management uses the fields in the deduplication key to assign multiple events to the same alert and reduce noise. The idea behind deduplication is that events with the same context should become part of the same alert.

For example, we might find out about a condition that affects one of our hosts with a warning event, which then escalates to a critical event. Since the key context is the same, Incident Management would assign those events to the same alert.

Here comes another event, and since the dedupe key is different, it is categorized into a separate alert. Like this, selecting the right fields for your use case is the key factor for successful deduplication. The default is set to be the combination of the Source, Service, and Check fields, but you can select what works for you.  Just ask yourself, what should be the common factors for two events to be considered duplicates.

Going  back to our custom integration, these are the default fields for deduplication. You can change them if you wish, but Incident Management recommends you use the defaults unless your business needs require that you use different fields.

In our case we don’t have a value for class, so we’ll remove that from the default keys.

Let’s test. Success! Here are the resulting deduplication keys.

Based on these values, some of the events have been assigned to the same alert.

Our custom integration is ready to go. Let’s save and activate it.  Note that it will stay in provisioned status until Incident Management processes an event.

The monitoring data is flowing into Incident Management through our custom integration. Thanks for watching!

Hello everyone! My name is Robert Pomichter and I'm a backend engineer on the Jawa team at Dell Technologies. Today, I will explain and demonstrate how to install and run a collector on a Linux machine using a forward proxy.

This tutorial assumes that a forward proxy is already in place. As we can see, currently, there is a Squid web Proxy Server actively running on this machine.

On the machine which we wish to install the collector, we should first export an environment variable called https_proxy, like so. The value should be equal to the public IP and port of the proxy.

Verify the https_proxy environment variable exists using the echo command.

Once the initial setup is complete and you've provisioned an instance of Incident Management, navigate to Integrations, Ingestion Services, Collectors, and Installation Tab. Choose either MacOS, Linux, or Windows. For the purposes of this demonstration, we’ll be choosing the Linux operating system.

Then navigate back to the terminal on which we wish to install the collector. Paste the installation command in the terminal. As it currently exists, we need to make a few edits to this command.

First, we'll supply an additional query parameter called proxy_url. This value should be equal to the https_proxy.

It looks like the collector has finished installing.

We can check the logs. Use the tail command to specify the number of lines from the logs we wish to print.

The health checks passed.

Here we have our Linux collector. Of course, this is only one option we can use to install a collector behind a forward proxy.

Let's clean up this collector and try another approach to install a collector behind a forward proxy. First, we'll delete this collector. Next, we'll do some cleanup on the terminal.

Remove the collector directory. The collector directory has been removed and the Collector has been stopped.

Verify the collector has been stopped using the following command.

The only process of the collector running is the grep. This means the collector has stopped.

While it's acceptable to export an environment variable called https_proxy to utilize the proxy, this isn't always necessary. First, let's unset the environment variable…

…and verify that the unset worked. There's no value for https_proxy, so currently the environment variable is not set.

Click Linux (and copy).

Once again, we need to make a few edits to the script. The first is, of course, adding the additional query parameter. Since we no longer have the environment variable handy, we'll need to specify it.

Additionally, we'll need to add something to this curl request. To route the curl request successfully through the forward proxy, add the following.

And now hit enter. It looks like the installation is complete.

Once again, we can check the collector logs. It looks like the health checks passed.

Once again, we have a Linux machine.

We can verify that the traffic properly passed through the forward proxy by checking the forward proxy access logs. Verifying the access logs, we see traffic coming from IP address 3.144.220.141, the machine on which we installed the collector. Traffic was passed from this IP address using the connect method to the Incident Management instance.

We can verify the time on which this occurred using the following command. This is a UTC time. This is roughly five hours ahead of my time zone, which is 12:56.

This demonstrates that all traffic was properly routed through the forward proxy. Now, we will explain and demonstrate how to install a collector on a Linux machine using a reverse proxy. This tutorial assumes that a reverse proxy is already in place. As we can see, this machine is actively running an nginx reverse proxy.

Copy the installation script and navigate back to the terminal.

We'll need to make a few edits. First, we'll supply an additional query parameter called base_url. This should be equal to the value specified in the nginx configuration file.

Next, we'll need to change the base_url currently being exported. This value once again should be equal to the same value we specified as the query parameter.

Since we have a trailing forward slash, we'll need to remove the trailing forward slash from the curl request.

Now we can hit enter. The installation is complete. Thanks for watching!

In this demo, we are going to install the APEX AIOps Incident Management collector to an EC2 instance that's running Docker. 

Here's the installation script.

It sets the API key...

And the controller variable...

...then downloads the installer script and plugs in those variables.

There's one more variable to be set, but we have to run this script first.

The install script is downloading now.

Here’s a prompt to set the file location variable.  Note that the file location will be different based on whether you are installing the collector as a root user or not.

Saving this is not a requirement.  But if you want to start, stop, and reload the collector later, either put this variable in one of your Linux startup config files, or export it like this.

Here’s the collector we just installed.

By default only the System plugin is enabled.

Let’s enable the Docker plugin. You can apply a filter to exclude certain container ID prefixes if you want.

And now we enabled the Docker plugin. Let’s check if the metrics are flowing in. A wide variety of metrics are available. These came from the docker plugin we just enabled,,and they include container, memory cpu, network, and block IO stats.

These came from the system plugin that was enabled by default.

Each of these metrics come with a variety of useful tags that you can use for troubleshooting or correlating alerts when something happens.

Now you know how to install a collector. Thanks for watching!

After watching this video, you will be able to configure AppDynamics to send events to the Incident Management API. Specifically, you will know how to:

Here’s our scenario. We want to collect application performance data from AppDynamics so we can deduplicate and correlate it in Incident Management.

You need two pieces of information to set up this integration: the APEX AIOps Incident Management URL and the API key.

Let’s go to your AppDynamics instance. Let’s say from the application you instrumented, we want to collect health and performance anomalies.

We are going to start by defining a new HTTP request template. With this template, we can set up an outbound webhook from AppDynamics to Incident Management. Name the template...

...and enter the POST method, and the URL from your Incident Management AppDynamics integration.

We need some custom request headers to accompany our data payload. The content type is application/json. And here is where we enter the apiKey from Incident Management.

Setting up the data payload is a pretty involved task, so let me park this for now and take care of the rest of the template first.

Here’s where you configure the response settings. Plug in the failure codes here. And the success codes here. A 200 means Incident Management accepts the data payload.

We’ll leave the rest of the settings the same. So that’s the template. Now let’s drill down into the payload setup section we skipped. Here’s a sample script from the Incident Management documentation.  You can use this to map the AppDynamics data fields to those in Incident Management.

You do not need to map every single source data field.  If the field value is not useful for deduplication, correlation, or troubleshooting, you can leave that field unmapped and drop the data.

Here’s a common use case. Service is an important field, but the source data from AppDynamics doesn’t contain that information. You are planning to get that information from a CMDB. What you can do in such a case, is to provide a default placeholder text. Then set up an event enrichment workflow to replace the placeholder value with the real information.

Let’s take a look at the sample script. The first part cleans up the AppDynamics messages.

The next parts extract and define the fields we want.

Here, we’re defining the Incident Management event object using the values extracted above.

Let’s copy this script into the payload section of the template. Set the type to application/json.

Our HTTP Request Template is ready. Here’s a tip. You need to save this first before you can test it. Now we can test it. What if the source event in AppDynamics said there’s a health rule violation...It says the event was successfully posted.

Let’s check on the Incident Management side. Here it is, a health rule violation alert is received!

Lastly, we are going to set up trigger logic to invoke the webhook when AppDynamics detects an  issue. To do that, we need to create an Incident ManagementAppDynamics Policy. We’ll name it .

Here’s the trigger logic we want.

When there are health rule violations, we want to trigger an action... which is to invoke our HTTP request template and send data to Incident Management.

Now you know how to set up a webhook integration with AppDynamics. In this demo we have configured only one application, but if you have a large AppDynamics implementation you can copy your configuration across multiple applications and controllers. Contact your AppDynamics support team for information about the AppDynamics Configuration Exporter utility. Thanks for watching!

In this video, we will go over how to integrate Amazon CloudWatch and send the data to Incident Management.

The instruction documentation is conveniently located right here.

We are going to add our AWS credentials.

We need to create a policy and a role in AWS to complete the configuration. To do so, we need this JSON script from Incident Management.

Grab this JSON script.

And now we are going to AWS.

Navigate to the policy section.

Paste the JSON script we grabbed from Incident Management...

...and provide a meaningful name.

Now we got the policy. Let's create the role now.

"Another AWS Account" is what we need here.

We need to provide the APEX AIOps Incident Management AWS account number here.

Let's go back to Incident Management. Copy this...

And paste it here.

Next, check "Require External ID". We need to grab it from Incident Management. Here it is.

To this role we are creating, we need to associate the policy that we just created. Let me search by the name I gave that policy.

Lastly, provide a name for the role.

And now we have the role created.

Let's go back to Incident Management and plug in the new role information. Here's the name of the role, and here's the AWS account number.

And the new credentials are now available for use.

Select a region, and test.

It works! Now, the data from Cloudwatch will start to flow into this instance of Incident Management.

Thanks for watching!

After watching this video, you will be able to create an integration to ingest New Relic incidents and set up a webhook in New Relic.

Here’s our scenario. We want to ingest alerts identified by New Relic so we can deduplicate and correlate them in Incident Management.

Here’s the New Relic integration.

The API is provisioned. We’ll make it active after we finish setting it up.

Next we are going to get data from New Relic. Incident Management will cache up to 10 New Relic events until the integration is active. These counters show how many payloads we’ve received - so far, in this case, none.

These are what we need.

Now going back to New Relic. We’ll start by configuring a notification channel to send issues to Incident Management.

The channel type is webhook.

...and this is where you paste the Incident Management Endpoint API URL we grabbed.

We need some custom request headers for our data payload. The content type is application/json.

And here is where we enter the apiKey.

We’ll use the default New Relic output, so we don’t need to set up a custom data payload and test these settings. Good!

Make sure to add this notification to the New Relic policy associated with the alerts you want to send to Incident Management.

The test notification showed up in Incident Management.

Here's the JSON payload from the test notification.

Field mapping is preconfigured to make it easy for you.

Optionally, at this point, you can also pick the deduplication key. Incident Management uses the dedupe key value to combine events with the same context into a single alert, while updating the alert fields that change over time like description and status. By default Incident Management uses the combination of source, class, service, and check fields. We’ll keep the defaults...

And test the deduplication key. Looks good! Now when another event with the same value for this field comes in, Incident Management will update alert fields like latest event arrival time and severity, as needed.

Our integration activates with data flowing in.

Now we can see our data from New Relic in the  Alert view. Thanks for watching!

After watching this video, you will be able to configure an integration to send events from Zabbix into Incident Management.

This demo uses Zabbix version 5.4, which may be different from your instance.  But it should orient you enough to make it easy. Let’s get started!

Name your integration, then grab the APEX AIOps Incident Management Endpoint and API key.

You also need a Zabbix media type file, which is available in our docs.

OK we have everything we need to make this happen.  Let’s go to our Zabbix instance. Import the media type file we just downloaded.

Then add the API key and the APEX AIOps Incident Management Endpoint.

This part of the media type definition maps Zabbix field names to Incident Management fields.

These fields are required.

Add optional fields based on your requirements.

This Javascript uses the mapped parameters to do further refinement, and compose the JSON payload to Incident Management. You can edit the script to add optional fields.

Let’s test to make sure it works. Looks good. The media type task is done.

Next, we need to create a user to receive event notifications using our Incident Management media type.

Then add the user to a  group to grant the right level of access. The Incident Management user won’t need access to the Zabbix front end, so we’ll choose this one.

The password is required but it won’t be used, so you can enter anything here.

This user needs to be able to use the Incident Management media type. This will send event notifications directly to Incident Management.

And this user needs a super admin role.

The user definition task is done.Now we have the pathway to Incident Management all set up. Next we want to add the trigger logic to use that pathway when events happen. We’ll do that by defining an action. You can update existing actions or define custom actions to trigger on specific problems, but here we’ll simply define an action to send all events to Incident Management.

If we wanted to restrict the severity of events or filter them in some other way, we could define a condition here.

Let’s say we want to send all events, so we won’t add a condition. Under Operations, specify what happens when an event, recovery, or update occurs.

We will send a message to the Incident Management user using the media type Incident Management.

We’ll repeat these settings for recovery and update events and add the action.

All set. The action definition task is done. Now we can see our Zabbix event data in Incident Management. Thanks for watching!

Use APIs to send sample data into APEX AIOps Incident Management

In this video, you will learn how to generate API keys, make requests to your instance of Incident Management using our APIs, and modify the sample command in the docs and test it.

Suppose you have a monitoring tool collecting metric and event data, and you want to route that data to Incident Management using our APIs.

We can send a test event from here. The Events endpoint is here...

...and the required attributes for the event payload are listed in this section.

Let’s go ahead and send in a test event. You can copy this…

…and paste it in a terminal. Feel free to edit the attributes to make it fit your purposes, and send...Success!

And the event is now in Incident Management.

There's another way to do this. You can build the integration right in our API documentation. Let me show you how.

First, we need an API key to push data to Incident Management. Best practice is to generate a key for each purpose here, so that you can grant and revoke access in a granular manner.

Let’s generate a key for us.

Make sure the name and description make sense to other administrators. Here’s our key.  For security reasons, you will not be able to reference it once you move away from this screen, so make sure you copy it.

Next, open up the Incident Management API documentation and navigate to the Event API section. This is the endpoint for the Events API.

And you get sample code in a variety of languages.

Plug in the API key you generated earlier. Now the sample code includes your key, so it will send requests to your own instance of Incident Management.

You can modify the sample code to make it fit your purpose. Let’s send this request and see what we get. Success!

Continue editing and testing here, and when you are done, you could send it from here, or copy and paste the resulting code into a CLI. We could also build shell scripts to add additional logic and conditionals that we simulated in the api docs site.

Metrics API works the same way. You can extend the sample and test it here.

Now you know how to take advantage of our API documentation. Thanks for watching!

In this video, you will learn how to create a custom integration to process multi-event data payloads, use Incident Management documentation to build sample data to test your integration, and understand the JSON structure of batch event payloads.

For this demo, let's use the Create Your Own Integration ingestion service, which allows you more flexibility over the fields to ingest, as well as the mapping.

Let me show you how it works. Give it a meaningful name, and add a description. Select Events.

Now we have a Incident Management endpoint, as well as the API Key.  We will need these in order to send over our batch events.

Now open the Incident Management documentation to pick up an example request. This sample curl command posts two events to Incident Management.  Grab and paste it in an editor.

To make it work with our instance, add your Incident Management endpoint and API key.

These are all the variables. 

We are passing in events...

...which are organized by this key.

There are a variety of elements in the sample code, and one thing we want to make sure of is that the default fields used for deduplication are available. By default, Incident Management uses the combination of these field values to determine duplicate events, so naturally we want to make sure these fields are included.

Now copy the curl command, and paste it in a terminal.

It successfully came in.

Make sure to enable batch processing, and select the path to the event list.  In this case, it’s events.key.

Next, map the payload fields to the Incident Management target fields.

Since all the fields used for deduplication are in the example payload, we don’t need to do anything for the deduplication key.

Let’s test.  Success!

Now you know how to process batch events using the Create Your Own Integration API. Thanks for watching!