Book Image

Splunk: Enterprise Operational Intelligence Delivered

By : Derek Mock, Betsy Page Sigman, Paul R. Johnson, Erickson Delgado, Josh Diakun, Ashish Kumar Tulsiram Yadav
Book Image

Splunk: Enterprise Operational Intelligence Delivered

By: Derek Mock, Betsy Page Sigman, Paul R. Johnson, Erickson Delgado, Josh Diakun, Ashish Kumar Tulsiram Yadav

Overview of this book

Splunk is an extremely powerful tool for searching, exploring, and visualizing data of all types. Splunk is becoming increasingly popular, as more and more businesses, both large and small, discover its ease and usefulness. Analysts, managers, students, and others can quickly learn how to use the data from their systems, networks, web traffic, and social media to make attractive and informative reports. This course will teach everything right from installing and configuring Splunk. The first module is for anyone who wants to manage data with Splunk. You’ll start with very basics of Splunk— installing Splunk— before then moving on to searching machine data with Splunk. You will gather data from different sources, isolate them by indexes, classify them into source types, and tag them with the essential fields. With more than 70 recipes on hand in the second module that demonstrate all of Splunk’s features, not only will you find quick solutions to common problems, but you’ll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization. Dive deep into Splunk to find the most efficient solution to your data problems in the third module. Create the robust Splunk solutions you need to make informed decisions in big data machine analytics. From visualizations to enterprise integration, this well-organized high level guide has everything you need for Splunk mastery. This learning path combines some of the best that Packt has to offer into one complete, curated package. It includes content from the following Packt products: • Splunk Essentials - Second Edition • Splunk Operational Intelligence Cookbook - Second Edition • Advanced Splunk
Table of Contents (6 chapters)

Chapter 8. HTTP Event Collector

In this chapter, we will learn about the HTTP Event Collector (HEC) that was released with Splunk v6. This addition has added a new way for websites and their processes to be monitored and improved.

In this chapter we will learn about the following topics:

  • What is the HEC?
  • How does the HEC work to improve website operations by sending HTTP and HTTPS events directly from an application to Splunk?
  • How data flows to the HEC
  • How to generate an HEC token
  • How to perform a POST request by placing the EC token in the request's authentication header
  • How to POST data (in JSON format) to the receiver for the HEC token
  • How to send data to the HEC using Web UI click events
  • How to log on to the HEC with JavaScript

What is the HEC?

Splunk (in versions 6.0 and up) has created a very useful data tool that is important for Splunk developers to be aware of. The HEC has the important function of collecting and sending HTTP and HTTPS events directly from a web application to Splunk, where it can then be used for analysis, creating alerts, and other effective functions of Splunk.

How does the HEC work?

HTTP and HTTPS events created by web applications contain event metadata, such as time, host, source, source type, and index, as well as other event data, found in curly brackets following the event key. The HEC makes it easy for app developers to add a minimal amount of code in order to send this data, so it's valuable for operational decision making, directly from their apps to Splunk. This is all done in a secure and efficient way, making it easy for apps to be able to Splunk their data.

Typically, an application generates its own log file or uses Document Object Model (DOM) tagging to generate some relevant functional metrics. This is useful and still applicable to traditional multi-page web applications. But web page development has leapt forward in recent years with a new framework called Single Page Application (SPA). The advance of SPA means that most of an application's work in showing HTML results now happens dynamically in the client's browser. Instead of going through different HTML pages, only one HTML page is loaded when the user interacts with the app.

This advance, though revolutionary, can pose a crucial dilemma for data monitoring. Since most of the application's interactions now occur on the client side, no server-side tracking can be done. This is where the HEC comes into its own, since you can create a very small line of JavaScript code that can essentially push data into Splunk. It is important to be aware that for applications in the cloud, this will only work if you are using Splunk Cloud or if your Splunk instance is accessible in the public domain.

There are use cases other than web applications that may also find the HEC useful. For internal applications, the HEC can easily be utilized to keep track of events occurring in the client's UI. This is also viable for the increasingly ubiquitous Internet of Things (known as IoT, a network of devices with a variety of purposes that is hooked up to a network). This is also a good way to monitor mobile applications.

How data flows to the HEC?

Let's begin by looking at how data flows to the HEC. This is a multi-step process that is important to understand before we go deeper.

Logging in data

First, data needs to be logged in, but before that it needs to be packaged from the source, which can be done in a number of different ways. These are listed as follows:

  • A Splunk logging library, such as Splunk logging for Java or Splunk logging for .NET
  • Another agent, such as a JavaScript request library
  • The Java Apache HTTP client
  • And lastly, some other client, as long as it will appropriately package the event data in JSON format

Before going further, let's review what the JSON format means. A couple of examples of key-value pairs in JSON format are shown here. The key is listed first, then a colon, then the value of that key. Sequences of key-value pairs must be separated by commas:

  "time": 1636289537, 
  "index": "main", 

Using a token with data

Second, the system needs to take each JSON data package and give it the same token in its authorization header. This will enable data to be brought into Splunk. The fact that the system is based on tokens means that the user doesn't have to include Splunk credentials in the application or in files that support the application. This token has been generated by Splunk in preparation for bringing in the data. It is created using the management endpoint or the Splunk Enterprise management URI and one of the following tools:

  • The HEC user interface
  • cURL, which is a way to use a BASH script to get information from a URL
  • The Splunk command-line interface (CLI)

Sending out the data request

Third, the JSON data packages, each of which has its correct token, are then sent out as an HTTP or HTTPS request to the input endpoint on the instance of Splunk Enterprise.

Verifying the token

Fourth, the token is checked and verified against the list of known tokens. If it passes this checkpoint, and is found to be a good token, then the data package is taken in by Splunk.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Logging in data

First, data needs to be logged in, but before that it needs to be packaged from the source, which can be done in a number of different ways. These are listed as follows:

  • A Splunk logging library, such as Splunk logging for Java or Splunk logging for .NET
  • Another agent, such as a JavaScript request library
  • The Java Apache HTTP client
  • And lastly, some other client, as long as it will appropriately package the event data in JSON format

Before going further, let's review what the JSON format means. A couple of examples of key-value pairs in JSON format are shown here. The key is listed first, then a colon, then the value of that key. Sequences of key-value pairs must be separated by commas:

  "time": 1636289537, 
  "index": "main", 

Using a token with data

Second, the system needs to take each JSON data package and give it the same token in its authorization header. This will enable data to be brought into Splunk. The fact that the system is based on tokens means that the user doesn't have to include Splunk credentials in the application or in files that support the application. This token has been generated by Splunk in preparation for bringing in the data. It is created using the management endpoint or the Splunk Enterprise management URI and one of the following tools:

  • The HEC user interface
  • cURL, which is a way to use a BASH script to get information from a URL
  • The Splunk command-line interface (CLI)

Sending out the data request

Third, the JSON data packages, each of which has its correct token, are then sent out as an HTTP or HTTPS request to the input endpoint on the instance of Splunk Enterprise.

Verifying the token

Fourth, the token is checked and verified against the list of known tokens. If it passes this checkpoint, and is found to be a good token, then the data package is taken in by Splunk.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Using a token with data

Second, the system needs to take each JSON data package and give it the same token in its authorization header. This will enable data to be brought into Splunk. The fact that the system is based on tokens means that the user doesn't have to include Splunk credentials in the application or in files that support the application. This token has been generated by Splunk in preparation for bringing in the data. It is created using the management endpoint or the Splunk Enterprise management URI and one of the following tools:

  • The HEC user interface
  • cURL, which is a way to use a BASH script to get information from a URL
  • The Splunk command-line interface (CLI)

Sending out the data request

Third, the JSON data packages, each of which has its correct token, are then sent out as an HTTP or HTTPS request to the input endpoint on the instance of Splunk Enterprise.

Verifying the token

Fourth, the token is checked and verified against the list of known tokens. If it passes this checkpoint, and is found to be a good token, then the data package is taken in by Splunk.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Sending out the data request

Third, the JSON data packages, each of which has its correct token, are then sent out as an HTTP or HTTPS request to the input endpoint on the instance of Splunk Enterprise.

Verifying the token

Fourth, the token is checked and verified against the list of known tokens. If it passes this checkpoint, and is found to be a good token, then the data package is taken in by Splunk.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Verifying the token

Fourth, the token is checked and verified against the list of known tokens. If it passes this checkpoint, and is found to be a good token, then the data package is taken in by Splunk.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Indexing the data

Fifth, the data is taken by Splunk from the JSON packet payload and is indexed. It can then be analyzed and examined.

We hope that by going over the preceding steps, you can begin to see how the HEC process we have just discussed can help an organization find operational efficiencies and uncover ways to improve functioning. Let's now get started with exactly how to do this, step by step.

We will show you specifically how Splunk can be used to improve the functioning of a web company. You will learn to do the following:

  • Enable the HEC
  • Create an authentication token
  • Perform a basic POST request with the HEC token using PowerShell
  • Verify the request and response
  • Use the JavaScript logging library to interface with HEC.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Enabling the HEC

In this section, we'll talk about the HEC and how to enable it, because, as we noted in the preceding section, it is not, by default, enabled. The HEC was created with the Splunk user in mind, as a way to make the transfer of large amounts of data from web applications to Splunk much easier. To enable the HEC in your local Splunk instance, perform the following steps, after which you can refer the following screenshot:

  1. Click on Settings | Data Inputs.
  2. Click on HTTP Event Collector.
  3. Click on the Global Settings button in the upper-right corner of the page.
  4. Enable All Tokens.
  5. In the Default Source Type, type access_custom.
  6. In the Default Index, type main.
  7. Uncheck Enable SSL for our testing purposes only. This is not recommended in production.
  8. Leave the rest at the default settings.
    Enabling the HEC

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Generating an HEC authentication token

Next, you will need to generate an HEC authentication token. The HEC token will ensure that no unknown applications will be able to send data to Splunk. This HEC authentication token will be sent in the HTTP headers of the incoming request to Splunk. Without this token, the response would typically indicate a status code 401 (Unauthorized error).

The HEC token will also enable you to override the source tag of all incoming data. This makes it easy to differentiate data streams later, based on where the data is coming from. It is best practice to create a separate token for each application. If something goes wrong with one application, say it started flooding your Splunk instance with unwanted data, it will then be easy enough to disable that associated token to mitigate the issue. Follow these instructions:

  1. Click Settings | Data Inputs.
  2. Find HTTP Event Collector.
  3. Click New Token.
  4. In the Name field, enter Demo1.
  5. Leave the other fields as-is.
  6. Click Next to proceed.
    Generating an HEC authentication token

You will see an Input Settings page that looks like the following screenshot. Follow these instructions:

  1. In the Input Settings page, you will create a new Source Type.
  2. In the first Source type section, click New.
  3. Type ec_demo1 as the Source Type.
    Generating an HEC authentication token
  4. In the Index section, select main as the allowed index.
  5. Select main as the Default Index as well.
    Generating an HEC authentication token
  6. Click Review to proceed.
  7. Verify your work against the following screenshot, then click on Submit.
    Generating an HEC authentication token
  8. Once you are done, go back to Data Inputs | HTTP Event Collector and you should see the newly-generated Token Value. Copy or take note of this value as you will need it for the exercises in this chapter. This value is unique to every Splunk instance. An example of a token value is highlighted in the following image:
    Generating an HEC authentication token

As you have learned in the previous chapters, everything that you change in the Splunk UI generally makes a change to a configuration file. In this case, the new EC Inputs modified the c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf file with the upcoming content.

You need to modify this file now and add the Cross Origin Resource Sharing (CORS) policy. This will allow our REST API calls to the HEC to work later on. This is not necessarily done in a production environment.

Run this Windows command to create the inputs.conf file.

C:\> notepad c:\splunk\etc\apps\splunk_httpinput\local\inputs.conf

Modify the file by adding the highlighted line under the [http] stanza. Take note of the port in the URL; if you used a different port to launch your HTTP server, then make sure you change it here as well. We use port 8080 here because it is commonly used for this purpose, but check to make sure it is correct for your system:

[http] 
disabled = 0 
index = main 
sourcetype = access_custom 
enableSSL = 0 
crossOriginSharingPolicy = http://localhost:8080 
 
[http://Demo1] 
disabled = 0 
index = main 
indexes = main 
sourcetype = ec_demo1 
token = 49CB0FF2-C685-4277-B744-6550516175CF 

Restart Splunk for the changes to take effect:

C:\> c:\splunk\bin\splunk restart

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

How to test the HEC with cURL and PowerShell

Now we want to test the HEC. Here, we want to do a HEC  POST request. Splunk will expect the information below, which is the basic information you need to make a successful HEC POST. Again, notice that the port used is 8088, which is commonly used, but this can be configured according to the needs of your system:

  • URL: http://localhost:8088/services/collector
  • Custom Header: Authorization
  • Custom Header Value: Splunk <token>
  • Content Type: application/json
  • Body: { "event": "this is your message" }

An easy and quick way to test the HTTP event collector is using cURL (or curl, as it is sometimes written). cURL is included in Mac OS X and Linux, and helps to transfer data to or from a server. For this purpose, it can use one of many different protocols, including HTTP and HTTPS (we will give you another way this time using Windows in the following section).

Here is the command for cURL:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk <token>' -d '{"event":"this is your message"}'

Although doing a one-time test in cURL is convenient, it is not included in Windows by default and you cannot further enhance your test programmatically.

Since we wrote this book in the context of Windows users, let us use PowerShell, which is a Windows-based scripting language and automation platform that is helpful for managing your information systems. It uses the .NET platform to give you the functionality needed to help control your Windows systems.

To bring up PowerShell, use the following steps:

  1. Do a search in your Windows machine for Windows PowerShell ISE.
  2. Right-click it and select Run as Administrator.

Notice that the ISE will have two panels; the top one is where you write your script, and the bottom one, which is an actual PowerShell shell, is where you can execute commands. Once you create your script, a clickable Run Script icon will execute the script and return the output at the bottom panel. You can also press  F5 to execute the script.

Now, to move ahead with this exercise, perform the following steps:

  1. Copy or write down the following script into the upper panel. Since all lines marked with a hash sign (#) are comments and are not executed, you can choose to include these or not.
  2. Pay very close attention to the first statement, where you will need to replace HEC TOKEN GOES HERE with the token you have just generated. It is the one you were asked to save and write down previously.
          # Change your HEC Token here 
          $token = 'HEC TOKEN GOES HERE' 
     
          # Disable SSL Validation 
          [System.Net.ServicePointManager]::ServerCertificateValidationCallback 
          = {$true} 
     
          # Create a dictionary object to contain the HTTP headers 
          $headers = New-Object 
          "System.Collections.Generic.Dictionary[[String],[String]]" 
     
          # Create an HTTP Authorization header with the EC Token 
          $headers.Add("Authorization", 'Splunk ' + $token) 
     
          # Create the JSON data that will be sent along with the POST request 
          $event = @{ 
            event="demo event 1" 
          } 
          $json = $event | ConvertTo-Json 
     
          # Initiate the POST request including the headers and the 
            JSON payload 
          $response = Invoke-RestMethod 
          'http://localhost:8088/services/collector' -Method Post 
          -Body $json -ContentType 'application/json' -Headers $headers 
     
          # Echo the response 
          Write-Host $response  
    
  3. Again, remember, if you have any problems, before executing, you needed to swap the $token value as indicated with the one that was generated by the creation of the HEC Token, which you created with these steps earlier: SettingsData InputsHTTP Event Collector (Token Value).  

    Here, we have produced a screenshot of the final script with a token in the machine. You will also see this in the lab section in GitHub: https://github.com/ericksond/splunk-essentials/tree/master/labs/chapter08.

  4. After you have replaced the token value, click the Run Script button highlighted in the following screenshot, or hit F5 on your keyboard.
    How to test the HEC with cURL and PowerShell
  5. If all goes well, you should see the output of the script as follows:
          @{text=Success; code=0}  
    
  6. Now that you know how the testing works, go into Splunk and run the following search command:
          SPL> index=main sourcetype=ec_demo1
    
    
  7. You should now see your events.

Although doing this test in PowerShell was a little more complex than a mere cURL command, any Windows admin can now use the very same script you wrote and for a more practical application. For example, you can use this base script to check the status of a particular application and based on the conditions, report it back to Splunk.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Using the HEC with dynamic UI events

In the previous chapter, we created a dashboard using D3.js that renders a chart with Splunk data in it. You also learned how to use the HTTP server module as a simple web server. We will use the same setup for the next exercise.

First, we will create a static HTML file with form buttons that we can click. Every click event will generate a Splunk HEC POST in the background to capture the details of the click event. The code that follows is by no means secure and should not be used in production. This is for example purposes only.

Create a new HTML file inside the c:\dashboard\public directory and call it ectest.html. Copy or type in the following contents of the HTML file:

<!-- c:\dashboard\public\ectest.html --> 
<!doctype html> 
<head> 
  <title>EC Tester</title> 
</head> 
<body> 
  <h3>HTTP Event Collector UI Tester</h3> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 1 button clicked.')">Click Event 1</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 2 button clicked.')">Click Event 2</button> 
  <button type="button" id="btn1" onClick="splunkIt(this, 'Event 3 button clicked.')">Click Event 3</button> 
  <script src="/ectest.js"></script> 
</body> 
</html> 

Now run the HTTP server. Open a command prompt, and type in the following command:

C:\> http-server -p 8080 c:\dashboard\public

Launch a browser tab and navigate to http://localhost:8080/ectest.html. The following screenshot shows how the page will look:

Using the HEC with dynamic UI events

Now let us create the JavaScript file that will be monitoring the button click events and sending the HEC data to Splunk.

Create a new JavaScript file inside the c:\dashboard\public directory and call it ectest.js. Change the ecToken value in the first line of the JavaScript file to your own HEC token:

// public/ectest.js 
var ecToken = '13EA3875-2879-4F44-AF0A-3ED336CB5BCA' 
 
function splunkIt(object, clickEvent) { 
  console.log(object, event); 
  var xhr = new XMLHttpRequest(); 
 
  xhr.open('POST', 'http://localhost:8088/services/collector', true); 
  xhr.setRequestHeader('Authorization', 'Splunk ' + ecToken); 
  xhr.withCredentials = true; 
  xhr.onload = function() { 
    if (xhr.status === 200) { 
      var userInfo = JSON.parse(xhr.responseText); 
    } 
  }; 
  xhr.send(JSON.stringify({ 
    event: clickEvent 
  })); 
}; 

Now that you have the JavaScript in place, reload your browser or navigate to the following URL again: http://localhost:8080/ectest.html. This was tested with Google Chrome.

You can open the Developer Tools and the JavaScript console within Google Chrome. The JavaScript console allows you to use basic JavaScript statements and commands specific to the console when the page is in the browser, thus allowing you to quickly and easily debug it. With the console, you can see any diagnostics, view both raw and structured data, filter and control the output, and modify various elements within the page.

Click the buttons and run the search query against Splunk. Set the time frame to a 5-minute window real time and watch the data flow in as soon as you click the buttons.

SPL> index=main sourcetype=ec_demo1
Using the HEC with dynamic UI events

At this point, you have successfully created a web page that sends signals to Splunk every time a UI event is performed.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

JavaScript logging with the HEC

Perhaps the best use case for using the HEC is with logging libraries. These allow the automatic collection of data when code is executing.

We will use the HEC Stream for Bunyan. Bunyan is a simple and fast JSON logging library for Node.js services. Here are additional references you can look at regarding these projects:

The following sample code will inject an event to our initially defined HEC. Every time this is executed, it will create a Splunk event with a random number. You can find the complete code by cloning the external dashboard repository:

https://github.com/ericksond/splunk-external-dashboard

After cloning or unzipping the package, run this series of commands to install the dependencies:

C:\> git clone https://github.com/ericksond/splunk-external-dashboard.git 
C:\> cd splunk-external-dashboard ( or cd dashboard) 
C:\dashboard> npm install 
C:\dashboard> notepad heclogging.js

Change the HEC token by replacing the value in this field:

var config = { 
  token: "TOKEN GOES HERE", 
  url: "http://localhost:8088" 
}; 

Save the file then run the code:

C:\> node heclogging.js

If all goes well, you should see an output similar to this:

C:\> C:\dashboard>node heclogging.js
Sending payload { event: 'logging event #0.8825588226318359' }

Now go back to Splunk and check if your data made it in:

SPL> index=main sourcetype=ec_demo1

Here is an example event that shows that the code worked:

JavaScript logging with the HEC

Here is the complete working code for the heclogging.js file:

var splunkBunyan = require("splunk-bunyan-logger"); 
var bunyan = require("bunyan"); 
 
var config = { 
  token: "13EA3875-2879-4F44-AF0A-3ED336CB5BCA", 
  url: "http://localhost:8088" 
}; 
var splunkStream = splunkBunyan.createStream(config); 
 
splunkStream.on("error", function(err, context) { 
  console.log("Error", err, "Context", context) 
}); 
 
var Logger = bunyan.createLogger({ 
    name: "logger", 
    streams: [ 
        splunkStream 
    ] 
}); 
 
var payload = { 
  event: "logging event #" + Math.random() 
}; 
 
console.log("Sending payload", payload); 
Logger.info(payload, "Posted successfully.") 

You can retrieve the code files used in this chapter in the following GitHub repository: https://github.com/ericksond/splunk-external-dashboard.

Summary

In this chapter, you have learned about the HEC and how it can be used to send HTTP and HTTPS data directly from an application to Splunk. To do this, you learned how to create a token and generate a token key to be used to access the data. You have also learned how to use cURL (for MAC or Linux) and PowerShell (for Windows) to collect the data. And you have created a basic website with buttons that will send the events back to Splunk. Lastly, you have learned how to search the data you have collected from the application.

In the next chapter, we'll go on and learn about recommended ways to work with Splunk.