Chapter 7. Splunk SDK for JavaScript and D3.js
In this chapter, we go on to learn about the Splunk Software Development Kit (SDK) and D3.js. Unlike previous chapters, here we will learn some special ways to interact with Splunk, ways that will allow us to create interesting and vibrant applications.
Specifically, in this chapter we will do the following:
- Learn about the Splunk SDK
- Discuss how the SDK can extract data from Splunk
- Find out how a website can be set up to show and use data extracted via the Splunk SDK
- Learn about another important software tool, D3.js, and how it can be used to create useful and impressive data visualizations
We'll begin by talking about Splunk SDKs and how they can be used.
Introduction to Splunk SDKs
A software development kit (also called a SDK or DevKit) is usually a set of software development tools that allows the creation of applications for a certain software package or software framework, but can also refer to a development kit for a computer system, operating system (OS), hardware platform, or even a video game system. We will use the Splunk SDK as a means of extracting data from Splunk and using it for external purposes (for example, a public website).
Splunk actually has several software development kits that sit on top of the REST API. These kits are for Python, Java, JavaScript, PHP, Ruby, and C#, and they allow developers to do all kinds of different things, such as integrating Splunk with third-party tools, logging directly into Splunk, extracting data to create archives, and others. They allow developers to do this using popular programming languages.
These Splunk SDKs do several specific tasks:
- They handle HTTP access.
- By utilizing a user ID and password, they authenticate the user.
- They manage name spaces, which are the user and app context for accessing a resource, specified by a Splunk user name, a Splunk app, and a sharing mode.
- They simplify access to REST endpoints, by making it easy to map a REST endpoint to a URI and HTTP method. (For more explanation, see https://github.com/Mach-II/Mach-II-Framework/wiki/Introduction-to-REST-Endpoints.)
- They build the appropriate URL for an endpoint.
- Rather than providing the raw XML, JSON, or CSV, they return the events in a structure that has clear key-value pairs, thus simplifying the output for searches.
Practical applications of Splunk's SDK
In this chapter, we will show you how you can extract data from Splunk and display it externally through a web server without the need to log in to Splunk. This is extremely useful when the use case demands a real-time dashboard that is publicly displayed within your organization and that does not require logging in to Splunk.
To achieve this, we will use the Splunk SDK to extract the data from Splunk using a Node.js cron job and dump the payload into JSON files. A cron job involves the cron expressions we learned about in Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches, which allow the developer to schedule alerts and other processes very precisely. The term payload is frequently used in computer programming to represent what the person receiving the output from the program is most interested in. The JSON files, which are files based on key-value pairs, will be served by a static web server. We will then create an HTML page with D3.js code that can utilize the JSON files to dynamically display the data in charts.
D3.js is an important tool for Splunk. It is a library, written in JavaScript, that uses data to change documents. It uses tools familiar to web designers, such as Hypertext Markup Language (HTML), Scalable Vector Graphics (SVG), and Cascading Style Sheets (CSS), to create complex and effective visualizations. These visualizations are based on the Document Object Model (DOM). The DOM is the interface used for HTML and XML documents and is used with a web browser. Using the DOM permits the developer to treat documents in a structured way, so that features can be duplicated throughout a website. In this way, a document's style, structure, and content can be easily manipulated.
We will not delve into the details of the Node.js programming language here. Suffice to say that it is an open source, cross-platform runtime environment designed for the development of server-side web applications. For more information, you can visit the website listed in the following Prerequisites section.
Prerequisites
There are several steps you need to take to prepare for using the Splunk SDK:
- To add Node.js, download the Node.js framework (LTS version) from https://nodejs.org/en/. Check that you have successfully installed the Node.js framework by running this command in the Windows command prompt. The command returns the version; your version may be different:
C:\> node -v V5.1.1 - Create your working directory:
C:\> mkdir dashboard C:\> cd dashboard - Install the Splunk SDK using the Node Package Manager (npm):
C:\> C:\dashboard> npm install splunk-sdkThis will create a
node_modulesdirectory that will contain the Splunk SDK for the JavaScript package and all its dependencies. Ignore any warnings. - Install the
cronpackage, which will allow us to run scheduled jobs. Again, you will use the Node Package Manager (npm). Ignore any warnings:C:\> C:\dashboard> npm install cron - Choose a text editor that works for you. We recommend the Atom text editor from https://atom.io/. Go there now, download the Windows installer, and launch Atom.
- You may download the code used in this chapter from the following link or use Git to clone it:
- https://github.com/ericksond/splunk-external-dashboard
- Git clone https://github.com/ericksond/splunk-external-dashboard.git
Creating a CRON Job
The CRON package allows us to use six or seven subexpressions to set up a scheduled alert. First, let us create something basic that will show us how to use the cron package to invoke a function. Using Atom or another text editor of choice, create a new file in the c:\dashboard directory and call it jobs.js. Paste the following code into the new file and save it:
// c:\dashboard\jobs.js
// require packages
var CronJob = require('cron').CronJob
// create a basic function
function hello_splunk() {
console.log('hello splunk. give me your data.')
}
// create a cron job that executes the hello_splunk
// function every 5 seconds
new CronJob('*/5 * * * * *', function() {
hello_splunk()
}, function() {
// execute when job stops
}, true)
Now run this script using the Node interpreter:
C:\> C:\dashboard> node jobs.js
The code will not terminate and will continuously emit the message every 5 seconds. You learned how to create cron schedules in Splunk in Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches. The same principle we used there applies here.
We will use this to query a Splunk saved search every few minutes and generate a JSON file out of the resulting data. In production scenarios, this is how often you want to query Splunk using the SDK. Later on, you will see that the code to query Splunk will include the username and password. This makes sense; if you use the SDK in a client-side JavaScript file, then anyone who knows how to open a JavaScript file could see the authentication credentials.
But for now, we need to stop the message or it will just keep going. To halt the execution of the Node script, hit Ctrl + C in the command window.
Creating a saved search
Next, we will create a saved search that we will eventually use to create a visualization. Follow these steps carefully:
- Open Splunk and select the Destinations app.
- Click Settings | Searches, reports, and alerts. Click on New.
- Ensure that the Destinations app is selected.
- In Search Name, type
sdk_status_codes. - In the Search, type this:
SPL> index=main | timechart span=1h count by http_status_code - In Start time, type
-24h@h. - In Finish time, type
now. - Click the checkbox for Accelerate this search. Select 7 Days.
- Use the following screenshot as a guide, then click Save:
Prerequisites
There are several steps you need to take to prepare for using the Splunk SDK:
- To add Node.js, download the Node.js framework (LTS version) from https://nodejs.org/en/. Check that you have successfully installed the Node.js framework by running this command in the Windows command prompt. The command returns the version; your version may be different:
C:\> node -v V5.1.1 - Create your working directory:
C:\> mkdir dashboard C:\> cd dashboard - Install the Splunk SDK using the Node Package Manager (npm):
C:\> C:\dashboard> npm install splunk-sdkThis will create a
node_modulesdirectory that will contain the Splunk SDK for the JavaScript package and all its dependencies. Ignore any warnings. - Install the
cronpackage, which will allow us to run scheduled jobs. Again, you will use the Node Package Manager (npm). Ignore any warnings:C:\> C:\dashboard> npm install cron - Choose a text editor that works for you. We recommend the Atom text editor from https://atom.io/. Go there now, download the Windows installer, and launch Atom.
- You may download the code used in this chapter from the following link or use Git to clone it:
- https://github.com/ericksond/splunk-external-dashboard
- Git clone https://github.com/ericksond/splunk-external-dashboard.git
Creating a CRON Job
The CRON package allows us to use six or seven subexpressions to set up a scheduled alert. First, let us create something basic that will show us how to use the cron package to invoke a function. Using Atom or another text editor of choice, create a new file in the c:\dashboard directory and call it jobs.js. Paste the following code into the new file and save it:
// c:\dashboard\jobs.js
// require packages
var CronJob = require('cron').CronJob
// create a basic function
function hello_splunk() {
console.log('hello splunk. give me your data.')
}
// create a cron job that executes the hello_splunk
// function every 5 seconds
new CronJob('*/5 * * * * *', function() {
hello_splunk()
}, function() {
// execute when job stops
}, true)
Now run this script using the Node interpreter:
C:\> C:\dashboard> node jobs.js
The code will not terminate and will continuously emit the message every 5 seconds. You learned how to create cron schedules in Splunk in Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches. The same principle we used there applies here.
We will use this to query a Splunk saved search every few minutes and generate a JSON file out of the resulting data. In production scenarios, this is how often you want to query Splunk using the SDK. Later on, you will see that the code to query Splunk will include the username and password. This makes sense; if you use the SDK in a client-side JavaScript file, then anyone who knows how to open a JavaScript file could see the authentication credentials.
But for now, we need to stop the message or it will just keep going. To halt the execution of the Node script, hit Ctrl + C in the command window.
Creating a saved search
Next, we will create a saved search that we will eventually use to create a visualization. Follow these steps carefully:
- Open Splunk and select the Destinations app.
- Click Settings | Searches, reports, and alerts. Click on New.
- Ensure that the Destinations app is selected.
- In Search Name, type
sdk_status_codes. - In the Search, type this:
SPL> index=main | timechart span=1h count by http_status_code - In Start time, type
-24h@h. - In Finish time, type
now. - Click the checkbox for Accelerate this search. Select 7 Days.
- Use the following screenshot as a guide, then click Save:
Creating a CRON Job
The CRON package allows us to use six or seven subexpressions to set up a scheduled alert. First, let us create something basic that will show us how to use the cron package to invoke a function. Using Atom or another text editor of choice, create a new file in the c:\dashboard directory and call it jobs.js. Paste the following code into the new file and save it:
// c:\dashboard\jobs.js
// require packages
var CronJob = require('cron').CronJob
// create a basic function
function hello_splunk() {
console.log('hello splunk. give me your data.')
}
// create a cron job that executes the hello_splunk
// function every 5 seconds
new CronJob('*/5 * * * * *', function() {
hello_splunk()
}, function() {
// execute when job stops
}, true)
Now run this script using the Node interpreter:
C:\> C:\dashboard> node jobs.js
The code will not terminate and will continuously emit the message every 5 seconds. You learned how to create cron schedules in Splunk in Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches. The same principle we used there applies here.
We will use this to query a Splunk saved search every few minutes and generate a JSON file out of the resulting data. In production scenarios, this is how often you want to query Splunk using the SDK. Later on, you will see that the code to query Splunk will include the username and password. This makes sense; if you use the SDK in a client-side JavaScript file, then anyone who knows how to open a JavaScript file could see the authentication credentials.
But for now, we need to stop the message or it will just keep going. To halt the execution of the Node script, hit Ctrl + C in the command window.
Creating a saved search
Next, we will create a saved search that we will eventually use to create a visualization. Follow these steps carefully:
- Open Splunk and select the Destinations app.
- Click Settings | Searches, reports, and alerts. Click on New.
- Ensure that the Destinations app is selected.
- In Search Name, type
sdk_status_codes. - In the Search, type this:
SPL> index=main | timechart span=1h count by http_status_code - In Start time, type
-24h@h. - In Finish time, type
now. - Click the checkbox for Accelerate this search. Select 7 Days.
- Use the following screenshot as a guide, then click Save:
Creating a saved search
Next, we will create a saved search that we will eventually use to create a visualization. Follow these steps carefully:
- Open Splunk and select the Destinations app.
- Click Settings | Searches, reports, and alerts. Click on New.
- Ensure that the Destinations app is selected.
- In Search Name, type
sdk_status_codes. - In the Search, type this:
SPL> index=main | timechart span=1h count by http_status_code - In Start time, type
-24h@h. - In Finish time, type
now. - Click the checkbox for Accelerate this search. Select 7 Days.
- Use the following screenshot as a guide, then click Save:
Creating the final dashboard\jobs.js
Here is the final jobs.js file we will use. Copy this block of code and overwrite the current jobs.js file. We will break this down into functions later and try to explain what the coding does, without putting too much emphasis on the Node.js code itself.
Note
WARNING
When copying from PDF and other files, the encoding may be different, such that it breaks the apostrophes and quotation marks. If you encounter this, just search and replace all apostrophes (') and quotation marks (").
var CronJob = require('cron').CronJob
var splunkjs = require('splunk-sdk')
var fs = require('fs')
new CronJob('*/30 * * * * *', function() {
// fetch the saved searchName
fetchSavedSearch(renderResults, 'sdk_status_codes')
}, function() {}, true)
var service = new splunkjs.Service({
username:"admin",
password:"changed", // Use your own admin password
scheme:"https",
host:"localhost",
port:"8089",
version:"6.4"
});
function renderResults(data, searchName) {
console.log(data) // print out the result in the console
fs.writeFile(__dirname + '/public/'+searchName+'.json', JSON.stringify(data),
function (err) {
if (err) throw err
console.log(new Date() + ' Written ' + searchName + '.json')
})
}
function fetchSavedSearch(callback, searchName) {
var savedSearches = service.savedSearches({
owner: "admin",
app: "destinations"
});
savedSearches.fetch(function (err, savedSearches) {
if (err) {
console.log(err)
callback('error', searchName)
} else {
if (savedSearches.item(searchName) != null) {
var savedSearch = savedSearches.item(searchName)
savedSearch.dispatch({
force_dispatch: false,
}, function(e, job) {
if (e) {
console.log(e)
callback('error', searchName)
} else {
job.setTTL(60, function(err, job) {});
job.track({
period: 200
}, {
done: function(job) {
job.results({
count: 0
}, function(err, results, job) {
console.log('Job Succeeded: ' + searchName)
callback({
fields: results.fields,
rows: results.rows
}, searchName)
});
},
failed: function(job) {
console.log("Job failed")
callback('failed', searchName)
},
error: function(err) {
console.log(err);
callback('error', searchName)
}
})
}
})
}
}
})
}
Tip
Be sure you have altered the password changed to reflect your actual admin password.
Before you actually run the previous code, you will need to first create a public directory: - c:\dashboard\public. In production work, code should be resilient enough to generate a directory if it needs it. However, we will not go over how to do that in this book; instead we will make a new public directory the traditional way, by using the mkdir command in the command window:
C:\> C:\dashboard> mkdir public
Now let us break the code down into sections to examine what it is doing:
// require packages
var CronJob = require('cron').CronJob
var splunkjs = require('splunk-sdk')
var fs = require('fs')
In this first code block, you are loading a Node.js module. You declare the variables CronJob, splunkjs, and fs, and then use require to read the JavaScript file, execute it, and return the export objects. These modules are the fundamental building blocks of Node.js and they are mapped directly to files. Remember they are located in the node_modules directory.
The important thing to take from this is that you are instantiating an object of the particular module. If the modules export objects, then you can access the exported functions via dot notation. Here is an example:
splunkjs.Service()
Let's go on to the next section:
// cron job runs every 30 sec
new CronJob('*/30*****',function() {
// fetch the saved searchName
fetchSavedSearch(renderResults, 'sdk_status_codes)
}, function() {}, true)
You have seen this before. This is the cron job. In this case, the cron job is scheduled to run every 30 seconds, executing the fetchSavedSearch() function in each iteration.
Note that the fetchSavedSearch() function takes two arguments. The first one is a callback function and the second is the saved search name.
Here is the next block of code:
// define the splunk service
var service=new splunkjs.Service({
username:"admin",
password:"changed", // Use your own admin account
scheme: "https",
host: "localhost",
port:"8089",
version:"6.4",
});
This code block is now using the Splunk SDK to declare a service. The data you see is straightforward and includes the basic information required to connect to Splunk.
This is the main reason why you should not create the SDK on a JavaScript file as part of your website. You do not want this information exposed to your users or the cloud. By creating a cron jobs file, you can safely run it on the server without exposing this information.
In this block, we will render the results of the saved search and write the results to a JSON file:
// render the results of the saved search as part of the callback
// and write the payload to a JSON file
function renderResults(data, searchName) {
console.log(data) //print out the result in the console
// generate the json file
fs.writeFile(_dirname + '/public/'+searchName+'.json', JSON.stringify(data),
function(err) {
if (err) throw err
console.log(new Date() + 'Written ' + searchName + '.json'
})
}
The renderResults() function is responsible for printing the result of the saved search to the console and writing it into a JSON file. To achieve this, we used the native Node.js File System module (fs). The code expects that a public directory has been created beforehand. The code for the fetchSavedSearch function is listed below:
// fetch saved searches function
function fetchSavedSearch(callback, searchName) {
var savedSearches=service.savedSearches({
owner: "admin",
app: "destinations"
});
savedSearches.fetch(function (err, savedSearches) {
if (err) {
console.log(err)
callback('error', searchName)
} else {
if (savedSearches.item(searchName) != null) {
var savedSearch = savedSearches.item(searchName)
savedSearch.dispatch({
force_dispatch: false,
}, function(e, job) {
if (e) {
console.log(e)
callback('error', searchName)
} else {
job.setTTL(60, function(err, job) {});
job.track({
period: 200
}, {
done: function(job) {
job.results({
count: 0
}, function(err, results, job) {
console.log('Job Succeeded: ' + searchName)
callback({
fields: results.fields,
rows: results.rows
}, searchName)
});
},
failed: function(job) {
console log("Job failed")
callback('failed', searchName)
},
error: function(job) {
console.log(err);
callback('error', searchName)
}
})
}
})
}
}
})
}
This function (fetchSavedSearch) does the bulk of the work for you. First, the function takes two arguments, a callback function and the saved search name. Read about JavaScript callbacks on the web to better understand them. (One place to look is https://www.sitepoint.com/demystifying-javascript-closures-callbacks-iifes/.) It might take a while to get your head around it, especially if you are more experienced in programming languages that do not treat functions as first-class objects. In JavaScript, you can pass around functions as objects. This is an important concept because JavaScript is asynchronous and will run functions independently of each other concurrently, unlike other programming languages that interpret the code consecutively.
The function takes in two arguments: the callback function and saved search name string:
fetchSavedSearch(renderResults, 'sdk_status_codes')
What this means is that anytime you see a callback() function executed within the fetchSavedSearch() function, it will actually invoke the renderResults() function and pass all arguments to it.
You can optionally add more saved searches by simply creating a second line of this function invocation, for example:
fetchSavedSearch(renderResults, 'my_second_saved_search')
Since the saved search name is being passed around, it will even make sure that the JSON file name is the same as the saved search name.
Now go ahead and run the job, and wait for about 30 seconds for the first run:
C:\> C:\dashboard> node jobs.js
Every time the cron executes, it will print the result of the saved search to the console. It will also tell you that it has successfully written the JSON file inside the public directory:
You can terminate the job with Ctrl + C. Later in the chapter, we will instruct you to turn it on again so we can see the real data stream against the D3 charts.
HTTP server
Now that we have the JSON file available in the public directory, it is time to make it live using a web server.
Install the http-server module with this command:
C:\> C:\dashboard> npm install http-server -g
The -g flag will ensure that this module is available globally, regardless of your current directory.
Now serve the files in the public directory with this command:
C:\> C:\dashboard> http-server -p 8080 C:\dashboard\public
Windows Firewall may prompt you to allow this app to use the port. Click on Allow access.
Now open a browser and go to the following URL: http://localhost:8080. If for some reason you have an application that uses port 8080, change it to something else (for example, 9999).
You will know that it works when you see the page shown in the following screenshot:
Note that you can just as easily click on the sdk_status_codes.json link and download it to your machine, just like any Internet object.
You now have your very own static web server. We will use this for the entire duration of the chapter to host the HTML and JavaScript files.
dashboard\public\index.html
For this to work, make sure you are using a modern browser (Google Chrome, Safari, or Firefox) and ensure that you have updated it.
Create a new document in c:\dashboard\public and name it index.html. Type in or copy this into the file:
<!-- c:\dashboard\public\index.html --> <!doctype html> <head> <title>Real Time Dashboard</title> <link rel="stylesheet" href="/status_codes_chart.css"></style> </head> <body> <h3>Real Time Dashboard</h3> <div id="status_codes_chart"></div> <script src="https://d3js.org/d3.v3.min.js" charset="utf-8"></script> <script src="/status_codes_chart.js"></script> </body> </html>
We are including the online version of the D3.v3 script in the script declaration. We also need to create a new status_codes_chart.js file. Go ahead and do it now. Writing D3.js is beyond the scope of this book, but we will highlight the most important parts of this code block in the event that you wish to customize it. If you are reading this in a book, download the source code from GitHub.
You can find the code in the URL or simply clone the Git repository at https://github.com/ericksond/splunk-external-dashboard.
// c:\dashboard\public\status_codes_chart.js
(function() {
var chartId = '#status_codes_chart'
var margin = {top: 60, right: 50, bottom: 20, left: 20}
var width = parseInt(d3.select(chartId).style('width')) - margin.left -
margin.right
var height = parseInt(d3.select(chartId).style('height')) - margin.top -
margin.bottom
//var width = 600;
//var height = 300;
var x = d3.scale.ordinal()
.rangeRoundBands([0, width]);
var y = d3.scale.linear()
.rangeRound([height, 0]);
var z = d3.scale.category10();
var xAxis = d3.svg.axis()
.scale(x)
.orient("bottom")
.tickFormat(d3.time.format("%I:%M"))
var yAxis = d3.svg.axis()
.scale(y)
.orient("right")
var svg = d3.select(chartId).append("svg")
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," +
margin.top + ")");
var categories = ["200", "301", "302", "404", "500"]
drawChart()
function drawChart() {
d3.json('/sdk_status_codes.json', function(error, json) {
if (error) return console.warn(error)
var data = []
for(var i = 0; i < json.rows.length; i++) {
data.push({
"date": new Date(json.rows[i][0]),
"200": parseInt(json.rows[i][1]),
"301": parseInt(json.rows[i][2]),
"302": parseInt(json.rows[i][3]),
"404": parseInt(json.rows[i][4]),
"500": parseInt(json.rows[i][5])
})
}
var layers = d3.layout.stack()(categories.map(function(c) {
return data.map(function(d) {
return {x: d.date, y: d[c]}
})
}))
x.domain(layers[0].map(function(d) { return d.x; }));
y.domain([0, d3.max(layers[layers.length - 1], function(d) {
return d.y0 + d.y; })]).nice();
xAxis.tickValues(x.domain().filter(function(d, i) { return !(i % 3); }))
var layer = svg.selectAll(".layer")
.data(layers)
.enter().append("g")
.attr("class", "layer")
.style("fill", function(d, i) { return z(i); })
layer.selectAll("rect")
.data(function(d) { return d; })
.enter().append("rect")
.transition()
.delay(function(d, i) {
return i * 20
})
.duration(200)
.attr("x", function(d) { return x(d.x); })
.attr("y", function(d) { return y(d.y + d.y0); })
.attr("height", function(d) { return y(d.y0) - y(d.y + d.y0); })
.attr("width", x.rangeBand() - 1)
layer.selectAll("text")
.data(function(d) { return d; })
.enter().append("text")
.text(function(d) { if (d.y > 50) { return d.y }; } )
.attr({
'text-anchor': 'middle',
x: function(d, i) { return x(d.x) + x.rangeBand() / 2 },
y: function(d) { return y(d.y + d.y0) + 12 },
'font-family': 'sans-serif',
'font-size': '9px',
fill: '#fff'
})
var legend = svg.selectAll(".legend")
.data(categories)
.enter().append("g")
.attr("class", "legend")
.attr("transform", function(d, i) {
return "translate(" + i * -120 + ", 15)"; } )
legend.append("rect")
.attr({
"x": width - 18,
"y": -50,
"width": 18,
"height": 18,
})
.style("fill", function(d, i) { return z(i); } )
legend.append("text")
.attr({
"x": width - 24,
"y": -41,
"dy": ".35em",
"font-size": "11px",
"font-face": "sans-serif",
"fill": "#999"
})
.style("text-anchor", "end")
.text(function(d) { return d; })
svg.append("g")
.attr("class", "axis axis--x")
.attr("transform", "translate(0," + height + ")")
.attr("fill", "#999")
.transition()
.delay(function(d, i) {
return i * 50
})
.duration(500)
.call(xAxis);
svg.append("g")
.attr("class", "axis axis--y")
.attr("transform", "translate(" + width + ",0)")
.attr("fill", "#999")
.transition()
.delay(function(d, i) {
return i * 50
})
.duration(500)
.call(yAxis);
});
}
})()
If you wish to learn how to write D3.js scripts, go to https://d3js.org/.
We will now highlight the different parts of the JavaScript file:
(function() {
var chartId = '#status_codes_chart'
var margin = {top: 60, right: 50, bottom: 20, left: 20}
var width = parseInt(d3.select(chartId).style('width')) - margin.left -
margin.right
var height = parseInt(d3.select(chartId).style('height')) - margin.top -
margin.bottom
//var width = 600;
//var height = 300;
...
})()
This is the entry point of the script. Here you are declaring the div ID status_codes_chart that matched the one in the index.html file. The margins are also being defined here. The width and height can either be dynamically assigned (first set) or statically assigned (second set). If dynamically assigned, they will either fall back to the width and height defined in the CSS, or they will adjust based on the window size. Feel free to tweak this later on and see how the chart changes:
var x = d3.scale.ordinal()
.rangeRoundBands([0, width]);
var y = d3.scale.linear()
.rangeRound([height, 0]);
var z = d3.scale.category10();
var xAxis = d3.svg.axis()
.scale(x)
.orient("bottom")
.tickFormat(d3.time.format("%I:%M"))
var yAxis = d3.svg.axis()
.scale(y)
.orient("right")
This block of code consists of D3 elements that are being declared. Suffice to say that these are required to create the SVG screenshot needed by the chart:
var svg = d3.select(chartId).append("svg")
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," +
margin.top + ")");
var categories = ["200", "301", "302", "404", "500"]
drawChart()
The previous block of code is D3 creating the SVG object. Notice that d3.select(chartId) is actually equal to d3.select("#status_codes_chart"). This is telling the D3 script to create the SVG chart in that div.
The var categories[] array defines what fields will be shown in the chart. These numbers are the result of the query that you ran against Splunk. If you wish to omit a particular status code from the resulting bar chart, then remove it from this list. The list requires a minimum of one entry.
drawChart() is a function call, which we will show you now.
The drawChart() function will contain all the instructions to generate the data based on the JSON object that is being called through the d3.json() method:
function drawChart() {
d3.json('/sdk_status_codes.json', function(error, json) {
if (error) return console.warn(error)
var data = []
for(var i = 0; i < json.rows.length; i++) {
data.push({
"date": new Date(json.rows[i][0]),
"200": parseInt(json.rows[i][1]),
"301": parseInt(json.rows[i][2]),
"302": parseInt(json.rows[i][3]),
"404": parseInt(json.rows[i][4]),
"500": parseInt(json.rows[i][5])
})
}
The d3.json call essentially makes a GET request against the URL of our local JSON file, which then passes the data back to the function.
The for loop here is not part of D3.js. This is just needed to format the incoming data to match the data needed by D3:
var layers = d3.layout.stack()(categories.map(function(c) {
return data.map(function(d) {
return {x: d.date, y: d[c]}
})
}))
This block of code creates the layers of the chart.
dashboard\public\status_codes_chart.css
The source for this is at https://github.com/ericksond/splunk-external-dashboard.
Create a new CSS file in c:\dashboard\public and name it status_codes_chart.css:
text {
font-family: monospace;
}
.axis text {
font: 10px sans-serif;
}
.axis line,
.axis path {
fill: none;
stroke: #ccc;
shape-rendering: crispEdges;
}
.axis--x path {
display: none;
}
#status_codes_chart {
-webkit-touch-callout: none;
-webkit-user-select: none;
-khtml-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
/* Trying to get SVG to act like a greedy block in all browsers */
display: block;
width:50%;
height:400px;
}
Rendering the chart
Now that all the components are in place, reload the page and check the result at http://localhost:8080.
If this page is not loading, make sure the http-server process is running. Refer to the earlier sections in this chapter for how to run the http-server:
Now you have a stacked bar chart that is being fed with data from Splunk. Go ahead and run the jobs.js cron job again using the following command. You need to do this in a separate command prompt window:
C:\> C:\dashboard> node jobs.js
Wait a few minutes and refresh the page. See it change before your eyes. There are many ways to make this dynamically update in real time. You can use JavaScript or browser-based refresh plugins. You may also extend the D3.js code and create a redrawChart() function that updates certain data entry points. The important thing is that you can now create public dashboards that will display data from Splunk without the need to log in to the application. You will be in total control of this dashboard as well.
HTTP server
Now that we have the JSON file available in the public directory, it is time to make it live using a web server.
Install the http-server module with this command:
C:\> C:\dashboard> npm install http-server -g
The -g flag will ensure that this module is available globally, regardless of your current directory.
Now serve the files in the public directory with this command:
C:\> C:\dashboard> http-server -p 8080 C:\dashboard\public
Windows Firewall may prompt you to allow this app to use the port. Click on Allow access.
Now open a browser and go to the following URL: http://localhost:8080. If for some reason you have an application that uses port 8080, change it to something else (for example, 9999).
You will know that it works when you see the page shown in the following screenshot:
Note that you can just as easily click on the sdk_status_codes.json link and download it to your machine, just like any Internet object.
You now have your very own static web server. We will use this for the entire duration of the chapter to host the HTML and JavaScript files.
dashboard\public\index.html
For this to work, make sure you are using a modern browser (Google Chrome, Safari, or Firefox) and ensure that you have updated it.
Create a new document in c:\dashboard\public and name it index.html. Type in or copy this into the file:
<!-- c:\dashboard\public\index.html --> <!doctype html> <head> <title>Real Time Dashboard</title> <link rel="stylesheet" href="/status_codes_chart.css"></style> </head> <body> <h3>Real Time Dashboard</h3> <div id="status_codes_chart"></div> <script src="https://d3js.org/d3.v3.min.js" charset="utf-8"></script> <script src="/status_codes_chart.js"></script> </body> </html>
We are including the online version of the D3.v3 script in the script declaration. We also need to create a new status_codes_chart.js file. Go ahead and do it now. Writing D3.js is beyond the scope of this book, but we will highlight the most important parts of this code block in the event that you wish to customize it. If you are reading this in a book, download the source code from GitHub.
You can find the code in the URL or simply clone the Git repository at https://github.com/ericksond/splunk-external-dashboard.
// c:\dashboard\public\status_codes_chart.js
(function() {
var chartId = '#status_codes_chart'
var margin = {top: 60, right: 50, bottom: 20, left: 20}
var width = parseInt(d3.select(chartId).style('width')) - margin.left -
margin.right
var height = parseInt(d3.select(chartId).style('height')) - margin.top -
margin.bottom
//var width = 600;
//var height = 300;
var x = d3.scale.ordinal()
.rangeRoundBands([0, width]);
var y = d3.scale.linear()
.rangeRound([height, 0]);
var z = d3.scale.category10();
var xAxis = d3.svg.axis()
.scale(x)
.orient("bottom")
.tickFormat(d3.time.format("%I:%M"))
var yAxis = d3.svg.axis()
.scale(y)
.orient("right")
var svg = d3.select(chartId).append("svg")
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," +
margin.top + ")");
var categories = ["200", "301", "302", "404", "500"]
drawChart()
function drawChart() {
d3.json('/sdk_status_codes.json', function(error, json) {
if (error) return console.warn(error)
var data = []
for(var i = 0; i < json.rows.length; i++) {
data.push({
"date": new Date(json.rows[i][0]),
"200": parseInt(json.rows[i][1]),
"301": parseInt(json.rows[i][2]),
"302": parseInt(json.rows[i][3]),
"404": parseInt(json.rows[i][4]),
"500": parseInt(json.rows[i][5])
})
}
var layers = d3.layout.stack()(categories.map(function(c) {
return data.map(function(d) {
return {x: d.date, y: d[c]}
})
}))
x.domain(layers[0].map(function(d) { return d.x; }));
y.domain([0, d3.max(layers[layers.length - 1], function(d) {
return d.y0 + d.y; })]).nice();
xAxis.tickValues(x.domain().filter(function(d, i) { return !(i % 3); }))
var layer = svg.selectAll(".layer")
.data(layers)
.enter().append("g")
.attr("class", "layer")
.style("fill", function(d, i) { return z(i); })
layer.selectAll("rect")
.data(function(d) { return d; })
.enter().append("rect")
.transition()
.delay(function(d, i) {
return i * 20
})
.duration(200)
.attr("x", function(d) { return x(d.x); })
.attr("y", function(d) { return y(d.y + d.y0); })
.attr("height", function(d) { return y(d.y0) - y(d.y + d.y0); })
.attr("width", x.rangeBand() - 1)
layer.selectAll("text")
.data(function(d) { return d; })
.enter().append("text")
.text(function(d) { if (d.y > 50) { return d.y }; } )
.attr({
'text-anchor': 'middle',
x: function(d, i) { return x(d.x) + x.rangeBand() / 2 },
y: function(d) { return y(d.y + d.y0) + 12 },
'font-family': 'sans-serif',
'font-size': '9px',
fill: '#fff'
})
var legend = svg.selectAll(".legend")
.data(categories)
.enter().append("g")
.attr("class", "legend")
.attr("transform", function(d, i) {
return "translate(" + i * -120 + ", 15)"; } )
legend.append("rect")
.attr({
"x": width - 18,
"y": -50,
"width": 18,
"height": 18,
})
.style("fill", function(d, i) { return z(i); } )
legend.append("text")
.attr({
"x": width - 24,
"y": -41,
"dy": ".35em",
"font-size": "11px",
"font-face": "sans-serif",
"fill": "#999"
})
.style("text-anchor", "end")
.text(function(d) { return d; })
svg.append("g")
.attr("class", "axis axis--x")
.attr("transform", "translate(0," + height + ")")
.attr("fill", "#999")
.transition()
.delay(function(d, i) {
return i * 50
})
.duration(500)
.call(xAxis);
svg.append("g")
.attr("class", "axis axis--y")
.attr("transform", "translate(" + width + ",0)")
.attr("fill", "#999")
.transition()
.delay(function(d, i) {
return i * 50
})
.duration(500)
.call(yAxis);
});
}
})()
If you wish to learn how to write D3.js scripts, go to https://d3js.org/.
We will now highlight the different parts of the JavaScript file:
(function() {
var chartId = '#status_codes_chart'
var margin = {top: 60, right: 50, bottom: 20, left: 20}
var width = parseInt(d3.select(chartId).style('width')) - margin.left -
margin.right
var height = parseInt(d3.select(chartId).style('height')) - margin.top -
margin.bottom
//var width = 600;
//var height = 300;
...
})()
This is the entry point of the script. Here you are declaring the div ID status_codes_chart that matched the one in the index.html file. The margins are also being defined here. The width and height can either be dynamically assigned (first set) or statically assigned (second set). If dynamically assigned, they will either fall back to the width and height defined in the CSS, or they will adjust based on the window size. Feel free to tweak this later on and see how the chart changes:
var x = d3.scale.ordinal()
.rangeRoundBands([0, width]);
var y = d3.scale.linear()
.rangeRound([height, 0]);
var z = d3.scale.category10();
var xAxis = d3.svg.axis()
.scale(x)
.orient("bottom")
.tickFormat(d3.time.format("%I:%M"))
var yAxis = d3.svg.axis()
.scale(y)
.orient("right")
This block of code consists of D3 elements that are being declared. Suffice to say that these are required to create the SVG screenshot needed by the chart:
var svg = d3.select(chartId).append("svg")
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," +
margin.top + ")");
var categories = ["200", "301", "302", "404", "500"]
drawChart()
The previous block of code is D3 creating the SVG object. Notice that d3.select(chartId) is actually equal to d3.select("#status_codes_chart"). This is telling the D3 script to create the SVG chart in that div.
The var categories[] array defines what fields will be shown in the chart. These numbers are the result of the query that you ran against Splunk. If you wish to omit a particular status code from the resulting bar chart, then remove it from this list. The list requires a minimum of one entry.
drawChart() is a function call, which we will show you now.
The drawChart() function will contain all the instructions to generate the data based on the JSON object that is being called through the d3.json() method:
function drawChart() {
d3.json('/sdk_status_codes.json', function(error, json) {
if (error) return console.warn(error)
var data = []
for(var i = 0; i < json.rows.length; i++) {
data.push({
"date": new Date(json.rows[i][0]),
"200": parseInt(json.rows[i][1]),
"301": parseInt(json.rows[i][2]),
"302": parseInt(json.rows[i][3]),
"404": parseInt(json.rows[i][4]),
"500": parseInt(json.rows[i][5])
})
}
The d3.json call essentially makes a GET request against the URL of our local JSON file, which then passes the data back to the function.
The for loop here is not part of D3.js. This is just needed to format the incoming data to match the data needed by D3:
var layers = d3.layout.stack()(categories.map(function(c) {
return data.map(function(d) {
return {x: d.date, y: d[c]}
})
}))
This block of code creates the layers of the chart.
dashboard\public\status_codes_chart.css
The source for this is at https://github.com/ericksond/splunk-external-dashboard.
Create a new CSS file in c:\dashboard\public and name it status_codes_chart.css:
text {
font-family: monospace;
}
.axis text {
font: 10px sans-serif;
}
.axis line,
.axis path {
fill: none;
stroke: #ccc;
shape-rendering: crispEdges;
}
.axis--x path {
display: none;
}
#status_codes_chart {
-webkit-touch-callout: none;
-webkit-user-select: none;
-khtml-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
/* Trying to get SVG to act like a greedy block in all browsers */
display: block;
width:50%;
height:400px;
}
Rendering the chart
Now that all the components are in place, reload the page and check the result at http://localhost:8080.
If this page is not loading, make sure the http-server process is running. Refer to the earlier sections in this chapter for how to run the http-server:
Now you have a stacked bar chart that is being fed with data from Splunk. Go ahead and run the jobs.js cron job again using the following command. You need to do this in a separate command prompt window:
C:\> C:\dashboard> node jobs.js
Wait a few minutes and refresh the page. See it change before your eyes. There are many ways to make this dynamically update in real time. You can use JavaScript or browser-based refresh plugins. You may also extend the D3.js code and create a redrawChart() function that updates certain data entry points. The important thing is that you can now create public dashboards that will display data from Splunk without the need to log in to the application. You will be in total control of this dashboard as well.
Rendering the chart
Now that all the components are in place, reload the page and check the result at http://localhost:8080.
If this page is not loading, make sure the http-server process is running. Refer to the earlier sections in this chapter for how to run the http-server:
Now you have a stacked bar chart that is being fed with data from Splunk. Go ahead and run the jobs.js cron job again using the following command. You need to do this in a separate command prompt window:
C:\> C:\dashboard> node jobs.js
Wait a few minutes and refresh the page. See it change before your eyes. There are many ways to make this dynamically update in real time. You can use JavaScript or browser-based refresh plugins. You may also extend the D3.js code and create a redrawChart() function that updates certain data entry points. The important thing is that you can now create public dashboards that will display data from Splunk without the need to log in to the application. You will be in total control of this dashboard as well.
Summary
In this chapter, we showed you how to use the Splunk SDK to safely extract data from Splunk using JavaScript without the risk of exposing authentication credentials to viewers. You have also learned how to use Node.js and the npm. Additionally, you wrote a cron script that pulls data from Splunk and writes it into a local JSON file. And you used the http-server module to initialize a lightweight web server. Finally, you created the web application using D3.js to display a stacked bar chart using Splunk data. In the next chapter, Chapter 8, HTTP Event Collector, you will go on to learn about this event collector and how it can be used for many business purposes.