Book Image

Splunk: Enterprise Operational Intelligence Delivered

By : Derek Mock, Betsy Page Sigman, Paul R. Johnson, Erickson Delgado, Josh Diakun, Ashish Kumar Tulsiram Yadav
Book Image

Splunk: Enterprise Operational Intelligence Delivered

By: Derek Mock, Betsy Page Sigman, Paul R. Johnson, Erickson Delgado, Josh Diakun, Ashish Kumar Tulsiram Yadav

Overview of this book

Splunk is an extremely powerful tool for searching, exploring, and visualizing data of all types. Splunk is becoming increasingly popular, as more and more businesses, both large and small, discover its ease and usefulness. Analysts, managers, students, and others can quickly learn how to use the data from their systems, networks, web traffic, and social media to make attractive and informative reports. This course will teach everything right from installing and configuring Splunk. The first module is for anyone who wants to manage data with Splunk. You’ll start with very basics of Splunk— installing Splunk— before then moving on to searching machine data with Splunk. You will gather data from different sources, isolate them by indexes, classify them into source types, and tag them with the essential fields. With more than 70 recipes on hand in the second module that demonstrate all of Splunk’s features, not only will you find quick solutions to common problems, but you’ll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization. Dive deep into Splunk to find the most efficient solution to your data problems in the third module. Create the robust Splunk solutions you need to make informed decisions in big data machine analytics. From visualizations to enterprise integration, this well-organized high level guide has everything you need for Splunk mastery. This learning path combines some of the best that Packt has to offer into one complete, curated package. It includes content from the following Packt products: • Splunk Essentials - Second Edition • Splunk Operational Intelligence Cookbook - Second Edition • Advanced Splunk
Table of Contents (6 chapters)

Chapter 3. Search Processing Language

In the previous chapters, you learned how to collect and index data to prepare it for searching, and how to do a simple search. In this chapter, we will cover more about how to use search and other commands to analyze our data. In a nutshell, we will cover the following topics:

  • Anatomy of a search
  • Search pipeline
  • Time modifiers
  • Filtering searches
  • Search command-stats
  • Search command-top/rare
  • Search commands-chart and timechart
  • Search command-eval
  • Search command-rex

Anatomy of a search

Search Processing Language (SPL), a special-purpose processing language, was developed to enable fast searching on machine-generated data that has been indexed into Splunk. The language was originally set up to be based on Unix piping and Standard Query Language (SQL). We'll explain piping later. SQL is the language most widely used to query databases. The Search Processing Language (SPL, as opposed to SQL) is a library of all search processing commands and their functions, arguments, and clauses. With a search command you can group different events, filter data based on a constraint, extract fields using regular expressions, perform statistical calculations, and other tasks.

Let us dissect a search query so you can understand exactly how it works. This will also help you to understand what pipes are. As you will see, a pipe basically takes the data that has come from an earlier step and after it has been acted on, filtered, or extracted, sends it on to the next step in processing.

We'll use the Destinations app here to show you a simple example:

  1. Go to the Splunk home page.
  2. Click on your Destinations app.
  3. In your Destinations app's Search page, type in the following:
      SPL> index=_internal sourcetype=splunk* | top limit=5 name 
           | sort - name

The following diagram will allow you to visualize the data as it goes through one delimiting pipe (|) and another; in this case, from the internal index of Splunk, to limiting it to the top five names, to sorting by name, which then gets sent to the Final Results table. We will go through this step by step, as shown in the following screenshot:

Anatomy of a search

Search pipeline

The arrows in the preceding visualization, each of which represents a pipe, mean that the resulting data from the previous command will be used against the next command in the series. To fully understand this command, let's go through the pipes one by one. Type the following commands in succession into the search window, but pause in every one of them and observe how the data changes.

The following command will display the raw events:

SPL> index=_internal sourcetype=splunk*

This command used the raw events in the previous results table, keyed them on the name field, tabulated it by the number of events that the particular field has, then limited the result to five:

SPL> index=_internal sourcetype=splunk* | top limit=5 name  

Finally, the results table of the | top command is passed on to another command | sort for sorting transformation.

SPL> index=_internal sourcetype=splunk* | top limit=5 name | sort - name

This chaining of commands is called the search pipeline.

Time modifiers

Every time you execute a search, always be aware that you are running a query against a set of data that is bound by date and time. The time-range picker is on the right side of the search bar. Splunk comes with predetermined time modifiers, as seen in the following screenshot. You can also use the time-range picker to set up a custom date/time range or other advanced ranges (

Time modifiers

There are two types of time modifier: real-time and relative. In the preceding screenshot, the predetermined real-time modifiers are in the leftmost column, and the relative time modifiers are in the middle column.

Real-time modifiers mean that Splunk will run an ongoing, real-time search based on the specified time. For example, a real-time search that is in a 5-minute window will continuously display data within the last five minutes. If new data comes in, it will push out the oldest event within the time frame.


Real-time searches are resource-intensive. Use them sparingly. Other ways to efficiently show real-time data will be shown in later chapters.

Relative time modifiers are just that; they collect data based on relative time measures, and will find data within the specified timeframe.

What you do not see when you are using the time modifier drop-down is that in the background, Splunk is defining the earliest time and the latest time in specific variables.

The last 15 minutes preset, for example, is equivalent to this SPL modifier:

SPL> earliest=-15m latest=now

The presets built into Splunk automatically insert the latest=now modifier. Run this search command in your Destinations app Search bar:

SPL> index=main earliest=-30m latest=now | timechart count span=5m

Notice that even if you have not changed the time modifier selected in the drop-down menu (which will not change unless you use it), the data will show that your earliest event is 30 minutes ago and your last data is recent. In other words, what you put in the search bar overrides the time modifier drop-down menu.

You can use a number of alternative ways to identify each of the time units; the most commonly supported time units listed by Splunk are:

  • Second: s, sec, secs, second, seconds
  • Minute: m, min, minute, minute, minutes
  • Hour: h, hr, hrs, hour, hours
  • Day: d, day, days
  • Week: w, week, weeks
  • Month: mon, month, months
  • Quarter: q, qtr, qtrs, quarter, quarters
  • Year: y, yr, yrs, year, years

Filtering search results

Splunk is currently one of the best enterprise search engines, that is, a search engine that can serve the needs of any size organization currently on the market. Using a search command, you can filter your results using key phrases just the way you would with a Google search. Here are some examples for you to try out:

SPL> index=main /booking/confirmation

The preceding filters search results and only shows those with /booking/confirmation in the _raw data.

You may also add further filters by adding another phrase. It is very important to note, however, that by default, Splunk will assume that your phrases are logically chained based on an AND operator. For example:

SPL> index=main /booking 200

The preceding line of code is equivalent to the following:

SPL> index=main /booking AND 200

Similarly, you can use the OR operator to find data based on multiple filters. The following command will return all events with /booking or /destinations in the text. It is important to remember that an OR operator will always give you at least as many (or more) events than an AND operator, and AND is the default operator:

SPL> index=main /booking OR /destinations

Like any mathematical operation, you may also use parentheses to group conditions:

SPL> index=main (/booking OR /destinations) AND 200

If you have a phrase that has white space in it, it is best practice to enclose it with quotation marks, as seen in the following example:

SPL> index=main "iPhone OS"

You may also filter search results using fields. Fields are case-sensitive and a search using a specified field is generally faster than a full text search. Filtering using fields will only work if there is a defined field. In Chapter 2, Bringing in Data, you extracted new fields from the eventgen data source. Let's use that now to filter search results using custom fields:

SPL> index=main method=GET url="/home"

You can use this concept to combine data even if it comes from different indexes. Here is a query that shows the concepts we have discussed so far. Although the following query will result in a data set that will not make any sense, we include it to show how to combine data from different indexes:

SPL> (index=wineventlogs OR index=main) LogName=Application OR /booking

Search command - stats

The most common use of the stats command is to get a count of the total number of events that are the product of a search. To see how this works, run the following search query. Notice that the pipe that precedes the stats command filters the data that will be included in the final count:

SPL> index=main earliest=-30m latest=now | stats count

The preceding query will result in a single number that represents the total of all events within the given time modifier. Change the time modifier and the number should be reduced:

SPL> index=main earliest=-15m latest=now | stats count

You may be wondering where the count came from. The true format of a stats command is stats function(X). This asks the system to return the result of the function based on the field X. When the function count is used without parentheses, Splunk assumes that you are looking for the count of all events in the given search.

The stats command becomes a very powerful tool especially when you need to group counts by fields. Here is an example:

SPL> index=main  | stats count by method

This will result in two rows of data that will show the counts of the GET and the POST methods, as shown in the following screenshot. These are two methods that are used in HTTP (website communication rules for client and server) to ask for information (GET) and submit data (POST):

Search command - stats

You can also use the avg(X) function to get the average value of all the events based on URLs. Here is an example that you can use:

SPL> index=main | stats count by url | stats avg(count)

Some of the widely used stats functions are:

  • avg(X): Returns the average of the values of the field X
  • dc(X): Returns the count of distinct values of the field X
  • max(X): Returns the maximum value of the field X
  • min(X): Returns the minimum value of the field X
  • perc<X>(Y): Returns the Xth percentile of the field X, for example perc95(X)
  • sum(X): Returns the sum of the values of the field X


To learn more about the other stats functions, go to

Search command - top/rare

A quick way to get a summarized table based on fields is by using the top and rare commands. Run this search command:

SPL> index=main | top url

Notice that the result automatically grouped the URLs by count, calculated the percentage of each row against the whole data set, and sorted them by count in descending order. You can see a sample result in the following screenshot:

Search command - top/rare

You may further tweak this search command by adding command options such as limit and showperc. Say, for example, you only want to see the top five URLs but you do not want to see the percent column. This is the command to achieve that:

SPL> index=main | top url limit=5 showperc=false

Now try the same commands, but use rare instead of top. The term rare will find those events that are the most unlikely ones. This can be a useful qualifier to use for determining outliers or unusual cases that may be due to data entry errors.

Search commands - chart and timechart

The chart command is an aggregation command that provides output in tabular or chartable format. It is a very important command that is used for many different types of visualization. Notice that if you run the following search query, it is identical to the output of the stats command:

SPL> index=main | chart count by method

For all basic purposes, you can use stats and chart interchangeably. However, there will be differences in how stats and chart group data together. It will be up to you to determine which one is your intended result. To show the differences, here are some examples:

SPL> index=main | stats count by method url
Search commands - chart and timechart
SPL> index=main | chart count by method url
Search commands - chart and timechart

The timechart command, on the other hand, creates a time series chart with statistical aggregation of the indicated fields. This command is widely used when creating different types of chart. The most common use of timechart is for examining the trends of metrics over time for visualizations including line charts, column charts, bar charts, and area charts, among others:

SPL> index=main earliest=-4h latest=now | timechart span=15m count by url 

An important option that is part of the timechart command is span. The span essentially determines how it will bucket the data based on time slices.  span=15m means it will aggregate the data and slice it every 15 minutes.

The statistical result of the command looks like this:

Search commands - chart and timechart

Although admittedly the preceding data looks dull, this very same information, when viewed in the Visualizations tab, looks much more interesting, as seen in the following screenshot. There will be more on creating dashboard panels and dashboards in Chapter 6, Panes of Glass:

Search commands - chart and timechart

Search command - eval

The eval command is perhaps the most advanced and powerful command in SPL. It allows you to store the resulting value of the eval operation in a field. A myriad of functions available today can be used with eval. Let us try some of the simpler and more common ones.

The simplest type of eval command performs a simple calculation and stores it in the newly created field. For example, if you want to create the new_salary field, which adds together old_salary plus a field named raise, it would look like this (but don't try this, as there are no such fields in our data):

SPL> eval new_salary = old_salary + raise

There are also countless functions that can be used effectively with eval. Later we discuss some of them:

SPL> round(X, Y)

Run the search command below, then modify it to include the eval function round(X, Y). Watch how the percent column values were transformed as they are rounded to the nearest integer with two decimal values:

SPL> index=main | top url 
     index=main | top url | eval percent=round(percent, 2) 

Use this function to transform the URL strings into uppercase:

SPL> index=main | top url 
     index=main | top url | eval url=upper(url) 
     case(X, "Y", ...)

The case function is especially useful when transforming data based on a Boolean condition. If X is true, then assign to the variable the string Y. Here is one example:

SPL> index=main | top url showperc=false  
     | eval Tag=case(url=="/home", "Home", url="/auth", "Auth")

The resulting table shows that a new column called Tag has been created and all instances of /home have been marked as Home and all instances of /auth have been marked as Auth:

Search command - eval

Search command - rex

The rex or regular expression command is extremely useful when you need to extract a field during search time that has not already been extracted automatically. The rex command even works in multi-line events. The following sample command will get all the versions of the Chrome browser that are defined in the highlighted User Agent string part of the following raw data. Let's say this is your raw data, and you need to get the highlighted value:

016-07-21 23:58:50:227303,,GET,/destination/LAX/details,-,80, 
-,,Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) 
AppleWebKit/537.36 (KHTML; like Gecko) Chrome/29.0.1547.76 

You can use this search command to get it:

SPL> index=main | rex field=http_user_agent 
     "Chrome/(?<Chrome_Version>.+?)?Safari" | top Chrome_Version

The rex command extracted a field called Chrome_Version during the search and made it available for all succeeding commands. The results are shown in the following screenshot:

Search command - rex


In this chapter, we introduced you to the SPL. You have learned that the search pipeline is crucial in the transformation of data as it is piped between search commands and eventually to the final results table. We also introduced you to time modifiers and how to filter search results. Lastly, you were introduced to multiple search commands that are commonly used. In Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches, we will go on to use our search processing skills to create useful reports and learn about developing alerts that will increase organizational efficiency and prevent errors. We will also learn more about how to best optimize our searches.