Every time you execute a search, always be aware that you are running a query against a set of data that is bound by date and time. The time-range picker is on the right side of the search bar. Splunk comes with predetermined time modifiers, as seen in the following screenshot. You can also use the time-range picker to set up a custom date/time range or other advanced ranges (https://docs.splunk.com/Splexicon:Timerangepicker):

There are two types of time modifier: real-time and relative. In the preceding screenshot, the predetermined real-time modifiers are in the leftmost column, and the relative time modifiers are in the middle column.

Real-time modifiers mean that Splunk will run an ongoing, real-time search based on the specified time. For example, a real-time search that is in a 5-minute window will continuously display data within the last five minutes. If new data comes in, it will push out the oldest event within the time frame.

Relative time modifiers are just that; they collect data based on relative time measures, and will find data within the specified timeframe.

What you do not see when you are using the time modifier drop-down is that in the background, Splunk is defining the earliest time and the latest time in specific variables.

The last 15 minutes preset, for example, is equivalent to this SPL modifier:

SPL> earliest=-15m latest=now

The presets built into Splunk automatically insert the latest=now modifier. Run this search command in your Destinations app Search bar:

SPL> index=main earliest=-30m latest=now | timechart count span=5m

Notice that even if you have not changed the time modifier selected in the drop-down menu (which will not change unless you use it), the data will show that your earliest event is 30 minutes ago and your last data is recent. In other words, what you put in the search bar overrides the time modifier drop-down menu.

You can use a number of alternative ways to identify each of the time units; the most commonly supported time units listed by Splunk are:

Splunk is currently one of the best enterprise search engines, that is, a search engine that can serve the needs of any size organization currently on the market. Using a search command, you can filter your results using key phrases just the way you would with a Google search. Here are some examples for you to try out:

SPL> index=main /booking/confirmation

The preceding filters search results and only shows those with /booking/confirmation in the _raw data.

You may also add further filters by adding another phrase. It is very important to note, however, that by default, Splunk will assume that your phrases are logically chained based on an AND operator. For example:

SPL> index=main /booking 200

The preceding line of code is equivalent to the following:

SPL> index=main /booking AND 200

Similarly, you can use the OR operator to find data based on multiple filters. The following command will return all events with /booking or /destinations in the text. It is important to remember that an OR operator will always give you at least as many (or more) events than an AND operator, and AND is the default operator:

SPL> index=main /booking OR /destinations

Like any mathematical operation, you may also use parentheses to group conditions:

SPL> index=main (/booking OR /destinations) AND 200

If you have a phrase that has white space in it, it is best practice to enclose it with quotation marks, as seen in the following example:

SPL> index=main "iPhone OS"

You may also filter search results using fields. Fields are case-sensitive and a search using a specified field is generally faster than a full text search. Filtering using fields will only work if there is a defined field. In Chapter 2, Bringing in Data, you extracted new fields from the eventgen data source. Let's use that now to filter search results using custom fields:

SPL> index=main method=GET url="/home"

You can use this concept to combine data even if it comes from different indexes. Here is a query that shows the concepts we have discussed so far. Although the following query will result in a data set that will not make any sense, we include it to show how to combine data from different indexes:

SPL> (index=wineventlogs OR index=main) LogName=Application OR /booking

The most common use of the stats command is to get a count of the total number of events that are the product of a search. To see how this works, run the following search query. Notice that the pipe that precedes the stats command filters the data that will be included in the final count:

SPL> index=main earliest=-30m latest=now | stats count

The preceding query will result in a single number that represents the total of all events within the given time modifier. Change the time modifier and the number should be reduced:

SPL> index=main earliest=-15m latest=now | stats count

You may be wondering where the count came from. The true format of a stats command is stats function(X). This asks the system to return the result of the function based on the field X. When the function count is used without parentheses, Splunk assumes that you are looking for the count of all events in the given search.

The stats command becomes a very powerful tool especially when you need to group counts by fields. Here is an example:

SPL> index=main  | stats count by method

This will result in two rows of data that will show the counts of the GET and the POST methods, as shown in the following screenshot. These are two methods that are used in HTTP (website communication rules for client and server) to ask for information (GET) and submit data (POST):

You can also use the avg(X) function to get the average value of all the events based on URLs. Here is an example that you can use:

SPL> index=main | stats count by url | stats avg(count)

Some of the widely used stats functions are:

A quick way to get a summarized table based on fields is by using the top and rare commands. Run this search command:

SPL> index=main | top url

Notice that the result automatically grouped the URLs by count, calculated the percentage of each row against the whole data set, and sorted them by count in descending order. You can see a sample result in the following screenshot:

You may further tweak this search command by adding command options such as limit and showperc. Say, for example, you only want to see the top five URLs but you do not want to see the percent column. This is the command to achieve that:

SPL> index=main | top url limit=5 showperc=false

Now try the same commands, but use rare instead of top. The term rare will find those events that are the most unlikely ones. This can be a useful qualifier to use for determining outliers or unusual cases that may be due to data entry errors.

The chart command is an aggregation command that provides output in tabular or chartable format. It is a very important command that is used for many different types of visualization. Notice that if you run the following search query, it is identical to the output of the stats command:

SPL> index=main | chart count by method

For all basic purposes, you can use stats and chart interchangeably. However, there will be differences in how stats and chart group data together. It will be up to you to determine which one is your intended result. To show the differences, here are some examples:

SPL> index=main | stats count by method url

SPL> index=main | chart count by method url

The timechart command, on the other hand, creates a time series chart with statistical aggregation of the indicated fields. This command is widely used when creating different types of chart. The most common use of timechart is for examining the trends of metrics over time for visualizations including line charts, column charts, bar charts, and area charts, among others:

SPL> index=main earliest=-4h latest=now | timechart span=15m count by url 

An important option that is part of the timechart command is span. The span essentially determines how it will bucket the data based on time slices.  span=15m means it will aggregate the data and slice it every 15 minutes.

The statistical result of the command looks like this:

Although admittedly the preceding data looks dull, this very same information, when viewed in the Visualizations tab, looks much more interesting, as seen in the following screenshot. There will be more on creating dashboard panels and dashboards in Chapter 6, Panes of Glass:

The eval command is perhaps the most advanced and powerful command in SPL. It allows you to store the resulting value of the eval operation in a field. A myriad of functions available today can be used with eval. Let us try some of the simpler and more common ones.

The simplest type of eval command performs a simple calculation and stores it in the newly created field. For example, if you want to create the new_salary field, which adds together old_salary plus a field named raise, it would look like this (but don't try this, as there are no such fields in our data):

SPL> eval new_salary = old_salary + raise

There are also countless functions that can be used effectively with eval. Later we discuss some of them:

SPL> round(X, Y)

Run the search command below, then modify it to include the eval function round(X, Y). Watch how the percent column values were transformed as they are rounded to the nearest integer with two decimal values:

SPL> index=main | top url 
 
     index=main | top url | eval percent=round(percent, 2) 
 
     upper(X)

Use this function to transform the URL strings into uppercase:

SPL> index=main | top url 
 
     index=main | top url | eval url=upper(url) 
 
     case(X, "Y", ...)

The case function is especially useful when transforming data based on a Boolean condition. If X is true, then assign to the variable the string Y. Here is one example:

SPL> index=main | top url showperc=false  
     | eval Tag=case(url=="/home", "Home", url="/auth", "Auth")

The resulting table shows that a new column called Tag has been created and all instances of /home have been marked as Home and all instances of /auth have been marked as Auth:

The rex or regular expression command is extremely useful when you need to extract a field during search time that has not already been extracted automatically. The rex command even works in multi-line events. The following sample command will get all the versions of the Chrome browser that are defined in the highlighted User Agent string part of the following raw data. Let's say this is your raw data, and you need to get the highlighted value:

016-07-21 23:58:50:227303,96.32.0.0,GET,/destination/LAX/details,-,80, 
-,10.2.1.33,Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) 
AppleWebKit/537.36 (KHTML; like Gecko) Chrome/29.0.1547.76 
Safari/537.36,500,0,0,823,3053 

You can use this search command to get it:

SPL> index=main | rex field=http_user_agent 
     "Chrome/(?<Chrome_Version>.+?)?Safari" | top Chrome_Version

The rex command extracted a field called Chrome_Version during the search and made it available for all succeeding commands. The results are shown in the following screenshot:

In this chapter, we introduced you to the SPL. You have learned that the search pipeline is crucial in the transformation of data as it is piped between search commands and eventually to the final results table. We also introduced you to time modifiers and how to filter search results. Lastly, you were introduced to multiple search commands that are commonly used. In Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches, we will go on to use our search processing skills to create useful reports and learn about developing alerts that will increase organizational efficiency and prevent errors. We will also learn more about how to best optimize our searches.