While most of the data you will index with Splunk will be collected in real time, there might be instances where you have a set of data that you would like to put into Splunk, either to backfill some missing or incomplete data, or just to take advantage of its searching and reporting tools.
This recipe will show you how to perform one-time bulk loads of data from files located on the Splunk server. We will also use this recipe to load the data samples that will be used throughout the subsequent chapters as we build our operational intelligence app in Splunk.
There are two files that make up our sample data. The first is access_log, which represents the data from our web layer and is modeled on an Apache web server. The second file is app_log, which represents the data from our application layer and is modeled on log4j log data from our custom middleware application.
To step through this recipe, you will need a running Splunk server and you should have a copy of the sample data generation app (OpsDataGen.spl) for this book.
Follow the given steps to load the sample data generator on your system:
Log in to your Splunk server using your credentials.
From the Apps menu in the upper left-hand corner of the home screen, click on the gear icon.
The Apps settings page will load. Then click on the Install app from file button.
Select the location of the
OpsDataGen.splfile on your computer, and then, click on the Upload button to install the application.
After installation, a message should appear in a blue bar at the top of the screen, letting you know that the app has installed successfully. You should also now see the
OpsDataGenapp in the list of apps.
By default, the app installs with the data-generation scripts disabled. In order to generate data, you will need to enable either a Windows or Linux script, depending on your Splunk operating system. To enable the script, select the Settings menu from the top right-hand side of the screen, and then, select Data inputs.
From the Data inputs screen that follows, select Scripts.
On the Scripts screen, locate the OpsDataGen script for your operating system and click on Enable.
For Linux, it will be
$SPLUNK_HOME/etc/apps/OpsDataGen/bin/AppGen.pathFor Windows, it will be
$SPLUNK_HOME\etc\apps\OpsDataGen\bin\AppGen-win.path
The following screenshot displays both the Windows and Linux inputs that are available after installing the OpsDataGen app. It also displays where to click to enable the correct one based on the operating system Splunk is installed on.
Select the Settings menu from the top right-hand side of the screen, select Data inputs, and then select Files & directories.
On the Files & directories screen, locate the two OpsDataGen inputs for your operating system and for each click on Enable.
For Linux, it will be
$SPLUNK_HOME/etc/apps/OpsDataGen/data/access_logand$SPLUNK_HOME/etc/apps/OpsDataGen/data/app_logFor Windows, it will be
$SPLUNK_HOME\etc\apps\OpsDataGen\data\access_logand$SPLUNK_HOME\etc\apps\OpsDataGen\data\app_log
The following screenshot displays both the Windows and Linux inputs that are available after installing the OpsDataGen app. It also displays where to click to enable the correct one based on the operating system Splunk is installed on.
The data will now be generated in real time. You can test this by navigating to the Splunk search screen and running the following search over an All time (real-time) time range:
index=main sourcetype=log4j OR sourcetype=access_combined
After a short while, you should see data from both the source types flowing into Splunk. The data generation is now working, as displayed in the following screenshot:
In this case, you installed a Splunk application that leverages a scripted input. The script we wrote generates data for two source types. The access_combined source type contains sample web access logs, and the log4j source type contains application logs. These data sources will be used throughout the recipes in the book. Applications will also be discussed in more detail later on.