While most of the data you will index with Splunk will be collected in real time, there might be instances where you have a set of data that you would like to put into Splunk, either to backfill some missing or incomplete data, or just to take advantage of its searching and reporting tools.
This recipe will show you how to perform one-time bulk loads of data from files located on the Splunk server. We will also use this recipe to load the data samples that will be used throughout subsequent chapters as we build our Operational Intelligence app in Splunk.
There are two files that make up our sample data. The first is access_log, which represents data from our web layer and is modeled on an Apache web server. The second file is app_log, which represents data from our application layer and is modeled on the log4j application log data.
To step through this recipe, you will need a running Splunk server and should have a copy of the sample data generation app (OpsDataGen.spl) for this book.
Follow the given steps to load the sample data generator on your system:
From the home launcher, select the Apps menu in the top-left corner and click on Manage Apps.
Select Install App from file.
Select the location of the
OpsDataGen.splfile on your computer, and then click on the Upload button to install the application.
After installation, a message should appear in a blue bar at the top of the screen, letting you know that the app has installed successfully. You should also now see the OpsDataGen app in the list of apps.
By default, the app installs with the data-generation scripts disabled. In order to generate data, you will need to enable either a Windows or Linux script, depending on your Splunk operating system. To enable the script, select the Settings menu from the top-right corner of the screen, and then select Data inputs.
From the Data inputs screen that follows, select Scripts.
On the Scripts screen, locate the OpsDataGen script for your operating system and click on Enable.
For Linux, it will be
$SPLUNK_HOME/etc/apps/OpsDataGen/bin/AppGen.pathFor Windows, it will be
$SPLUNK_HOME\etc\apps\OpsDataGen\bin\AppGen-win.path
The following screenshot displays both the Windows and Linux inputs that are available after installing the OpsDataGen app. It also displays where to click to enable the correct one based on the operating system Splunk is installed on.
Select the Settings menu from the top-right corner of the screen, select Data inputs, and then select Files & directories.
On the Files & directories screen, locate the two OpsDataGen inputs for your operating system and for each click on Enable.
For Linux, it will be:
$SPLUNK_HOME/etc/apps/OpsDataGen/data/access_log$SPLUNK_HOME/etc/apps/OpsDataGen/data/app_logFor Windows, it will be:
$SPLUNK_HOME\etc\apps\OpsDataGen\data\access_log$SPLUNK_HOME\etc\apps\OpsDataGen\data\app_log
The following screenshot displays both the Windows and Linux inputs that are available after installing the OpsDataGen app. It also displays where to click to enable the correct one based on the operating system Splunk is installed on.
The data will now be generated in real time. You can test this by navigating to the Splunk search screen and running the following search over an All time (real-time) time range:
index=main sourcetype=log4j OR sourcetype=access_combined
After a short while, you should see data from both source types flowing into Splunk, and the data generation is now working as displayed in the following screenshot:
In this case, you installed a Splunk application that leverages a scripted input. The script we wrote generates data for two source types. The access_combined source type contains sample web access logs, and the log4j source type contains application logs. These data sources will be used throughout the recipes in the book. Applications will also be discussed in more detail later on.