Introducing the example projects

Although the data is a little old, I think it can give an interesting insight into the web traffic for the NASA website. The data is from 1995 and contains two traces of two months of all HTTP requests to the web server at the Kennedy Space Centre in Florida. The log files are Squid proxy logs and provide details on the host making the request, timestamp, request being made, HTTP reply code, and bytes in the reply.

A download of the data can be found at the following location: http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html.

The example projects will help analyze the web traffic hitting the NASA website and provide visualization and insights into the site's usage. The data will allow us to start with basic visualizations within the SplunkWeb Framework.

If you have been working in development, even for a short period of time, I am sure you will have heard of Conway's Game of Life. Even though it's called a game, it's more of a simulation of biological cells, where we can watch the cells evolve to either live or fail. The cells are governed by a set of rules that determines if they live or die through each generation or step in the simulation:

The logs presented here are random, but will consist of the grid where the cells will live, a timestamp, and the cells that are present through each generation of the life cycle (https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life).

I have created a GitHub repository with a basic example of Conway's Game Of Life, but I have also produced logs for the script for 2 hours to give you some sample data that can be worked with through the examples. The sample Python script and log file can be found by going to the following link:  https://github.com/vincesesto/game_of_life_splunk.

From here, you can index the file called game_of_life.log. If you are using at least version 6 of Splunk, the logs will be indexed correctly with the events separated correctly for each date and timestamp. The sample log file will look similar to the following image:

The example data that we have will allow us to analyze the simulation of cells, and although the data is not very complex, we should hopefully provide some interesting visualizations and take our skills with the Splunk Web Framework further.

Yahoo! Finance provides an API that allows people to download historical stock market data directly to their environment. In our example, we will take a few different companies and download their historical data for the year 2015, displaying the date stamp, opening value for the day, highest value of the day, lowest value for the day, closing value, volume traded for the day, and adjusted close value of the stock. The sample data will be in CSV form and the API call will be similar to the following URL:  http://ichart.finance.yahoo.com/table.csv?s=YHOO&d=0&e=28&f=2016&g=d&a=3&b=12&c=2015&ignore=.csv.

The API call is pretty straightforward and the commands are listed here:

For more details on different company symbols and more explanations of the data that the API can provide, go to the Yahoo! Finance site at https://finance.yahoo.com/.

The data presented is an interesting sample is varied, allowing for interesting trend analysis. This is where we will take our skills further and start to use more of the advanced features of the Splunk Web Framework.