Generally, the definition of Big Data is data in large sizes that go beyond the ability of commonly used software tools to collect, manage, and process within a tolerable elapsed time. More formally, the definition of Big Data should go beyond the size of the data to include other properties. In this recipe, we will outline the properties that define Big Data in a formal way.
Ideally, data has the following three important properties: volume, velocity, and variety. In this book, we treat the value property of Big Data as the fourth important property. And, the value property also explains the reason why the Big Data problem exists.
Defining a Big Data problem involves the following steps:
Estimate the volume of data. The volume should not only include the current data volume, for example in gigabytes or terabytes, but also should include the expected volume in the future.
There are two types of data in the real world: static and nonstatic data. The volume of static data, for example national census data and human genomic data, will not change over time. While for nonstatic data, such as streaming log data and social network streaming data, the volume increases over time.
Estimate the velocity of data. The velocity estimate should include how much data can be generated within a certain amount of time, for example during a day. For static data, the velocity is zero.
The velocity property of Big Data defines the speed that data can be generated. This property will not only affect the volume of data, but also determines how fast a data processing system should handle the data.
Identify the data variety. In other words, the data variety means the different sources of data, such as web click data, social network data, data in relational databases, and so on.
Variety means that data differs syntactically or semantically. The difference requires specifically designed modules for each data variety to be integrated into the Big Data platform. For example, a web crawler is needed for getting data from the Web, and a data translation module is needed to transfer data from relational databases to a nonrelational Big Data platform.
Define the expected value of data.
The value property of Big Data defines what we can potentially derive from and how we can use Big Data. For example, frequent item sets can be mined from online click-through data for better marketing and more efficient deployment of advertisements.
A Big Data platform can be described with the IPO (http://en.wikipedia.org/wiki/IPO_Model) model, which includes three components: input, process, and output. For a Big Data problem, the volume, velocity, and variety properties together define the input of the system, and the value property defines the output.