Big data is large amount of data that does not fit in the memory of a single machine. In other words, the size of data itself becomes a part of the issue when studying it. Besides volume, two other major characteristics of big data are variety and velocity; these are the famous three Vs of big data. Velocity means data process rate or how fast the data is being processed. Variety denotes various data source types. Noises arise more frequently in big data source sets and affect the mining results, which require efficient data preprocessing algorithms.
As a result, distributed filesystems are used as tools for successful implementation of parallel algorithms on large amounts of data; it is a certainty that we will get even more data with each passing second. Data analytics and visualization techniques are the primary factors of the data mining tasks related to massive data. The characteristics of massive data appeal to many new data mining technique-related platforms, one of which is RHadoop. We'll be describing this in a later section.
Some data types that are important to big data are as follows:
The data from the camera video, which includes more metadata for analysis to expedite crime investigations, enhanced retail analysis, military intelligence, and so on.
The second data type is from embedded sensors, such as medical sensors, to monitor any potential outbreaks of virus.
The third data type is from entertainment, information freely published through social media by anyone.
The last data type is consumer images, aggregated from social medias, and tagging on these like images are important.
Here is a table illustrating the history of data size growth. It shows that information will be more than double every two years, changing the way researchers or companies manage and extract value through data mining techniques from data, revealing new data mining studies.
|
Year |
Data Sizes |
Comments |
|---|---|---|
|
N/A |
1 MB (Megabyte) = | |
|
N/A |
1 PB (Petabyte) = | |
|
1999 |
1 EB |
1 EB (Exabyte) = |
|
2007 |
281 EB |
The world produced about 281 Exabyte of unique information. |
|
2011 |
1.8 ZB |
1 ZB (Zetabyte)= |
|
Very soon |
1 YB(Yottabytes)= |
Efficiency, scalability, performance, optimization, and the ability to perform in real time are important issues for almost any algorithms, and it is the same for data mining. There are always necessary metrics or benchmark factors of data mining algorithms.
As the amount of data continues to grow, keeping data mining algorithms effective and scalable is necessary to effectively extract information from massive datasets in many data repositories or data streams.
The storage of data from a single machine to wide distribution, the huge size of many datasets, and the computational complexity of the data mining methods are all factors that drive the development of parallel and distributed data-intensive mining algorithms.