We live in a digital era where data is generated everywhere, from smart connected devices to social media. In 2014, every second over 5,700 tweets were sent and 800 links were shared using Facebook, and the digital universe expanded by about 1.7 MB per minute for every person on earth (source: IDC 2014 report). This amount of data sharing and storing is unprecedented and is contributing to what is known as big data. In 2013, about 4.4 ZB were created and in 2020 the forecast is 44 ZB, which is 44 trillion GB (source: http://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm).
Figure 1.1: Data growth
IDC predicts that organizations that are able to analyze this big data and derive actionable insights will see an additional $430 billion in productivity benefits over their peers (source: IDC FutureScape: Worldwide Big Data and Analytics 2016 Predictions, https://www.cloudera.com/content/dam/www/static/documents/analyst-reports/idc-futurescape.pdf).
Let's look at some real use cases that have benefited from big data:
IT systems in all major banks are constantly monitoring fraudulent activities and alerting customers within milliseconds. These systems apply complex business rules and analyze historical data, geography, type of vendor, and other parameters based on the customer to get accurate results and protect millions of customers across the globe.
Commercial drones are transforming agriculture by analyzing real-time aerial images and identifying the problem areas. These drones are cheaper and more efficient than satellite imagery, as they fly under the clouds and can take images anytime. They identify irrigation issues related to water, pests, or fungal infections, which thereby increases the crop productivity and quality. These drones are equipped with technology to capture high quality images every second and transfer them to a cloud-hosted big data system for further processing (you can refer to http://www.technologyreview.com/featuredstory/526491/agricultural-drones/ for more information).
Almost all shopping websites are using recommendation engines to improve customer experience like Amazon, Netflix, and Pandaro. These engines are sophisticated systems that perform big data analysis on historical buying preferences of the customer, ratings from social media, and associations rules from other similar customer purchases. This is now feasible due to advances in big data storage, compute, and in-memory analytics making these systems more intelligent and effective. You can refer to this article for more information at http://www.sas.com/en_us/insights/articles/big-data/recommendation-systems.html.
This unprecedented growth of data has resulted in need for faster insights, quicker co-relations, and the need to democratize data and analysis. Let's next look at the current BI landscape, key features they provide, and also their limitations.