We saw in the previous chapters how to work with data using pandas and Matplotlib for visualization and the other tools in the Python data science stack. So far, the datasets that we have used have been relatively small and with a relatively simple structure. Real-life datasets can be orders of magnitude larger than can fit into the memory of a single machine, the time to process these datasets can be long, and the usual software tools may not be up to the task. This is the usual definition of what big data is: an amount of data that does not fit into memory or cannot be processed or analyzed in a reasonable amount of time by common software methods. What is big data for some may not be big data for others, and this definition can vary depending on who you ask.
Big Data is also associated with the 3 V’s (later extended to 4 V’s):
Volume: Big data, as the name suggests, is usually associated with very large volumes of data. What is large depends on the context: for one system,...