Book Image

Hands-On Big Data Modeling

By : James Lee, Tao Wei, Suresh Kumar Mukhiya
Book Image

Hands-On Big Data Modeling

By: James Lee, Tao Wei, Suresh Kumar Mukhiya

Overview of this book

Modeling and managing data is a central focus of all big data projects. In fact, a database is considered to be effective only if you have a logical and sophisticated data model. This book will help you develop practical skills in modeling your own big data projects and improve the performance of analytical queries for your specific business requirements. To start with, you’ll get a quick introduction to big data and understand the different data modeling and data management platforms for big data. Then you’ll work with structured and semi-structured data with the help of real-life examples. Once you’ve got to grips with the basics, you’ll use the SQL Developer Data Modeler to create your own data models containing different file types such as CSV, XML, and JSON. You’ll also learn to create graph data models and explore data modeling with streaming data using real-world datasets. By the end of this book, you’ll be able to design and develop efficient data models for varying data sizes easily and efficiently.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Big Data Modeling
James Lee | Tao Wei | Suresh Kumar Mukhiya
Nov, 2018 | 306 pages
Small book image
Practical Big Data Analytics
Nataraj Dasgupta
Jan, 2018 | 412 pages
Small book image
Simplifying Data Engineering and Analytics with Delta
Anindita Mahapatra
Jul, 2022 | 334 pages
Small book image
Hands-On Exploratory Data Analysis with Python
Usman Ahmed | Suresh Kumar Mukhiya
Mar, 2020 | 352 pages
Table of Contents (17 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Introduction to Big Data and Data Management
Introduction to Big Data and Data Management
The concept of big data
Sources and types of big data
Introduction to big data modeling
Introduction to managing big data
Importance and implications of big data modeling and management
Setting up big data modeling platforms
Summary
Further reading
2
Data Modeling and Management Platforms
Data Modeling and Management Platforms
Big data management
Big data management services
Big data management vendors
Big data storage and data models
Big data programming models
Getting started with Python and R
Summary
Further reading
3
Defining Data Models
Defining Data Models
Data model structures
Data operations
Data constraints
A unified approach to big data modeling and data management
Summary
Further reading
4
Categorizing Data Models
Categorizing Data Models
Levels of data modeling
Types of data model
Summary
Further reading
5
Structures of Data Models
Structures of Data Models
Semi-structured data models
VSM with Lucene
Graph-data models
Summary
Further reading
6
Modeling Structured Data
Modeling Structured Data
Getting started with structured data
Modeling structured data using Python
Summary
Further reading
7
Modeling with Unstructured Data
Modeling with Unstructured Data
Getting started with unstructured data
Tools for analyzing unstructured data
Summary
Further reading
8
Modeling with Streaming Data
Modeling with Streaming Data
Data stream and data model versus data format
Why is streaming data different?
Importance and implications of streaming data
Exploring streaming sensor data from the Twitter API
Summary
Further reading
9
Streaming Sensor Data
Streaming Sensor Data
Sensor data
Data lakes
Exploring streaming sensor data from a weather station
Summary
Further study
10
Concept and Approaches of Big-Data Management
Concept and Approaches of Big-Data Management
Non-DBMS-based approach to big data
DBMS-based approach to big data
Parallel and distributed DBMS
DBMS and MapReduce-style systems
Summary
Further reading
11
DBMS to BDMS
DBMS to BDMS
Characteristics of BDMS
Exploring data management with Redis
Aerospike
AsterixDB
Summary
Further reading
12
Modeling Bitcoin Data Points with Python
Modeling Bitcoin Data Points with Python
Introduction to Bitcoin data
Theory
Importing Bitcoin data into iPython
Preprocessing and model creation
Predicting Bitcoin price using Recurrent Neural Network
Summary
Further reading
13
Modeling Twitter Feeds Using Python
Modeling Twitter Feeds Using Python
Importing Twitter feed data
Modeling Twitter feeds
Summary
Further reading
14
Modeling Weather Data Points with Python
Modeling Weather Data Points with Python
Introduction to weather data
Importing data
Modeling with data
Summary
Further reading
15
Modeling IMDb Data Points with Python
Modeling IMDb Data Points with Python
Introduction to IMDb data
Theory
Modeling with the IMDb dataset
Summary
Further reading
16
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Setting up big data modeling platforms

In this section, we are going to set up Cloudera VM on both Windows and macOS. We are going to use this VM for most of the exercises in this book.

Getting started on Windows

We are going to install Cloudera virtual machine (VM) in our system to get started with big data modeling. Follow these instructions to download and install the Cloudera Quickstart VM with VirtualBox on Windows:

  1. Download the software from https://www.virtualbox.org/wiki/Downloads. Once the download is complete, install the downloaded VirtualBox software on your computer.
  2. Download the Cloudera VM from https://downloads.cloudera.com/demo_vm/virtualbox/cloudera-quickstart-vm-5.4.2-0-virtualbox.zip. The VM is over 4 GB. It will take some time to download the software.
  3. Right-click Cloudera-quickstart-vm-5.4.2-0-virtualbox.zip and choose Extract All….
  4. Start the VirtualBox.
  5. Import the VM by going to File | Import Appliance...:
  1. Click the folder icon, select cloudera-quickstart-vm-5.4.2-0-virtualbox.ovf from the folder where you unzipped the VirtualBox VM, and click Open. The following screenshot is provided for assistance:
  2. Click Next to proceed, and then click Import, as shown in the following screenshot:

  1. The VM image will be imported. As this is a big file, it can take some time:
  1. When the import is finished, launch the Cloudera VM. cloudera-quickstart-vm-5.4.2-0 VM will appear on the left in the VirtualBox window. Select the machine and click the Start button to initiate the VM:
  1. It takes several minutes for Cloudera VM booting to start up initially. It takes a long time, since many Hadoop tools are loaded and started at this booting process:

  1. Once the booting process is finished, you will see the Cloudera VM desktop on the screen:

Getting started on macOS

Setting up the Cloudera VM on a Mac is very similar to setting it up on Windows. If you have macOS, we can set up the Cloudera Quickstart VM with VirtualBox on macOS. Perform the following steps:

  1. Go to https://www.virtualbox.org/wiki/Downloads and download the virtual box. Once downloaded, install VirtualBox in macOS.
  2. Download the Cloudera VM from https://downloads.cloudera.com/demo_vm/virtualbox/cloudera-quickstart-vm-5.4.2-0-virtualbox.zip. The VM is over 4 GB, so it will take some time to download.
  1. Unzip the Cloudera VM. You can do this by double-clicking on the cloudera-quickstart-vm-5.4.2-0-virtualbox.zip folder.
  2. Start the VirtualBox and begin importing. We can import the VM by going to File | Import Appliance.
  3. Click the Folder icon, select cloudera-quickstart-vm-5.4.2-0-virtualbox.ovf from the folder where you unzipped the VirtualBox VM, and click Open:
  1. Click Continue to proceed:
  1. Click Import:

  1. The VM image will be imported. As this is a big file, it can take some time:

  1. When the import is finished, launch the Cloudera VM. cloudera-quickstart-vm-5.4.2-0 VM will appear on the left in the VirtualBox window. Select the machine and click the Start button to initiate the VM:

  1. It takes several minutes for Cloudera VM booting to start up initially. It takes a long time, since many Hadoop tools are loaded and started during this booting process:

  1. Once the booting process is finished, you will see the Cloudera VM desktop on the screen:
Previous Section
End of Section 7
Next Section