Book Image

Hands-On Big Data Modeling

By : James Lee, Tao Wei, Suresh Kumar Mukhiya
Book Image

Hands-On Big Data Modeling

By: James Lee, Tao Wei, Suresh Kumar Mukhiya

Overview of this book

Modeling and managing data is a central focus of all big data projects. In fact, a database is considered to be effective only if you have a logical and sophisticated data model. This book will help you develop practical skills in modeling your own big data projects and improve the performance of analytical queries for your specific business requirements. To start with, you’ll get a quick introduction to big data and understand the different data modeling and data management platforms for big data. Then you’ll work with structured and semi-structured data with the help of real-life examples. Once you’ve got to grips with the basics, you’ll use the SQL Developer Data Modeler to create your own data models containing different file types such as CSV, XML, and JSON. You’ll also learn to create graph data models and explore data modeling with streaming data using real-world datasets. By the end of this book, you’ll be able to design and develop efficient data models for varying data sizes easily and efficiently.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Big Data Modeling
James Lee | Tao Wei | Suresh Kumar Mukhiya
Nov, 2018 | 306 pages
Small book image
Practical Big Data Analytics
Nataraj Dasgupta
Jan, 2018 | 412 pages
Small book image
Simplifying Data Engineering and Analytics with Delta
Anindita Mahapatra
Jul, 2022 | 334 pages
Small book image
Hands-On Exploratory Data Analysis with Python
Usman Ahmed | Suresh Kumar Mukhiya
Mar, 2020 | 352 pages
Table of Contents (17 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Introduction to Big Data and Data Management
Introduction to Big Data and Data Management
The concept of big data
Sources and types of big data
Introduction to big data modeling
Introduction to managing big data
Importance and implications of big data modeling and management
Setting up big data modeling platforms
Summary
Further reading
2
Data Modeling and Management Platforms
Data Modeling and Management Platforms
Big data management
Big data management services
Big data management vendors
Big data storage and data models
Big data programming models
Getting started with Python and R
Summary
Further reading
3
Defining Data Models
Defining Data Models
Data model structures
Data operations
Data constraints
A unified approach to big data modeling and data management
Summary
Further reading
4
Categorizing Data Models
Categorizing Data Models
Levels of data modeling
Types of data model
Summary
Further reading
5
Structures of Data Models
Structures of Data Models
Semi-structured data models
VSM with Lucene
Graph-data models
Summary
Further reading
6
Modeling Structured Data
Modeling Structured Data
Getting started with structured data
Modeling structured data using Python
Summary
Further reading
7
Modeling with Unstructured Data
Modeling with Unstructured Data
Getting started with unstructured data
Tools for analyzing unstructured data
Summary
Further reading
8
Modeling with Streaming Data
Modeling with Streaming Data
Data stream and data model versus data format
Why is streaming data different?
Importance and implications of streaming data
Exploring streaming sensor data from the Twitter API
Summary
Further reading
9
Streaming Sensor Data
Streaming Sensor Data
Sensor data
Data lakes
Exploring streaming sensor data from a weather station
Summary
Further study
10
Concept and Approaches of Big-Data Management
Concept and Approaches of Big-Data Management
Non-DBMS-based approach to big data
DBMS-based approach to big data
Parallel and distributed DBMS
DBMS and MapReduce-style systems
Summary
Further reading
11
DBMS to BDMS
DBMS to BDMS
Characteristics of BDMS
Exploring data management with Redis
Aerospike
AsterixDB
Summary
Further reading
12
Modeling Bitcoin Data Points with Python
Modeling Bitcoin Data Points with Python
Introduction to Bitcoin data
Theory
Importing Bitcoin data into iPython
Preprocessing and model creation
Predicting Bitcoin price using Recurrent Neural Network
Summary
Further reading
13
Modeling Twitter Feeds Using Python
Modeling Twitter Feeds Using Python
Importing Twitter feed data
Modeling Twitter feeds
Summary
Further reading
14
Modeling Weather Data Points with Python
Modeling Weather Data Points with Python
Introduction to weather data
Importing data
Modeling with data
Summary
Further reading
15
Modeling IMDb Data Points with Python
Modeling IMDb Data Points with Python
Introduction to IMDb data
Theory
Modeling with the IMDb dataset
Summary
Further reading
16
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Modeling structured data using Python

Let's see what type of data we are going to analyze today using IPython. Download the CSV file from the Chapter 6 codebase from GitHub. You should have access to the data file in the CH06 folder. The CSV files contain the following entries:

id,
date,
price,
bedrooms,
bathrooms,
sqft_living,
sqft_lot,
floors,waterfront,
view,
condition,
grade,
sqft_above,
sqft_basement,
yr_built,
yr_renovated,
zipcode,
lat,
long,
sqft_living15,
sqft_lot1

First things first, we import our libraries and dataset. Then, using the head function, we check out the first few pieces of data to see how they look. After that, we use the describe function to see the percentiles and other key statistics. Let's start our IPython notebook by using the following command:

jupyter notebook

Now let's import the required libraries:

import numpy as np
import pandas as pd
import matplotlib...
Previous Section
End of Section 3
Next Section