Functions | 50 Hours of Big Data, PySpark, AWS, Scala, and Scraping

Book Overview & Buying
Table Of Contents

50 Hours of Big Data, PySpark, AWS, Scala, and Scraping

By : AI Sciences

5 (1)

Buy this Video

50 Hours of Big Data, PySpark, AWS, Scala, and Scraping

5 (1)

By: AI Sciences

Buy this Video

Overview of this book

Part 1 is designed to reflect the most in-demand Scala skills. It provides an in-depth understanding of core Scala concepts. We will wrap up with a discussion on Map Reduce and ETL pipelines using Spark from AWS S3 to AWS RDS (includes six mini-projects and one Scala Spark project). Part 2 covers PySpark to perform data analysis. You will explore Spark RDDs, Dataframes, a bit of Spark SQL queries, transformations, and actions that can be performed on the data using Spark RDDs and dataframes, the ecosystem of Spark and Hadoop, and their underlying architecture. You will also learn how we can leverage AWS storage, databases, computations, and how Spark can communicate with different AWS services. Part 3 is all about data scraping and data mining. You will cover important concepts such as Internet Browser execution and communication with the server, synchronous and asynchronous, parsing data in response from the server, tools for data scraping, Python requests module, and more. In Part 4, you will be using MongoDB to develop an understanding of the NoSQL databases. You will explore the basic operations and explore the MongoDB query, project and update operators. We will wind up this section with two projects: Developing a CRUD-based application using Django and MongoDB and implementing an ETL pipeline using PySpark to dump the data in MongoDB. By the end of this course, you will be able to relate the concepts and practical aspects of learned technologies with real-world problems. All the resources of this course are available at https://github.com/PacktPublishing/50-Hours-of-Big-Data-PySpark-AWS-Scala-and-Scraping

Part 1 - Data Scraping and Data Mining for Beginners to Pro with Python

Why Data Scraping

Applications of Data Scraping

Introduction of Instructor

Introduction to Course, Scraping, Tools

Projects Overview for Data Scraping and Data Mining

Requests

Introduction to Python Requests

Hands-On with Requests

Extracting Quotes Manually

Quiz (Extracting Authors)

Solution (Extracting Authors)

Pagination

Quiz (Extracting Author and Quotes)

Solution 01 (Extracting Author and Quotes)

Solution 02 (Extracting Author and Quotes)

Ajax Requests

Ajax Requests for Cricinfo

Ajax Requests Pagination

Quiz (Extracting Top Stats from Cricinfo)

Solution 01 (Extracting Top Stats from Cricinfo)

Solution 02 (Extracting Top Stats from Cricinfo)

Beautiful Soup 4 (BS4)

Introduction to BS4

Quiz (Difference Between Requests and BS4)

Solution (Difference Between Requests and BS4)

Hands-On with BS4

Extracting Data from Tree

Extracting Quotes from the Website

Quiz (Extracting Author Names)

Solution (Extracting Author Names)

Attributes of Tags in BS4

Multi-Valued Attributes of Tags in BS4

Scraping Movie Names from IMDB

Quiz (Getting the Ratings, Year, and Name of the Movie)

Solution 01 (Getting the Ratings, Year, and Name of the Movie)

Solution 02 (Getting the Ratings, Year, and Name of the Movie)

Scraping Time, Genre, and Releasing Date from IMDB 01

Scraping Time, Genre, and Releasing Date from IMDB 02

Combining Two Requests Data for IMDB

Movies Recommender System (Creating Movie URL)

Movies Recommender System (Creating Director URL)

Movies Recommender System Using BS4 (Getting Top Four Movies)

Movies Recommender System Using BS4 (Merge All Requests Together)

CSS Selectors

Introduction to CSS Selectors

CSS Selectors Hands-On (Tags)

Quiz (Tags)

Solution (Tags)

CSS Selectors Hands-On (Descendants, ID, Class)

Quiz (Descendants)

Solution (Descendants)

Quiz (ID)

Solution (ID)

Solution (Class) Part 1

Solution (Class) Part 2

CSS Selectors Hands-On (Nested Tags, ID Tags, Class Tags)

Quiz (Class with Tag)

Solution (Class with Tag)

CSS Selectors Hands-On (Comma Separator, Universal Selectors

Quiz (Combining Two Selectors)

Solution (Combining Two Selectors)

CSS Selectors Hands-On (Sibling Notations and Direct Child)

Quiz (Adjacent Sibling)

Solution (Adjacent Sibling)

Quiz (General Sibling)

Solution (General Sibling)

CSS Selectors Hands-On (Child Selectors)

Quiz (First Child)

Solution (First Child)

Quiz (Only Child)

Solution (Only Child)

Quiz (Last Child)

Solution (Last Child)

CSS Selectors Hands-On (Negations, Attributes)

Quiz (Negation)

Solution (Negation)

CSS Selectors Hands-On (Attributes, Attributes Values)

Quiz (Attributes Values)

Solution (Attributes Values)

CSS Selectors Hands-On (Attributes Wild Cards Values)

Quiz (Attributes Wild Card)

Solution (Attributes Wild Card)

Scrapy

Introduction to Scrapy

Comparison of Scrapy and Requests

Scrapy at a Glance Documentation

Getting Started with Scrapy

Running Documentation Spider 1

Running Documentation Spider 2

Writing Spider from the Scratch

Understanding the Response (URL, Status)

Understanding the Response (Headers)

Understanding the Response (Values in Headers)

Understanding the Response (Body)

Understanding the Response (Request)

Understanding the Response (Meta)

Understanding the Response (flags, certificate, ip_address, copy)

Understanding the Response (replace, urljoin, follow, follow_all)

Response CSS and Scrapy Shell

Extracting quotes with Scrapy

Understanding Nested Selectors

Extracting the Author and Quotes

Checking for Next Page

Checking for Next Page in Spider

Checking for Next Page URL

Scraping Quotes from Next Pages

Exporting Extracted Data

Quiz (Get the Tags)

Solution (Get the Tags)

Next Website

CSS Selectors for Movie Names and URLs

Combined CSS Selectors for Movie Names and URLs

Sent Request to the Film Info Page

Merge Data from Two Callbacks

Extracting Movie Duration and Genres

Exporting the Extracted Data

Quiz (Extracting the Year)

Solution (Extracting the Year)

Getting Director Name and URL

Getting Top Four Movies of Directors

Extracting Data Anomaly (dont_filter Flag)

Scrapy Project

Hugo Boss Website for Scraping

Understanding Site Structure

Writing CSS Selectors for Listings

Listings in Scrapy Shell

Sending Request to Listings URLs

Extracting Products URL from the Listings

Sending Requests to Products of the Listings

Writing CSS to Get the Product Info

Getting the Bigger Images of the Product

Checking Next Page URL

Adding Pagination to Spider and Running It

Output of the Spider

Selenium

Introduction to Selenium

Getting Started with Selenium

Configuring the Webdriver

Extracting Quotes with Selenium

Extracting Quotes and Author Names

Quiz (Extracting Quotes)

Solution (Extracting Quotes)

Clicking on Button

Pagination and Extracting Data

Exception Handling for Unavailable Element

Navigating the Website for Login

Quiz (Login and Extract Quote)

Solution (Login and Extract Quote)

Project Selenium

Overview of Project

Closing the Cookie Button

Setting the Language for Translation

Sending the Text for Translation

Downloading the Translation

Reading Data from File for Translation

Part 2 - Scala and Spark - Master Big Data with Scala and Spark

Why Scala

Scala Applications

About the Instructor

Introduction to Scala and Spark Section

Projects Overview for Scala and Spark

Scala Overview

What is Scala

Scala Setup (Local Machine)

Scala Setup (Online)

Variables in Scala

Arithmetic Operations on Variables-1

Arithmetic Operations on Variables-2

Quiz (Arithmetic Operations)

Solution (Arithmetic Operations)

Quiz (Strings)

Solution (Strings)

Type Casting

Taking Input from User

Quiz (User Input and Type Casting)

Solution (User Input and Type Casting)

Flow Control

Overview of Control Statements

If Else Statements

Conditions in If

Quiz (If Statement)

Solution (If Statement)

Nested If Else

Quiz (Nested If Else)

Solution (Nested If Else)

Logical Operators

Quiz (Logical Operators)

Solution (Logical Operators)

If Else If

Quiz (If Else If)

Solution (If Else If)

Overview of Loops

Overview of While Loop

While Loop

Quiz (While Loop)

Solution 1 (While Loop)

Solution 2 (While Loop)

Do While Loop

For Loop

Quiz 1 (For Loop)

Solution 1 (For Loop)

Quiz 2 (For Loop)

Solution 2 (For Loop)

Break

Break Fix

Project Overview for Flow control

Project Solution Design

Project Solution Code 1

Project Solution Code 2

Project Solution Code 3

Project Solution Code 4

Functions

Overview of Functions

Writing Addition Function

Quiz (Basic Function)

Solution (Basic Function)

Functions Common Issues

Named Arguments

Quiz (String Concatenation Function)

Solution (String Concatenation Function)

Quiz (Dividing Code in Functions)

Solution (Dividing Code in Functions)

Default Arguments

Quiz (Default Arguments)

Solution (Default Arguments)

Anonymous Functions

Quiz (Anonymous Functions)

Solution (Anonymous Functions)

Scopes

Project Overview for Functions

Checking Credentials

Prompting the menu

Basic Functions

Breaking Code in More Functions

Final Run (Functions)

Classes

Introduction to Classes

Creating Class

Class Constructor

Functions and Classes

Project Overview for Classes

Basic Structure

Final Run

Data Structures

Introduction of Data Structures

Lists Introduction

Lists Create and Delete Elements

Lists Take

ListBuffer Introduction

Add Data in ListBuffer

Remove Data from ListBuffer

Take Data from ListBuffer

Project Overview for Data Structures

Project Architecture Discussion

Project Architecture Implementation

User Input for Objects

Implementing the Control Flow

Creating Required Functions Inside Class

Overview of Maps

Creating Maps

Check Key in Map

Update Value in Map

Add and Remove Items from Maps

Iterating on Maps

Project Overview for Data Structures

Project Architecture for Data Structures

Project Structure Code

Using Maps for Word Count

Final Run

Sets Overview

Add and Remove Item from the Set

Set Operations

Overview of Stack

Push and Pop in Stack

Stack Attributes

Project Overview

Project Architecture

Extra Closing Bracket Use Case

Extra Starting Bracket Use Case

Project for Scala and Spark

Project Introduction

Why Spark

Hadoop Ecosystem

Spark Architecture

Spark Ecosystem

DataBricks Account

Setting up DataBricks Cluster

Spark Local Setup

Spark Hadoop Setup

Spark RDDs

Spark RDDs (textFile, collect)

Spark Local Run

Understanding Map

Understanding Flat Map

Understanding Reduce by Key

Word Count Example

Spark DFs

Spark DF Read Data

Spark Print Schema, Select

Spark GroupBy

Spark DF Write

Creating S3 Bucket

Creating Database in RDS

Performing ETL

Part 3 - PySpark and AWS - Master Big Data with PySpark and AWS

Why Big Data

Applications of PySpark

Introduction to Instructor

Introduction to Course

Projects Overview

Introduction to Hadoop, Spark Ecosystems and Architectures

Why Spark

Hadoop Ecosystem

Spark Architecture and Ecosystem

DataBricks Signup

Create DataBricks Notebook

Download Spark and Dependencies

Java Setup on Windows

Python Setup on Windows

Spark Setup on Windows

Hadoop Setup on Windows

Running Spark on Windows

Java Download on MAC

Installing JDK on MAC

Setting Java Home on MAC

Java check on MAC

Installing Python on MAC

Set Up Spark on MAC

Spark RDDs

Spark RDDs Introduction

Creating Spark RDD

Running Spark Code Locally

RDD Map (Lambda)

RDD Map (Simple Function)

Quiz (Map)

Solution 1 (Map)

Solution 2 (Map)

RDD FlatMap

RDD Filter

Quiz (Filter)

Solution (Filter)

RDD Distinct

RDD GroupByKey

RDD ReduceByKey

Quiz (Word Count) with Spark RDDs

Solution (Word Count) with Spark RDDs

RDD (Count and CountByValue)

RDD (saveAsTextFile)

RDD (Partition)

Finding Average-1

Finding Average-2

Quiz (Average)

Solution (Average)

Finding Min and Max

Quiz (Min and Max)

Solution (Min and Max)

Project Overview for Spark RDDs

Total Students

Total Marks by Male and Female Student

Total Passed and Failed Students

Total Enrolments Per Course

Total Marks Per Course

Average Marks Per Course

Finding Minimum and Maximum Marks

Average Age of Male and Female Students

Spark DFs

Introduction to Spark DFs

Creating Spark DFs

Spark Infer Schema

Spark Provide Schema

Create DF from RDD

Rectifying the Error

Select DF Columns

Spark DF withColumn

Spark DF withColumnRenamed and Alias

Spark DF Filter Rows

Quiz (select, withColumn, filter)

Solution (select, withColumn, filter)

Spark DF (Count, Distinct, Duplicate)

Quiz (Distinct, Duplicate)

Solution (Distinct, Duplicate)

Spark DF (sort, orderBy)

Quiz (sort, orderBy)

Solution (sort, orderBy)

Spark DF (Group By)

Spark DF (Group By - Multiple Columns and Aggregations)

Spark DF (Group By -Visualization)

Spark DF (Group By - Filtering)

Quiz (Group By)

Solution (Group By)

Quiz (Word Count) with Spark DFs

Solution (Word Count) with Spark DFs

Spark DF (UDFs)

Quiz (UDFs)

Solution (UDFs)

Solution (Cache and Persist)

Spark DF (DF to RDD)

Spark DF (Spark SQL)

Spark DF (Write DF)

Project Overview

Project (Count and Select)

Project (Group By)

Project (Group By, Aggregations, and Order By)

Project (Filtering)

Project (UDF and WithColumn)

Project (Write)

Collaborative Filtering

Introduction to Collaborative Filtering

Utility Matrix

Explicit and Implicit Ratings

Expected Results

Dataset

Joining Dataframes

Train and Test Data

ALS Model

Hyperparameter Tuning and Cross Validation

Best Model and Evaluate Predictions

Recommendations

Spark Streaming

Introduction to Spark Streaming

Spark Streaming with RDD

Spark Streaming Context

Spark Streaming Reading Data

Spark Streaming Cluster Restart

Spark Streaming RDD Transformations

Spark Streaming DF

Spark Streaming Display

Spark Streaming DF Aggregations

ETL Pipeline

Introduction to ETL

ETL Pipeline Flow

Dataset with ETL Pipeline

Extracting Data

Transforming Data

Loading Data (Creating RDS-I)

Load Data (Creating RDS-II)

RDS Networking

Downloading Postgres

Installing Postgres

Connect to RDS Through PgAdmin

Loading Data

Project - Change Data Capture / Replication On Going

Introduction to Project

Project Architecture

Creating RDS MySQL Instance

Creating S3 Bucket

Creating DMS Source Endpoint

Creating DMS Destination Endpoint

Creating DMS Instance

MySQL WorkBench

Connecting with RDS and Dumping Data

Querying RDS

DMS Full Load

DMS Replication Ongoing

Stopping Instances

Glue Job (Full Load)

Glue Job (Change Capture)

Glue Job (CDC)

Creating Lambda Function and Adding Trigger

Checking Trigger

Getting S3 File Name in Lambda

Creating Glue Job

Adding Invoke for Glue Job

Testing Invoke

Writing Glue Shell Job

Full Load Pipeline

Change Data Capture Pipeline

Part 4 - MongoDB-Mastering MongoDB for Beginners (Theory and Projects)

Why MongoDB

Applications of MongoDB

Instructor Introduction

Whatâ€™s Inside

Methodology

Project

Overview

SQL Schema

NoSQL Schema

Installing MongoDB

Setting Environment Variable

Analogies

Basic Mongo Operations

Basic Database commands Part 1

Basic Database commands Part 2

Basic Collection Commands

Introduction to Module

Create Document (Single)

Create Documents (Many)

Quiz (Create Documents)

Solution (Create Documents)

Quiz (Create Document)

Solution (Create Document)

Outro

Basic Update Operation

Introduction

Update Documents (Single Filter)

Update Documents

Quiz 1 (Update Operation)

Solution 1 (Update Operation)

Quiz 2 (Update Operation)

Solution 2.1 (Update Operation)

Solution 2.2 (Update Operation)

Outro

Basic Read Operation

Introduction

Read Documents

Quiz 1 (Read Documents)

Solution 1 (Read Documents)

Quiz 2 (Read Documents)

Solution 2 (Read Documents)

Outro

Basic Delete Operation

Introduction

Delete Document

Quiz 1 (Delete Operation)

Solution 1 (Delete Operation)

Quiz 2 (Delete Operation)

Solution 2 (Delete Operation)

Outro

Query and projection operators

Module Introduction

$eq Operator

$gt Operator

$lt Operator

$in Operator

$ne Operator

$nin operator

$and Operator

$or Operator

$not Operator

$exists Operator

$types Operator

$expr Operator

$mod Operator

$text Operator

$all Operator

$elemMatch Operator

$size Operator

$ Operator

$slice Operator

Quiz ($eq)

Solution ($eq)

Quiz ($gt)

Solution ($gt)

Quiz ($gte)

Solution ($gte)

Quiz ($in)

Solution ($in)

Quiz ($lt)

Solution ($lt)

Quiz ($lte)

Solution ($lte) Part F10401

Solution ($lte)

Quiz ($ne)

Solution ($ne)

Quiz ($nin)

Solution ($nin) Part 1

Solution ($nin) Part 2

Solution ($nin) Part 3

Quiz ($and)

Solution ($and)

Quiz ($or)

Solution ($or) Part 1

Solution ($or) Part 2

Quiz ($not)

Solution ($not) Part 1

Solution ($not) Part 2

Solution ($not) Part 3

Quiz ($exists)

Solution ($exists)

Quiz ($expr)

Solution ($expr)

Quiz ($mod)

Solution ($mod)

Quiz ($text)

Solution ($text)

Quiz ($all)

Solution ($all) Part 1

Solution ($all) Part 2

Quiz ($elemMatch)

Solution ($elemMatch) Part 1

Solution ($elemMatch) Part 2

Quiz ($size)

Solution ($size)

Update Operators

$currentDate Operator

$inc Operator Part 1

$inc Operator Part 2

$min Operator

$max Operator

$mul Operator

$rename Operator

$set Operator Part 1

$set Operator Part 2

$unset Operator

$addToSet Operator

$pop Operator

$pull Operator

$push Operator

$each Operator

$position Operator

$sort Operator

Quiz 1 (Update Operators)

Solution 1 (Update Operators) Part 1

Solution 1 (Update Operators) Part 2

Solution 1 (Update Operators) Part 3

Solution 1 (Update Operators) Part 4

Quiz 2 (Update Operators)

Solution 2 (Update Operators) Part 1

Solution 2 (Update Operators) Part 2

Solution 2 (Update Operators) Part 3

Mongo with Node

Installing Node on Local Machine

Installing VS Code

Mongo Atlas

Create Cluster on Mongo Atlas

Creating User in Atlas

Network Access

Database and Collections

Connect Node with Mongo

Get Databases

Insert in Mongo Using Node

Read from Mongo Using Node

Update in Mongo Using Node

Delete from Mongo Using Node

Mongo with Python

PyCharm

Creating Connection

Insert in Mongo Using Python

Read from Mongo Using Python

Update in Mongo Using Python

Delete in Mongo Using Python

Django with Mongo

Django Installation

Creating App

Setting Up Django with Mongo

Django Migrations

Django URLs and Views

Django with Postman

Django Get Data from Postman

Insert in Mongo Using Django

Read from Mongo Using Django

Update in Mongo Using Django

Delete in Mongo Using Django

Spark with Mongo

Databricks for Spark

Installing Libraries

Data Overview

ETL

50 Hours of Big Data, PySpark, AWS, Scala, and Scraping

By : AI Sciences

50 Hours of Big Data, PySpark, AWS, Scala, and Scraping

By: AI Sciences

Overview of this book

Confirmation

Buy this book with your credits?

Submit Your Feedback

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access