Book Image

Learning Hadoop 2

Book Image

Learning Hadoop 2

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Learning Hadoop 2
Feb, 2015 | 382 pages
No titles found
Table of Contents (18 chapters)
Learning Hadoop 2
Learning Hadoop 2
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Introduction
Introduction
A note on versioning
The background of Hadoop
Components of Hadoop
Hadoop 2 – what's the big deal?
Distributions of Apache Hadoop
A dual approach
AWS – infrastructure on demand from Amazon
Getting started
Running the examples
Data processing with Hadoop
Summary
2
Storage
Storage
The inner workings of HDFS
Command-line access to the HDFS filesystem
Protecting the filesystem metadata
Apache ZooKeeper – a different type of filesystem
Automatic NameNode failover
HDFS snapshots
Hadoop filesystems
Managing and serializing data
Storing data
Summary
3
Processing – MapReduce and Beyond
Processing – MapReduce and Beyond
MapReduce
Java API to MapReduce
Writing MapReduce programs
Walking through a run of a MapReduce job
YARN
YARN in the real world – Computation beyond MapReduce
Summary
4
Real-time Computation with Samza
Real-time Computation with Samza
Stream processing with Samza
Summary
5
Iterative Computation with Spark
Iterative Computation with Spark
Apache Spark
The Spark ecosystem
Processing data with Apache Spark
Comparing Samza and Spark Streaming
Summary
6
Data Analysis with Apache Pig
Data Analysis with Apache Pig
An overview of Pig
Getting started
Running Pig
Fundamentals of Apache Pig
Programming Pig
Extending Pig (UDFs)
Analyzing the Twitter stream
Summary
7
Hadoop and SQL
Hadoop and SQL
Why SQL on Hadoop
Prerequisites
Hive architecture
Hive and Amazon Web Services
Extending HiveQL
Programmatic interfaces
Stinger initiative
Impala
Summary
8
Data Lifecycle Management
Data Lifecycle Management
What data lifecycle management is
Building a tweet analysis capability
Challenges of external data
Collecting additional data
Pulling it all together
Summary
9
Making Development Easier
Making Development Easier
Choosing a framework
Hadoop streaming
Kite Data
Apache Crunch
Summary
10
Running a Hadoop Cluster
Running a Hadoop Cluster
I'm a developer – I don't care about operations!
Cloudera Manager
Ambari – the open source alternative
Operations in the Hadoop 2 world
Sharing resources
Building a physical cluster
Building a cluster on EMR
Cluster tuning
Security
Monitoring
Troubleshooting
Summary
11
Where to Go Next
Where to Go Next
Alternative distributions
Other computational frameworks
Other interesting projects
Other programming abstractions
AWS resources
Sources of information
Summary
Index
Index

Programming Pig

Pig Latin comes with a number of built-in functions (the eval, load/store, math, string, bag, and tuple functions) and a number of scalar and complex data types. Additionally, Pig allows function and data-type extension by means of UDFs and dynamic invocation of Java methods.

Pig data types

Pig supports the following scalar data types:

  • int: a signed 32-bit integer

  • long: a signed 64-bit integer

  • float: a 32-bit floating point

  • double: a 64-bit floating point

  • chararray: a character array (string) in Unicode UTF-8 format

  • bytearray: a byte array (blob)

  • boolean: a boolean

  • datetime: a datetime

  • biginteger: a Java BigInteger

  • bigdecimal: a Java BigDecimal

Pig supports the following complex data types:

  • map: an associative array enclosed by [], with the key and value separated by #, and items separated by ,

  • tuple: an ordered list of data, where elements can be of any scalar or complex type enclosed by (), with items separated by ,

  • bag: an unordered collection of tuples enclosed by {} and...

Previous Section
End of Section 6
Next Section