2. Storage | Learning Hadoop 2

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Learning Hadoop 2

By : GABRIELE MODENA

3.8 (4)

Learning Hadoop 2

3.8 (4)

By: GABRIELE MODENA

Overview of this book

If you are a system or application developer interested in learning how to solve practical problems using the Hadoop framework, then this book is ideal for you. You are expected to be familiar with the Unix/Linux command-line interface and have some experience with the Java programming language. Familiarity with Hadoop would be a plus.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Introduction

1. Introduction

A note on versioning

The background of Hadoop

Components of Hadoop

Hadoop 2 – what's the big deal?

Distributions of Apache Hadoop

A dual approach

AWS – infrastructure on demand from Amazon

Getting started

Running the examples

Data processing with Hadoop

Summary

2. Storage

2. Storage

The inner workings of HDFS

Command-line access to the HDFS filesystem

Protecting the filesystem metadata

Apache ZooKeeper – a different type of filesystem

Automatic NameNode failover

HDFS snapshots

Hadoop filesystems

Managing and serializing data

Storing data

Summary

3. Processing – MapReduce and Beyond

3. Processing – MapReduce and Beyond

MapReduce

Java API to MapReduce

Writing MapReduce programs

Walking through a run of a MapReduce job

YARN

YARN in the real world – Computation beyond MapReduce

Summary

4. Real-time Computation with Samza

4. Real-time Computation with Samza

Stream processing with Samza

Summary

5. Iterative Computation with Spark

5. Iterative Computation with Spark

Apache Spark

The Spark ecosystem

Processing data with Apache Spark

Comparing Samza and Spark Streaming

Summary

6. Data Analysis with Apache Pig

6. Data Analysis with Apache Pig

An overview of Pig

Getting started

Running Pig

Fundamentals of Apache Pig

Programming Pig

Extending Pig (UDFs)

Analyzing the Twitter stream

Summary

7. Hadoop and SQL

7. Hadoop and SQL

Why SQL on Hadoop

Prerequisites

Hive architecture

Hive and Amazon Web Services

Extending HiveQL

Programmatic interfaces

Stinger initiative

Impala

Summary

8. Data Lifecycle Management

8. Data Lifecycle Management

What data lifecycle management is

Building a tweet analysis capability

Challenges of external data

Collecting additional data

Pulling it all together

Summary

9. Making Development Easier

9. Making Development Easier

Choosing a framework

Hadoop streaming

Kite Data

Apache Crunch

Summary

10. Running a Hadoop Cluster

10. Running a Hadoop Cluster

I'm a developer – I don't care about operations!

Cloudera Manager

Ambari – the open source alternative

Operations in the Hadoop 2 world

Sharing resources

Building a physical cluster

Building a cluster on EMR

Cluster tuning

Security

Monitoring

Troubleshooting

Summary

11. Where to Go Next

11. Where to Go Next

Alternative distributions

Other computational frameworks

Other interesting projects

Other programming abstractions

AWS resources

Sources of information

Summary

Index

Index

Command-line access to the HDFS filesystem

Within the Hadoop distribution, there is a command-line utility called hdfs, which is the primary way to interact with the filesystem from the command line. Run this without any arguments to see the various subcommands available. There are many, though; several are used to do things like starting or stopping various HDFS components. The general form of the hdfs command is:

hdfs <sub-command> <command> [arguments]

The two main subcommands we will use in this book are:

dfs: This is used for general filesystem access and manipulation, including reading/writing and accessing files and directories
dfsadmin: This is used for administration and maintenance of the filesystem. We will not cover this command in detail, though. Have a look at the -report command, which gives a listing of the state of the filesystem and all DataNodes:
```
$ hdfs dfsadmin -report
```

Note

Note that the dfs and dfsadmin commands can also be used with the main Hadoop command...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Learning Hadoop 2

Search

Your notes and bookmarks