Book Image

Raspberry Pi Super Cluster

By : Andrew K. Dennis
Book Image

Raspberry Pi Super Cluster

By: Andrew K. Dennis

Overview of this book

A cluster is a type of parallel/distributed processing system which consists of a collection of interconnected stand-alone computers cooperatively working together. Using Raspberry Pi computers, you can build a two-node parallel computing cluster which enhances performance and availability. This practical, example-oriented guide will teach you how to set up the hardware and operating systems of multiple Raspberry Pi computers to create your own cluster. It will then navigate you through how to install the necessary software to write your own programs such as Hadoop and MPICH before moving on to cover topics such as MapReduce. Throughout this book, you will explore the technology with the help of practical examples and tutorials to help you learn quickly and efficiently. Starting from a pile of hardware, with this book, you will be guided through exciting tutorials that will help you turn your hardware into your own super-computing cluster. You'll start out by learning how to set up your Raspberry Pi cluster's hardware. Following this, you will be taken through how to install the operating system, and you will also be given a taste of what parallel computing is about. With your Raspberry Pi cluster successfully set up, you will then install software such as MPI and Hadoop. Having reviewed some examples and written some programs that explore these two technologies, you will then wrap up with some fun ancillary projects. Finally, you will be provided with useful links to help take your projects to the next step.
Table of Contents (15 chapters)
Raspberry Pi Super Cluster
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Index

A

  • A1 Overscan / The installation process
  • A2 Hostname / The installation process
  • A3 Memory Split / The installation process
  • A4 SSH / The installation process
  • A5 Update / The installation process
  • Adafruit
    • URL / Raspberry Pi cases and clusters
  • Amazon website
    • URL / Power over Ethernet
  • Apache Foundation
    • URL / MPI, Hadoop, and parallel computing
  • Apache Hadoop
    • about / A brief introduction to Apache Hadoop
    • installing / Installing Apache Hadoop
    • configuration settings / Hadoop configuration
    • server, testing / Testing our Hadoop server
    • second Raspberry Pi, setting up / Setting up our second Raspberry Pi
  • Apache Hadoop (High-availability distributed object-orientated platform)
    • about / Programming languages and frameworks
  • Apache website
    • URL / Hadoop configuration
  • Apach Hadoop
    • tar.gz file, downloading / Installing Apache Hadoop
  • application
    • testing / Testing our application
  • Archiver
    • URL / Mac OS X
  • AWS / Cloud computing
  • AWS Elastic Compute Cloud (EC2) / Cloud computing

B

  • batteries / Battery power
  • BerryBoot version 2
    • about / BerryBoot version 2
    • ZIP file, downloading / Downloading the BerryBoot version 2 ZIP file
  • BerryBoot version 2 ZIP file
    • Mac OS X / Mac OS X
    • Windows 8 / Windows 8
    • Linux / Linux
  • big data
    • about / Big data
  • BitBucket website
    • URL / HDFS – The Hadoop distributed file system
  • blocking_send.c file / MPI – point-to-point communication
  • booting
    • from external USB HDD / Booting from an external USB HDD

C

  • C
    • used, for calculating Pi / Pi with C language and MPI
  • Cardiff University
    • URL / MPI, Hadoop, and parallel computing
  • CDC 6600 / Supercomputers
  • CHameleon
    • about / MPI implementations – MPICH and OpenMPI
  • cloud computing
    • about / Cloud computing
  • concurrency
    • about / A very short history of parallel computing
  • conf directory / Hadoop configuration
  • Control Data Corporation (CDC) / Supercomputers
  • core-site.xml file / Hadoop configuration, Setting up our second Raspberry Pi
  • COTS
    • about / Commodity hardware clusters
  • Cray-1 / Supercomputers

D

  • DataNodes / MapReduce in Hadoop
  • dfs.replication parameter / HDFS – The Hadoop distributed file system
  • Duke University
    • URL / MPI, Hadoop, and parallel computing

E

  • eLinux.org
    • URL / Installing Java
  • eLinux page
    • USB drive formatting, URL / Booting from an external USB HDD
  • eLinux verified peripherals
    • URL / USB hard drives (optional)
  • eLinux website
    • URL / USB keyboard and mouse
  • eLinux Wiki
    • URL / Two micro-USB power units
  • Expand Filesystem / The installation process

F

  • FAT (File Allocation Table) / SD card setup
  • file system image (fsimage) / Hadoop configuration
  • Fortran
    • about / Programming languages and frameworks, Experimenting with MPI and Fortran
    • installing / Installing Fortran
    • based MPI programs, URL / Experimenting with MPI and Fortran
  • Fortran 77 programming guide
    • URL / Fortran and C/C++
  • Fortran and MPI
    • URL / Fortran and C/C++
  • Fortran Company
    • URL / Fortran and C/C++
  • Fortran Specialist Group
    • URL / Fortran and C/C++
  • fstab file / Booting from an external USB HDD

G

  • Gribble Lab
    • URL / MPI, Hadoop, and parallel computing

H

  • H (hit) / Monte Carlo simulators
  • Hadoop
    • website, URL / HDFS – The Hadoop distributed file system
  • Hadoop application
    • used, for calculating Pi / A Hadoop application to calculate Pi
  • Hadoop Distributed File System (HDFS) / A brief introduction to Apache Hadoop
  • Hadoop FileSystem shell / HDFS – The Hadoop distributed file system
  • hadoop version command / Hadoop configuration
  • Hadoop Wiki
    • URL / MapReduce in Hadoop
  • HDFS
    • about / HDFS – The Hadoop distributed file system
  • hdfs-site.xml file / Hadoop configuration
  • hello_rpi.c file / MPI – point-to-point communication
  • High Performance Computing (HPC) / A very short history of parallel computing
  • Hive / Big data
  • HiveQL / Big data
  • hostfile / Configuring your Raspberry Pi to run with MPICH

I

  • initial setup
    • about / Completing the initial setup
  • insidecircle() function / Pi with C language and MPI
  • insidercircle() function / Pi with C language and MPI
  • installation
    • MPICH, testing / Testing our MPICH installation
  • Intel
    • URL / MPI, Hadoop, and parallel computing
  • Internationalisation Options / The installation process

J

  • Java
    • installing / Installing Java
  • Java Development Kit (JDK) / Installing Java
  • JobTracker / MapReduce in Hadoop
  • JRE (Java Run-time Environment) / Installing Java

L

  • Lego
    • about / Building a Lego enclosure
    • website, URL / Building a Lego enclosure
    • custom brick ordering, URL / Building a Lego enclosure
  • Lego enclosure
    • building / Building a Lego enclosure
  • Linux
    • SD card formatting, instructions / Linux instructions for SD card formatting
    • Raspberry Pi (second), building / Linux
  • Linux.org website
    • URL / Starting up the Raspberry Pi
  • Local Area Network (LAN)
    • about / Commodity hardware clusters

M

  • machines
    • multi core / Multi-core and multiprocessor machines
    • multiprocessor / Multi-core and multiprocessor machines
  • Mac OS X
    • SD card formatting, instructions / Mac OS X SD card formatting instructions
    • Raspberry Pi (second), building / Mac OS X
  • main() function / Pi with C language and MPI
  • Map function / MapReduce
  • mapred-site.xml file / Hadoop configuration
  • MapReduce
    • about / MapReduce
    • working / MapReduce
    • in Hadoop / MapReduce in Hadoop
  • MapReduce, in Hadoop
    • NameNode / MapReduce in Hadoop
    • JobTracker / MapReduce in Hadoop
    • DataNodes / MapReduce in Hadoop
    • TaskTracker / MapReduce in Hadoop
    • default configuration file / MapReduce in Hadoop
    • site-specific configuration / MapReduce in Hadoop
  • MapReduce program
    • WordCount example / The WordCount MapReduce program
  • Monte Carlo simulators
    • about / Monte Carlo simulators
  • MPI
    • about / Programming languages and frameworks, MPI – Message Passing Interface
    • implementations / MPI implementations – MPICH and OpenMPI
    • OpenMPI / MPI implementations – MPICH and OpenMPI
    • CH / MPI implementations – MPICH and OpenMPI
    • used, for calculating Pi / Pi with C language and MPI
    • resources, URL / MPI, Hadoop, and parallel computing
    • tutorials, URL / MPI, Hadoop, and parallel computing
  • MPI-based application
    • writing / Writing an MPI-based application
  • MPI based application
    • creating, Fortran used / Experimenting with MPI and Fortran
  • MPICH
    • URL / MPI implementations – MPICH and OpenMPI
    • about / Creating an environment and downloading MPICH
    • downloading / Creating an environment and downloading MPICH
    • building / Building and installing MPICH
    • installing / Building and installing MPICH
    • Makefile, running / Building and installing MPICH
    • installation, testing / Testing our MPICH installation

N

  • NameNode / MapReduce in Hadoop
  • nano
    • URL / Editing text files on Raspbian
  • Network Of Workstations (NOW) / Commodity hardware clusters
  • NOOBS / SD card setup

O

  • OOP (Object Orientated Programming) / MapReduce
  • OpenMP / Multi-core and multiprocessor machines
  • OpenMPI
    • about / MPI implementations – MPICH and OpenMPI
    • URL / MPI implementations – MPICH and OpenMPI
  • Overclock / The installation process

P

  • parallel computing
    • about / A very short history of parallel computing
    • and Raspberry Pi / Raspberry Pi and parallel computing
  • parallelism
    • about / A very short history of parallel computing
  • parallel virtual machine (PVM)
    • URL / Cloud computing
  • PATH variable / Hadoop configuration
  • Pi
    • calculating / Monte Carlo simulators
    • calculating, Hadoop application used / A Hadoop application to calculate Pi
    • calculating, C used / Pi with C language and MPI
    • calculating, MPI used / Pi with C language and MPI
  • pifile / Configuring your Raspberry Pi to run with MPICH
  • point-to-point communication
    • about / MPI – point-to-point communication
  • POSIX Threads (Pthreads) / Multi-core and multiprocessor machines
  • power, for multiple devices
    • about / Power for multiple devices
    • USB wall plate / USB wall plates
    • battery power / Battery power
    • PC power supply, using / Using a PC power supply
    • Ethernet connection / Power over Ethernet
  • prerequisites, Raspberry Pi
    • setting up / Setting up our work environment
    • HDMI capable monitor / HDMI-capable monitor or VGA/DVI monitor and adapter
    • VGA/DVI monitor and adapter / HDMI-capable monitor or VGA/DVI monitor and adapter
    • USB keyboard and mouse / USB keyboard and mouse
    • two micro-USB power units / Two micro-USB power units
    • desk mounted power strip / A desk-mounted power strip with both USB and mains outlets (optional)
    • Ethernet/RJ45 network cables / Three Ethernet/RJ45 network cables
    • network switch / A small network switch
    • Internet connection / An existing Internet connection
    • SD cards, Raspberry Pi compatible / Two SD cards that are compatible with the Raspberry Pi
    • housing units, for boards / Housing units for the Raspberry Pi boards and Lego (optional)
    • housing units, for lego / Housing units for the Raspberry Pi boards and Lego (optional)
    • USB hard drives (optional) / USB hard drives (optional)
  • programming languages
    • about / Programming languages and frameworks
  • property tag / Hadoop configuration
  • PuTTY
    • Windows 8 users / Windows 8 users with PuTTY
  • PuTTY executable file
    • URL / Connecting via SSH

Q

  • Quick format option / Formatting our card

R

  • Raspberry Pi
    • and parallel computing / Raspberry Pi and parallel computing
    • URL / Raspberry Pi and parallel computing
    • prerequisites / Setting up our work environment
    • work environment / Setting up our work environment
    • initial setup / Completing the initial setup
    • storage device, SD card using as / Using an SD card as our Raspberry Pi's storage device
    • about / Starting up the Raspberry Pi
    • installation / The installation process
    • SSH, testing / Testing SSH and setting up keys
    • keys, setting up / Testing SSH and setting up keys
    • SSH, connecting via / Connecting via SSH
    • Mac OS X and Linux users / Mac OS X and Linux users
    • Windows 8 users, with PuTTY / Windows 8 users with PuTTY
    • SSH, running / SSH running successfully
    • configuring, to run with MPICH / Configuring your Raspberry Pi to run with MPICH, Testing our MPICH installation
    • booting, from USB HDD / Booting from an external USB HDD
    • Lego enclosure / Building a Lego enclosure
    • power, for multiple devices / Power for multiple devices
    • USB wall plate, installing / USB wall plates
    • batteries / Battery power
    • powering via battery, URL / Battery power
    • PC power supply, using / Using a PC power supply
    • Ethernet connection / Power over Ethernet
  • Raspberry Pi (second)
    • building / Building our second Raspberry Pi
    • SD card, cloning in Windows / Windows 8
    • building, on Mac OS X / Mac OS X
    • SD card, cloning on Linux / Linux
    • powering up / Powering up the second Raspberry Pi
    • RSA key, setting up for SSH / RSA key setup for SSH
  • Raspberry Pi case design
    • Adafruit, URL / Raspberry Pi cases and clusters
    • Southampton University, URL / Raspberry Pi cases and clusters
  • Raspberry Pi cases
    • Thingiverse, URL / Raspberry Pi cases and clusters
  • Raspberry Pi Cloud blog
    • URL / MPI, Hadoop, and parallel computing
  • Raspberry Pi Software Configuration Tool (raspi-config) / The installation process
  • Raspbian
    • URL / Starting up the Raspberry Pi
    • text files, editing on / Editing text files on Raspbian
  • raspi-config screen / The installation process
  • Red Hat
    • URL / MPI, Hadoop, and parallel computing
  • Reduce function / MapReduce
  • Remote Procedure Calls (RPC) / HDFS – The Hadoop distributed file system
  • RSA (Rivest, Shamir, and Adleman)
    • about / Setting up your SSH RSA keys
    • URL / Setting up your SSH RSA keys

S

  • safer / MPI – point-to-point communication
  • screen
    • URL / The ssh-agent and ssh-add tools
    • terminal multiplexer with / Terminal multiplexing with Screen
  • SD card
    • URL / Two SD cards that are compatible with the Raspberry Pi
    • using, as Raspberry Pi storage device / Using an SD card as our Raspberry Pi's storage device
    • brands, URL / Using an SD card as our Raspberry Pi's storage device
    • set up / SD card setup
    • formatting / Formatting our card
    • Mac OS X SD card formatting, instructions / Mac OS X SD card formatting instructions
    • Windows 8 SD card formatting, instructions / Windows 8 SD card formatting instructions
    • Linux instructions, for formatting / Linux instructions for SD card formatting
  • source command / Hadoop configuration
  • Southampton University
    • URL / Raspberry Pi cases and clusters
  • SSH
    • running / SSH running successfully
    • RSA key, setting up / RSA key setup for SSH
  • ssh-add tools / The ssh-agent and ssh-add tools
  • ssh-agent / The ssh-agent and ssh-add tools
  • SSH RSA keys
    • about / Setting up your SSH RSA keys
    • ssh-agent / The ssh-agent and ssh-add tools
    • ssh-add tools / The ssh-agent and ssh-add tools
    • SSH, setup / SSH setup complete
  • start-dfs.sh shell script / HDFS – The Hadoop distributed file system
  • start-mapred.sh script / HDFS – The Hadoop distributed file system
  • synchronous
    • about / MPI – point-to-point communication

T

  • T (total) / Monte Carlo simulators
  • TaskTracker / MapReduce in Hadoop
  • terminal multiplexer
    • about / Terminal multiplexing with Screen
  • Thingiverse
    • URL / Raspberry Pi cases and clusters

U

  • USB HDD
    • booting from / Booting from an external USB HDD
    • setting up / Booting from an external USB HDD
  • USB wall plate / USB wall plates

V

  • Vim
    • URL / Editing text files on Raspbian
  • Virtual Box
    • URL / MPI, Hadoop, and parallel computing
  • VoIP (Voice over IP) phone / Power over Ethernet

W

  • Win32 Disk Imager / Windows 8
  • Windows 8
    • SD card formatting, instructions / Windows 8 SD card formatting instructions
    • Raspberry Pi (second), building / Windows 8
  • WinZip
    • for Mac, URL / Mac OS X
    • URL / Windows 8

Z

  • 7-zip
    • URL / Windows 8