Linux Shell Scripting Cookbook - Third Edition

By : Clif Flynt, Sarath Lakshman, Shantanu Tushar

Linux Shell Scripting Cookbook - Third Edition

By: Clif Flynt, Sarath Lakshman, Shantanu Tushar

Overview of this book

The shell is the most powerful tool your computer provides. Despite having it at their fingertips, many users are unaware of how much the shell can accomplish. Using the shell, you can generate databases and web pages from sets of files, automate monotonous admin tasks such as system backups, monitor your system's health and activity, identify network bottlenecks and system resource hogs, and more. This book will show you how to do all this and much more. This book, now in its third edition, describes the exciting new features in the newest Linux distributions to help you accomplish more than you imagine. It shows how to use simple commands to automate complex tasks, automate web interactions, download videos, set up containers and cloud servers, and even get free SSL certificates. Starting with the basics of the shell, you will learn simple commands and how to apply them to real-world issues. From there, you'll learn text processing, web interactions, network and system monitoring, and system tuning. Software engineers will learn how to examine system applications, how to use modern software management tools such as git and fossil for their own work, and how to submit patches to open-source projects. Finally, you'll learn how to set up Linux Containers and Virtual machines and even run your own Cloud server with a free SSL Certificate from letsencrypt.org.

Preface

What this book covers

What you need for this book

Free Chapter

Shell Something Out

Introduction

Displaying output in a terminal

Using variables and environment variables

Function to prepend to environment variables

Math with the shell

Playing with file descriptors and redirection

Arrays and associative arrays

Visiting aliases

Grabbing information about the terminal

Getting and setting dates and delays

Debugging the script

Functions and arguments

Sending output from one command to another

Reading n characters without pressing the return key

Running a command until it succeeds

Field separators and iterators

Comparisons and tests

Customizing bash with configuration files

Have a Good Command

Introduction

Concatenating with cat

Recording and playing back terminal sessions

Finding files and file listing

Playing with xargs

Translating with tr

Checksum and verification

Cryptographic tools and hashes

Sorting unique and duplicate lines

Temporary file naming and random numbers

Splitting files and data

Slicing filenames based on extensions

Renaming and moving files in bulk

Spell–checking and dictionary manipulation

Automating interactive input

Making commands quicker by running parallel processes

Examining a directory, files and subdirectories in it

File In, File Out

Introduction

Generating files of any size

The intersection and set difference (A-B) on text files

Finding and deleting duplicate files

Working with file permissions, ownership, and the sticky bit

Making files immutable

Generating blank files in bulk

Finding symbolic links and their targets

Enumerating file type statistics

Using loopback files

Creating ISO files and hybrid ISO

Finding the difference between files, and patching

Using head and tail for printing the last or first 10 lines

Listing only directories - alternative methods

Fast command-line navigation using pushd and popd

Counting the number of lines, words, and characters in a file

Printing the directory tree

Manipulating video and image files

Texting and Driving

Introduction

Using regular expressions

Searching and mining text inside a file with grep

Cutting a file column-wise with cut

Using sed to perform text replacement

Using awk for advanced text processing

Finding the frequency of words used in a given file

Compressing or decompressing JavaScript

Merging multiple files as columns

Printing the nth word or column in a file or line

Printing text between line numbers or patterns

Printing lines in the reverse order

Parsing e-mail address and URLs from text

Removing a sentence in a file containing a word

Replacing a pattern with text in all the files in a directory

Text slicing and parameter operations

Tangled Web? Not At All!

Introduction

Downloading from a web page

Downloading a web page as plain text

A primer on cURL

Accessing unread Gmail e-mails from the command line

Parsing data from a website

Image crawler and downloader

Web photo album generator

Twitter command-line client

Accessing word definitions via a web server

Finding broken links in a website

Tracking changes to a website

Posting to a web page and reading the response

Downloading a video from the Internet

Summarizing text with OTS

Translating text from the command line

Repository Management

Introduction

Creating a new git repository

Cloning a remote git repository

Adding and committing changes with git

Creating and merging branches with git

Sharing your work

Pushing a branch to a server

Checking the status of a git repository

Viewing git history

Finding bugs

Tagging snapshots

Committing message ethics

Using fossil

Creating a new fossil repository

Cloning a remote fossil repository

Opening a fossil project

Adding and committing changes with fossil

Using branches and forks with fossil

Sharing your work with fossil

Updating your local fossil repository

Checking the status of a fossil repository

Viewing fossil history

The Backup Plan

Introduction

Archiving with tar

Archiving with cpio

Compressing data with gzip

Archiving and compressing with zip

Faster archiving with pbzip2

Creating filesystems with compression

Backing up snapshots with rsync

Differential archives

Creating entire disk images using fsarchiver

The Old-Boy Network

Introduction

Setting up the network

Let us ping!

Tracing IP routes

Listing all available machines on a network

Running commands on a remote host with SSH

Running graphical commands on a remote machine

Transferring files through the network

Connecting to a wireless network

Password-less auto-login with SSH

Port forwarding using SSH

Mounting a remote drive at a local mount point

Network traffic and port analysis

Measuring network bandwidth

Creating arbitrary sockets

Building a bridge

Sharing an Internet connection

Basic firewall using iptables

Creating a Virtual Private Network

Put On the Monitors Cap

Introduction

Monitoring disk usage

Calculating the execution time for a command

Collecting information about logged in users, boot logs, and boot failures

Listing the top ten CPU– consuming processes in an hour

Monitoring command outputs with watch

Logging access to files and directories

Logging with syslog

Managing log files with logrotate

Monitoring user logins to find intruders

Monitoring remote disk usage health

Determining active user hours on a system

Measuring and optimizing power usage

Monitoring disk activity

Checking disks and filesystems for errors

Examining disk health

Getting disk statistics

Administration Calls

Introduction

Gathering information about processes

What's what – which, whereis, whatis, and file

Killing processes, and sending and responding to signals

Sending messages to user terminals

The /proc filesystem

Gathering system information

Scheduling with a cron

Database styles and uses

Writing and reading SQLite databases

Writing and reading a MySQL database from Bash

User administration scripts

Bulk image resizing and format conversion

Taking screenshots from the terminal

Managing multiple terminals from one

Tracing the Clues

Introduction

Tracing packets with tcpdump

Finding packets with ngrep

Tracing network routes with ip

Tracing system calls with strace

Tracing dynamic library functions with ltrace

Tuning a Linux System

Introduction

Identifying services

Gathering socket data with ss

Gathering system I/O usage with dstat

Identifying a resource hog with pidstat

Tuning the Linux kernel with sysctl

Tuning a Linux system with config files

Changing scheduler priority using the nice command

Containers, Virtual Machines, and the Cloud

Introduction

Using Linux containers

Using Docker

Using Virtual Machines in Linux

Linux in the cloud

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Temporary file naming and random numbers

Shell scripts often need to store temporary data. The most suitable location to do this is /tmp (which will be cleaned out by the system on reboot). There are two methods to generate standard filenames for temporary data.

How to do it...

The mktemp command will create a unique temporary file or folder name:

Create a temporary file:

        $ filename=`mktemp`
        $ echo $filename
        /tmp/tmp.8xvhkjF5fH

This creates a temporary file, stores the name in filename, and then displays the name.

Create a temporary directory:

        $ dirname=`mktemp -d`
        $ echo $dirname
        tmp.NI8xzW7VRX

This creates a temporary directory, stores the name in filename, and displays the name.

To generate a filename without...

Linux Shell Scripting Cookbook - Third Edition

By : Clif Flynt, Sarath Lakshman, Shantanu Tushar

Linux Shell Scripting Cookbook - Third Edition

By: Clif Flynt, Sarath Lakshman, Shantanu Tushar

Overview of this book

Related Content you might be interested in

Current Title:

Linux Shell Scripting Cookbook - Third Edition

Bash Cookbook

CentOS Quick Start Guide

Learning Linux Shell Scripting