Book Image

Linux Shell Scripting Cookbook - Third Edition

By : Clif Flynt, Sarath Lakshman, Shantanu Tushar
Book Image

Linux Shell Scripting Cookbook - Third Edition

By: Clif Flynt, Sarath Lakshman, Shantanu Tushar

Overview of this book

The shell is the most powerful tool your computer provides. Despite having it at their fingertips, many users are unaware of how much the shell can accomplish. Using the shell, you can generate databases and web pages from sets of files, automate monotonous admin tasks such as system backups, monitor your system's health and activity, identify network bottlenecks and system resource hogs, and more. This book will show you how to do all this and much more. This book, now in its third edition, describes the exciting new features in the newest Linux distributions to help you accomplish more than you imagine. It shows how to use simple commands to automate complex tasks, automate web interactions, download videos, set up containers and cloud servers, and even get free SSL certificates. Starting with the basics of the shell, you will learn simple commands and how to apply them to real-world issues. From there, you'll learn text processing, web interactions, network and system monitoring, and system tuning. Software engineers will learn how to examine system applications, how to use modern software management tools such as git and fossil for their own work, and how to submit patches to open-source projects. Finally, you'll learn how to set up Linux Containers and Virtual machines and even run your own Cloud server with a free SSL Certificate from letsencrypt.org.

Related Content you might be interested in

Current Title:

Small book image
Linux Shell Scripting Cookbook - Third Edition
Clif Flynt | Sarath Lakshman | Shantanu Tushar
May, 2017 | 552 pages
Small book image
Bash Cookbook
Ganesh Naik | Ron Brash
Jul, 2018 | 264 pages
Small book image
CentOS Quick Start Guide
Shiwang Kalkhanda
Dec, 2018 | 320 pages
Small book image
Learning Linux Shell Scripting
Ganesh Naik
May, 2018 | 332 pages
Table of Contents (14 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Sections
Conventions
Reader feedback
Customer support
Free Chapter
1
Shell Something Out
Shell Something Out
Introduction
Displaying output in a terminal
Using variables and environment variables
Function to prepend to environment variables
Math with the shell
Playing with file descriptors and redirection
Arrays and associative arrays
Visiting aliases
Grabbing information about the terminal
Getting and setting dates and delays
Debugging the script
Functions and arguments
Sending output from one command to another
Reading n characters without pressing the return key
Running a command until it succeeds
Field separators and iterators
Comparisons and tests
Customizing bash with configuration files
2
Have a Good Command
Have a Good Command
Introduction
Concatenating with cat
Recording and playing back terminal sessions
Finding files and file listing
Playing with xargs
Translating with tr
Checksum and verification
Cryptographic tools and hashes
Sorting unique and duplicate lines
Temporary file naming and random numbers
Splitting files and data
Slicing filenames based on extensions
Renaming and moving files in bulk
Spell–checking and dictionary manipulation
Automating interactive input
Making commands quicker by running parallel processes
Examining a directory, files and subdirectories in it
3
File In, File Out
File In, File Out
Introduction
Generating files of any size
The intersection and set difference (A-B) on text files
Finding and deleting duplicate files
Working with file permissions, ownership, and the sticky bit
Making files immutable
Generating blank files in bulk
Finding symbolic links and their targets
Enumerating file type statistics
Using loopback files
Creating ISO files and hybrid ISO
Finding the difference between files, and patching
Using head and tail for printing the last or first 10 lines
Listing only directories - alternative methods
Fast command-line navigation using pushd and popd
Counting the number of lines, words, and characters in a file
Printing the directory tree
Manipulating video and image files
4
Texting and Driving
Texting and Driving
Introduction
Using regular expressions
Searching and mining text inside a file with grep
Cutting a file column-wise with cut
Using sed to perform text replacement
Using awk for advanced text processing
Finding the frequency of words used in a given file
Compressing or decompressing JavaScript
Merging multiple files as columns
Printing the nth word or column in a file or line
Printing text between line numbers or patterns
Printing lines in the reverse order
Parsing e-mail address and URLs from text
Removing a sentence in a file containing a word
Replacing a pattern with text in all the files in a directory
Text slicing and parameter operations
5
Tangled Web? Not At All!
Tangled Web? Not At All!
Introduction
Downloading from a web page
Downloading a web page as plain text
A primer on cURL
Accessing unread Gmail e-mails from the command line
Parsing data from a website
Image crawler and downloader
Web photo album generator
Twitter command-line client
Accessing word definitions via a web server
Finding broken links in a website
Tracking changes to a website
Posting to a web page and reading the response
Downloading a video from the Internet
Summarizing text with OTS
Translating text from the command line
6
Repository Management
Repository Management
Introduction
Creating a new git repository
Cloning a remote git repository
Adding and committing changes with git
Creating and merging branches with git
Sharing your work
Pushing a branch to a server
Checking the status of a git repository
Viewing git history
Finding bugs
Tagging snapshots
Committing message ethics
Using fossil
Creating a new fossil repository
Cloning a remote fossil repository
Opening a fossil project
Adding and committing changes with fossil
Using branches and forks with fossil
Sharing your work with fossil
Updating your local fossil repository
Checking the status of a fossil repository
Viewing fossil history
7
The Backup Plan
The Backup Plan
Introduction
Archiving with tar
Archiving with cpio
Compressing data with gzip
Archiving and compressing with zip
Faster archiving with pbzip2
Creating filesystems with compression
Backing up snapshots with rsync
Differential archives
Creating entire disk images using fsarchiver
8
The Old-Boy Network
The Old-Boy Network
Introduction
Setting up the network
Let us ping!
Tracing IP routes
Listing all available machines on a network
Running commands on a remote host with SSH
Running graphical commands on a remote machine
Transferring files through the network
Connecting to a wireless network
Password-less auto-login with SSH
Port forwarding using SSH
Mounting a remote drive at a local mount point
Network traffic and port analysis
Measuring network bandwidth
Creating arbitrary sockets
Building a bridge
Sharing an Internet connection
Basic firewall using iptables
Creating a Virtual Private Network
9
Put On the Monitors Cap
Put On the Monitors Cap
Introduction
Monitoring disk usage
Calculating the execution time for a command
Collecting information about logged in users, boot logs, and boot failures
Listing the top ten CPU– consuming processes in an hour
Monitoring command outputs with watch
Logging access to files and directories
Logging with syslog
Managing log files with logrotate
Monitoring user logins to find intruders
Monitoring remote disk usage health
Determining active user hours on a system
Measuring and optimizing power usage
Monitoring disk activity
Checking disks and filesystems for errors
Examining disk health
Getting disk statistics
10
Administration Calls
Administration Calls
Introduction
Gathering information about processes
What's what – which, whereis, whatis, and file
Killing processes, and sending and responding to signals
Sending messages to user terminals
The /proc filesystem
Gathering system information
Scheduling with a cron
Database styles and uses
Writing and reading SQLite databases
Writing and reading a MySQL database from Bash
User administration scripts
Bulk image resizing and format conversion
Taking screenshots from the terminal
Managing multiple terminals from one
11
Tracing the Clues
Tracing the Clues
Introduction
Tracing packets with tcpdump
Finding packets with ngrep
Tracing network routes with ip
Tracing system calls with strace
Tracing dynamic library functions with ltrace
12
Tuning a Linux System
Tuning a Linux System
Introduction
Identifying services
Gathering socket data with ss
Gathering system I/O usage with dstat
Identifying a resource hog with pidstat
Tuning the Linux kernel with sysctl
Tuning a Linux system with config files
Changing scheduler priority using the nice command
13
Containers, Virtual Machines, and the Cloud
Containers, Virtual Machines, and the Cloud
Introduction
Using Linux containers
Using Docker
Using Virtual Machines in Linux
Linux in the cloud

Using loopback files

Linux filesystems normally exist on devices such as disks or memory sticks. A file can also be mounted as a filesystem. This filesystem-in-a-file can be used for testing, for customized filesystems, or even as an encrypted disk for confidential information.

How to do it...

To create a 1 GB ext4 filesystem in a file, follow these steps:

  1. Use dd to create a 1 GB file:
        $ dd if=/dev/zero of=loobackfile.img bs=1G count=1
        1024+0 records in
        1024+0 records out
        1073741824 bytes (1.1 GB) copied, 37.3155 s, 28.8 MB/s

 The size of the created file exceeds 1 GB because the hard disk is a block device, and hence, storage must be allocated by integral multiples of blocks size.

  1. Format the 1 GB file to ext4 using the mkfs...
Previous Section
End of Section 11
Next Section