Playing with file descriptors and redirection
File descriptors are integers that are associated with file input and output. They keep track of opened files. The best-known file descriptors are stdin
, stdout
, and stderr
. We even can redirect the contents of one file descriptor to another. This recipe shows examples on how to manipulate and redirect with file descriptors.
Getting ready
While writing scripts we use standard input (stdin
), standard output (stdout
), and standard error (stderr
) frequently. Redirection of an output to a file by filtering the contents is one of the essential things we need to perform. While a command outputs some text, it can be either an error or an output (nonerror) message. We cannot distinguish whether it is output text or an error text by just looking at it. However, we can handle them with file descriptors. We can extract text that is attached to a specific descriptor.
File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved as follows:
0:
stdin
(standard input)1:
stdout
(standard output)2:
stderr
(standard error)
How to do it...
Redirecting or saving output text to a file can be done as follows:
$ echo "This is a sample text 1" > temp.txt
This would store the echoed text in
temp.txt
by truncating the file, the contents will be emptied before writing.To append text to a file, consider the following example:
$ echo "This is sample text 2" >> temp.txt
You can view the contents of the file as follows:
$ cat temp.txt This is sample text 1 This is sample text 2
Let us see what a standard error is and how you can redirect it.
stderr
messages are printed when commands output an error message. Consider the following example:$ ls + ls: cannot access +: No such file or directory
Here
+
is an invalid argument and hence an error is returned.Tip
Successful and unsuccessful commands
When a command returns after an error, it returns a nonzero exit status. The command returns zero when it terminates after successful completion. The return status can be read from special variable
$?
(runecho $?
immediately after the command execution statement to print the exit status).The following command prints the
stderr
text to the screen rather than to a file (and because there is nostdout
output,out.txt
will be empty):$ ls + > out.txt ls: cannot access +: No such file or directory
In the following command, we redirect
stderr
toout.txt
:$ ls + 2> out.txt # works
You can redirect
stderr
exclusively to a file andstdout
to another file as follows:$ cmd 2>stderr.txt 1>stdout.txt
It is also possible to redirect
stderr
andstdout
to a single file by convertingstderr
tostdout
using this preferred method:$ cmd 2>&1 output.txt
Or the alternate approach:
$ cmd &> output.txt
Sometimes, the output may contain unnecessary information (such as debug messages). If you don't want the output terminal burdened with the
stderr
details then you should redirect thestderr
output to/dev/null
, which removes it completely. For example, consider that we have three filesa1
,a2
, anda3
. However,a1
does not have the read-write-execute permission for the user. When you need to print the contents of files starting witha
, we use thecat
command. Set up the test files as follows:$ echo a1 > a1 $ cp a1 a2 ; cp a2 a3; $ chmod 000 a1 #Deny all permissions
While displaying contents of the files using wildcards (
a*
), it will show an error message for filea1
as it does not have the proper read permission:$ cat a* cat: a1: Permission denied a1 a1
Here,
cat: a1: Permission denied
belongs to thestderr
data. We can redirect thestderr
data into a file, whereasstdout
remains printed in the terminal. Consider the following code:$ cat a* 2> err.txt #stderr is redirected to err.txt a1 a1 $ cat err.txt cat: a1: Permission denied
Take a look at the following code:
$ cmd 2>/dev/null
When redirection is performed for
stderr
orstdout
, the redirected text flows into a file. As the text has already been redirected and has gone into the file, no text remains to flow to the next command through pipe (|
), and it appears to the next set of command sequences throughstdin
.However, there is a way to redirect data to a file, as well as provide a copy of redirected data as
stdin
for the next set of commands. This can be done using thetee
command. For example, to printstdout
in the terminal as well as redirectstdout
into a file, the syntax fortee
is as follows:command | tee FILE1 FILE2
In the following code, the
stdin
data is received by thetee
command. It writes a copy ofstdout
to theout.txt
file and sends another copy asstdin
for the next command. Thecat -n
command puts a line number for each line received fromstdin
and writes it intostdout
:$ cat a* | tee out.txt | cat -n cat: a1: Permission denied 1a1 2a1
Examine the contents of
out.txt
as follows:$ cat out.txt a1 a1
Note that
cat: a1: Permission denied
does not appear because it belongs tostderr
. Thetee
command can read fromstdin
only.By default, the
tee
command overwrites the file, but it can be used with appended options by providing the-a
option, for example,$ cat a* | tee -a out.txt | cat -n
.Commands appear with arguments in the format:
command FILE1 FILE2 …
or simplycommand FILE
.We can use
stdin
as a command argument. It can be done by using-
as the filename argument for the command as follows:$ cmd1 | cmd2 | cmd -
For example:
$ echo who is this | tee - who is this who is this
Alternately, we can use
/dev/stdin
as the output filename to usestdin
.Similarly, use
/dev/stderr
for standard error and/dev/stdout
for standard output. These are special device files that correspond tostdin
,stderr
, andstdout
.
How it works...
For output redirection, >
and >>
operators are different. Both of them redirect text to a file, but the first one empties the file and then writes to it, whereas the later one adds the output to the end of the existing file.
When we use a redirection operator, the output won't print in the terminal but it is directed to a file. When redirection operators are used, by default, they operate on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.
>
is equivalent to 1>
and similarly it applies for >>
(equivalent to 1>>
).
When working with errors, the stderr
output is dumped to the /dev/null
file. ./dev/null
is a special device file where any data received by the file is discarded. The null device is often known as a black hole as all the data that goes into it is lost forever.
There's more...
A command that reads stdin
for input can receive data in multiple ways. Also, it is possible to specify file descriptors of our own using cat
and pipes, for example:
$ cat file | cmd $ cmd1 | cmd
Redirection from a file to a command
By using redirection, we can read data from a file as stdin
as follows:
$ cmd < file
Redirecting from a text block enclosed within a script
Sometimes we need to redirect a block of text (multiple lines of text) as standard input. Consider a particular case where the source text is placed within the shell script. A practical usage example is writing a logfile header data. It can be performed as follows:
#!/bin/bash cat<<EOF>log.txt LOG FILE HEADER This is a test log file Function: System statistics EOF
The lines that appear between cat <<EOF >log.txt
and the next EOF
line will appear as the stdin
data. Print the contents of log.txt
as follows:
$ cat log.txt LOG FILE HEADER This is a test log file Function: System statistics
Custom file descriptors
A file descriptor is an abstract indicator for accessing a file. Each file access is associated with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for stdin
, stdout
, and stderr
.
We can create our own custom file descriptors using the exec
command. If you are already familiar with file programming with any other programming language, you might have noticed modes for opening files. Usually, the following three modes are used:
Read mode
Write with truncate mode
Write with append mode
<
is an operator used to read from the file to stdin
. >
is the operator used to write to a file with truncation (data is written to the target file after truncating the contents). >>
is an operator used to write to a file by appending (data is appended to the existing file contents and the contents of the target file will not be lost). File descriptors can be created with one of the three modes.
Create a file descriptor for reading a file, as follows:
$ exec 3<input.txt # open for reading with descriptor number 3
We could use it in the following way:
$ echo this is a test line > input.txt $ exec 3<input.txt
Now you can use file descriptor 3
with commands. For example, we will use cat <&3
as follows:
$ cat<&3 this is a test line
If a second read is required, we cannot re-use the file descriptor 3
. It is required that we reassign the file descriptor 3
for read using exec
for making a second read.
Create a file descriptor for writing (truncate mode) as follows:
$ exec 4>output.txt # open for writing
For example:
$ exec 4>output.txt $ echo newline >&4 $ cat output.txt newline
Create a file descriptor for writing (append mode) as follows:
$ exec 5>>input.txt
For example:
$ exec 5>>input.txt $ echo appended line >&5 $ cat input.txt newline appended line