File descriptors are integers associated with the input and output streams. The best-known file descriptors are stdin, stdout, and stderr. The contents of one stream can be redirected to another. This recipe shows examples on how to manipulate and redirect with file descriptors.

Shell scripts frequently use standard input (stdin), standard output (stdout), and standard error (stderr). A script can redirect output to a file with the greater-than symbol. Text generated by a command may be normal output or an error message. By default, both normal output (stdout) and error messages (stderr) are sent to the display. The two streams can be separated by specifying a specific descriptor for each stream.

File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved, as given here:

• 0: stdin
• 1: stdout
• 2: stderr

1. Use the greater-than symbol to append text to a file:

        $ echo "This is a sample text 1" > temp.txt

This stores the echoed text in temp.txt. If temp.txt already exists, the single greater-than sign will delete any previous contents.

2. Use double-greater-than to append text to a file:

        $ echo "This is sample text 2" >> temp.txt

3. Use cat to view the contents of the file:

        $ cat temp.txt
        This is sample text 1
        This is sample text 2

The next recipes demonstrate redirecting stderr. A message is printed to the stderr stream when a command generates an error message. Consider the following example:

$ ls +
ls: cannot access +: No such file or directory

Here + is an invalid argument and hence an error is returned.

The following command prints the stderr text to the screen rather than to a file (and because there is no stdout output, out.txt will be empty):

$ ls + > out.txt
ls: cannot access +: No such file or directory 

In the following command, we redirect stderr to out.txt with 2> (two greater-than):

$ ls + 2> out.txt # works

You can redirect stderr to one file and stdout to another file.

$ cmd 2>stderr.txt 1>stdout.txt

It is also possible to redirect stderr and stdout to a single file by converting stderr to stdout using this preferred method:

$ cmd 2>&1 allOutput.txt

This can be done even using an alternate approach:

$ cmd &> output.txt 

If you don't want to see or save any error messages, you can redirect the stderr output to /dev/null, which removes it completely. For example, consider that we have three files a1, a2, and a3. However, a1 does not have the read-write-execute permission for the user. To print the contents of all files starting with the letter a, we use the cat command. Set up the test files as follows:

$ echo A1 > a1
$ echo A2 > a2
$ echo A3 > a3
$ chmod 000 a1  #Deny all permissions

Displaying the contents of the files using wildcards (a*), will generate an error message for the a1 file because that file does not have the proper read permission:

$ cat a*
cat: a1: Permission denied
A2
A3

Here, cat: a1: Permission denied belongs to the stderr data. We can redirect the stderr data into a file, while sending stdout to the terminal.

$ cat a* 2> err.txt #stderr is redirected to err.txt
A2
A3

$ cat err.txt
cat: a1: Permission denied

Some commands generate output that we want to process and also save for future reference or other processing. The stdout stream is a single stream that we can redirect to a file or pipe to another program. You might think there is no way for us to have our cake and eat it too.

However, there is a way to redirect data to a file, while providing a copy of redirected data as stdin to the next command in a pipe. The tee command reads from stdin and redirects the input data to stdout and one or more files.

command | tee FILE1 FILE2 | otherCommand

In the following code, the stdin data is received by the tee command. It writes a copy of stdout to the out.txt file and sends another copy as stdin for the next command. The cat -n command puts a line number for each line received from stdin and writes it into stdout:

$ cat a* | tee out.txt | cat -n
cat: a1: Permission denied
     1 A2
     2 A3

Use cat to examine the contents of out.txt:

$ cat out.txt
A2
A3

By default, the tee command overwrites the file. Including the -a option will force it to append the new data.

$ cat a* | tee -a out.txt | cat -n

Commands with arguments follow the format: command FILE1 FILE2 ... or simply command FILE.

To send two copies of the input to stdout, use - for the filename argument:

$ cmd1 | cmd2 | cmd -

Consider this example:

$ echo who is this | tee -
who is this
who is this

Alternately, we can use /dev/stdin as the output filename to use stdin.
Similarly, use /dev/stderr for standard error and /dev/stdout for standard output. These are special device files that correspond to stdin, stderr, and stdout.

The redirection operators (> and >>) send output to a file instead of the terminal. The > and >> operators behave slightly differently. Both redirect output to a file, but the single greater-than symbol (>) empties the file and then writes to it, whereas the double greater-than symbol (>>) adds the output to the end of the existing file.

By default, the redirection operates on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.

The > operator is equivalent to 1> and similarly it applies for >> (equivalent to 1>>).

When working with errors, the stderr output is dumped to the /dev/null file. The ./dev/null file is a special device file where any data received by the file is discarded. The null device is often known as a black hole, as all the data that goes into it is lost forever.

Commands that read input from stdin can receive data in multiple ways. It is possible to specify file descriptors of our own, using cat and pipes. Consider this example:

$ cat file | cmd
$ cmd1 | cmd2

We can read data from a file as stdin with the less-than symbol (<):

$ cmd < file

Text can be redirected from a script into a file. To add a warning to the top of an automatically generated file, use the following code:

#!/bin/bash 
cat<<EOF>log.txt 
This is a generated file. Do not edit. Changes will be overwritten. 
EOF

The lines that appear between cat <<EOF >log.txt and the next EOF line will appear as the stdin data. The contents of log.txt are shown here:

$ cat log.txt 
This is a generated file. Do not edit. Changes will be overwritten. 

A file descriptor is an abstract indicator for accessing a file. Each file access is associated with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for stdin, stdout, and stderr.

The exec command can create new file descriptors. If you are familiar with file access in other programming languages, you may be familiar with the modes for opening files. These three modes are commonly used:

• Read mode
• Write with append mode
• Write with truncate mode

The < operator reads from the file to stdin. The > operator writes to a file with truncation (data is written to the target file after truncating the contents). The >> operator writes to a file by appending (data is appended to the existing file contents and the contents of the target file will not be lost). File descriptors are created with one of the three modes.

Create a file descriptor for reading a file:

$ exec 3<input.txt # open for reading with descriptor number 3

We can use it in the following way:

$ echo this is a test line > input.txt
$ exec 3<input.txt

Now you can use file descriptor 3 with commands. For example, we will use cat<&3:

$ cat<&3
this is a test line

If a second read is required, we cannot reuse the file descriptor 3. We must create a new file descriptor (perhaps 4) with exec to read from another file or re-read from the first file.

Create a file descriptor for writing (truncate mode):

$ exec 4>output.txt # open for writing

Consider this example:

$ exec 4>output.txt
$ echo newline >&4
$ cat output.txt
newline

Now create a file descriptor for writing (append mode):

$ exec 5>>input.txt

Consider the following example:

$ exec 5>>input.txt
$ echo appended line >&5
$ cat input.txt
newline
appended line