1. Introduction
Most shells offer the ability to alter the way that application input and output flows. This can direct output away from the terminal and into files or other applications or otherwise read input from files instead of the terminal.
In this tutorial we’ll discuss the concepts of pipes and redirection in Linux.
2. Standard Input and Output
Before we understand how redirection works in shells, we first need to understand the standard input and output process.
All applications have three unique file descriptors that connect them to the outside world. These are referred to as Standard Input, or stdin; Standard Output, or stdout; and Standard Error, or stderr.
Standard input is the default mechanism for getting input into an interactive program. This is typically a direct link to the keyboard when running directly in a terminal and isn’t connected to anything otherwise.
Standard output is the default mechanism for writing output from a program. This is typically a link to the output terminal but is often buffered for performance reasons. This can be especially important when the program is running over a slow network link.
3. Redirecting Into Files
One common need when we run applications is to direct the output into a file instead of the terminal. Specifically, what we’re doing here is replacing the standard output file descriptor with one that goes where we want: In this case, a file.
This is typically done with the > operator between the application to run and the file to write the output into. For example, we can send the output of the ls command into a file called files as follows:
$ ls > files
$ ls -1 *.txt > text-files
3.1. Appending To Files
When we use >, then we are writing the output to the file, replacing all of the contents. If needed, we can also append to the file using >> instead:
$ ls -1 *.txt > files
$ ls -1 *.text >> files
$ ls -1 *.log >> files
We can use this to build up files using all manner of output if we need to, including just using echo to exact output lines:
$ echo Text Files > files
$ ls -1 *.txt >> files
$ echo Log Files >> files
$ ls -1 *.log >> files
3.2. Redirecting Standard Error
On occasion, we need to redirect standard error instead of standard output. This works in the same way, but we need to specify the exact file descriptor.
All three of the standard file descriptors have ID values, as defined in the POSIX specification and used in the C language. Standard input is 0, standard output is 1, and the standard error is 2.
When we use the redirect operators, by default, this applies to standard output. We can explicitly specify the file descriptor to redirect, though, by prefixing it with the file descriptor ID.
For example, to redirect standard error from the cat command, we would use 2>:
$ cat does-not-exist 2> log
We can be consistent and use 1> to redirect standard output if we wish, though this is identical to what we saw earlier.
We can also use &> to redirect both standard output and standard error at the same time:
$ ls -1 &> log
This sends all output from the command, regardless of which file descriptor it is on, into the same file. This only works in certain shells, though – for example, we can use this in bash or zsh, but not in sh.
3.3. Redirecting into File Descriptors
$ echo Standard Output >&1
$ echo Standard Error >&2
We can combine this with the above to combine file descriptors by redirecting from one file descriptor into another. A common construct is to combine standard error into standard output so that both can be used together. We achieve this using 2>&1 – literally redirecting file descriptor 2 into a file descriptor 1:
# ls -1 2>&1
We can sometimes use this to create new file descriptors simply by using new IDs. This file descriptor must already have been used elsewhere first; otherwise, it is an error. Most often, this is used as a file descriptor source first.
For example, we can swap standard output and standard error by going via a third file descriptor, using 3>&2 2>&1 1>&3:
$ ls -1 3>&2 2>&1 1>&3
This construct doesn’t work correctly in all shells. In bash, the end result is that standard output and standard error are directly swapped. In zsh the result is that both standard output and standard error have ended up on standard output instead.
3.4. Redirecting Multiple File Descriptors
We can easily combine the above to redirect standard output and standard error at the same time. This mostly works exactly as we would expect – we simply combine the two different redirects on the same command:
$ ls -1 > stdout.log 2> stderr.log
$ ls -1 > log 2>&1
4. Reading From Files
Sometimes we also want to achieve the opposite – redirecting a file into an application. We do this using the < operator, and the contents of the file will replace the standard input for the application:
$ wc < /usr/share/dict/words
235886 235886 2493109
When we do this, the only input that the application can receive comes from this source, and it will all happen immediately. It’s effectively the same as when the user types out the entire contents of the file at the very start of the application.
However, the end of the file is signaled to the application as well, which many applications can use to stop processing.
5. Piping Between Applications
The final action that we can perform is to direct the output of one application into another one. This is commonly referred to as piping and uses the | operator instead:
$ ls | wc
11 11 138
This directly connects the standard output of our first application to the standard input of the second one and then lets the data flow directly between them.
5.1. Handling Standard Error
The standard error isn’t connected by default, so we’ll still get anything written to that file descriptor appearing in the console. This is by design since the standard error is designed for error reporting and not for normal application output.
If we want to redirect standard error as well, then we can use the technique from above to first redirect it into standard output and then pipe into the next application:
$ ls i-dont-exist | wc
ls: i-dont-exist: No such file or directory
0 0 0
$ ls i-dont-exist 2>&1 | wc
1 7 44
If we want only to pipe standard error, then we need to swap the standard output and standard error file descriptors around, as we saw earlier.
$ ls 3>&2 2>&1 1>&3 | wc -l
some-file
0
$ ls i-dont-exist 3>&2 2>&1 1>&3 | wc -l
1
5.2. Combining Pipes
When piping between applications, we can also build arbitrary chains where we are piping between many applications to achieve a result:
$ docker images | cut -d' ' -f1 | tail -n +2 | sort -u | wc -l
17
- docker images – Get a list of all Docker images
- cut -d’ ‘ -f1 – Cut this output to only return the first column, where columns are space-separated
- tail -n +2 – Limit this to start from line 2
- sort -u – Sort this list, only returning unique entries
- wc -l – Count the number of lines
So we have a command here to get the number of unique docker images, ignoring the version of the image.
Many console applications are designed for exactly this use, which is why they can often consume input from standard input and write to standard output.
Certain applications also have special modes that allow for this – git, for example, has what is termed porcelain and plumbing commands, where the plumbing commands are specially designed to be combined in this manner while the porcelain commands are designed for human consumption.
5.3. Named Pipes
Another significant concept when dealing with IO in Linux is the concept of named pipes. Named pipes are a form of inter-process communication that allows data to flow between two processes, but unlike unnamed pipes, they persist in the filesystem beyond the life of the process and can be accessed as a file. This functionality can be incredibly useful when setting up consistent communication channels between different parts of your system.
6. Summary
Here, we’ve seen several techniques for redirecting the input and output of applications either between files or other applications. These are compelling techniques that can give rise to complicated results from simple inputs.
Why not see what else we can do with these?