1. Introduction
Whenever we work on the Linux command line, we often have to pass data from one command to another, like feeding a list of find results into a grep. This is where streams come into play.
In this tutorial, we’ll take a look at what streams are and how we work with them.
2. What Are Streams?
We can think of a stream in its simplest form as a pipe that carries data – especially character data – from one point to another.
Some examples of input streams are the keyboard, text data stored in files and input from I/O devices. We can deliver the output to files, commands, windows, and other I/O devices.
3. Sample Data
Let’s create some sample files to use in the subsequent sections:
$ echo -e "tables\nladders\nchairs" > streamdata1
$ echo -e "planes\ntrains\nautomobiles" > streamdata2
We’ve created the streamdata1 and streamdata2 files and populated them with some data:
$ cat streamdata1
tables
ladders
chairs
$ cat streamdata2
planes
trains
automobiles
4. Redirecting Input
The first stream that we’re going to look at is STDIN.
STDIN refers to the standard input stream; usually, input from the keyboard. STDIN has a filehandle of 0 (zero).
The < operator is used to pass input from a file or I/O device data to a command.
Let’s say that we want to count the number of lines in a file without passing the file name through as a parameter to the wc command. We can do it by redirecting STDIN:
$ wc -l < streamdata1
5. Redirecting Output
Next, let’s take a look at STDOUT, the standard output stream. All output from this stream is typically sent to our terminal window. The filehandle for STDOUT is 1.
The > operator is used to direct output from a command to a source; usually a file. By default, a single > will:
- Create a new file if it doesn’t already exist
- Overwrite any pre-existing data in the file if the file already exists
Let’s see how we can use the cat command to emit the contents of streamdata1 and send the output to a new file:
$ cat streamdata1 > combinedstreamdata
When we print the contents of the combinestreamdata file it should look exactly like streamdata1:
$ cat combinedstreamdata
tables
ladders
chairs
6. Appending to an Existing File
While > overwrites data, the >> operator preserves data by appending to an existing file.
Let’s see how we can add the contents of streamdata2 to the combinedstreamdata file:
$ cat streamdata2 >> combinedstreamdata
The combinedstreamdata file now contains the contents of both our streamdata1 and streamdata2 files:
$ cat combinedstreamdata
tables
ladders
chairs
planes
trains
automobiles
7. Piping Output to Input
Chaining together multiple tasks is a common use case when working with Linux commands.
With the | (pipe) operator, we can chain many commands together by passing the output from one through as input to the next.
Let’s try using the | operator to stream the output from the cat command to the input stream of the wc command:
$ cat streamdata2 | wc -l
8. Redirecting Error
Now that we’ve got the basics of stream redirection down, let’s look at how we can work with multiple output files.
Let’s attempt to execute a script that does not exist and pipe its imaginary output to a log file:
$ exec doesnotexist.sh > out.log
We get this error message:
exec: doesnotexist.sh: not found
Let’s take a look and see what our command wrote to out.log:
$ cat out.log
Hmm, our log file is empty. But, we did see an error message – we might want to log that, too.
Let’s see how we can redirect STDOUT and STDERR to capture the output and error output:
$ exec doesnotexist.sh >out.log 2>err.log
In the above statement, we direct standard output to out.log and standard error to err.log.
More specifically, we referenced the standard error stream using its filehandle – 2>err.log. We didn’t have to specify the filehandle for standard output because its filehandle is the default.
Let’s check to see what the command wrote to err.log:
$ cat err.log
exec: doesnotexist.sh: not found
Our error message was successfully redirected to our error log file.
In this example, we handled both output streams (STDOUT, STDERR) and directed each to its own log file.
9. Merging Output and Error
Although we can direct STDOUT and STDERR to their own log files, we often prefer the simplicity of having a single log file to deal with.
The >& operator is a special operator that is used for directing output from one stream to another. We can use it to pipe the output of STDERR to STDOUT:
Let’s see how we can leverage file handles and >& to give us a single log file that contains the output of STDOUT and STDERR:
$ cat streamdata1 streamdata2 streamdata3 2>&1>out.log
$ cat out.log
tables
ladders
chairs
planes
trains
automobiles
cat: streamdata3: No such file or directory
As expected, the contents of streamdata1 and streamdata2 are found in out.log along with the anticipated error message since streamdata3 does, in fact, not exist.
10. Conclusion
In this tutorial, we looked at what Linux streams are and saw how to use them.
We worked through a few scenarios that demonstrated the different capabilities of stream redirection.