1. Overview
When working with the Linux command-line interface, it’s common to redirect the program’s output to be used as the input to another program.
In this tutorial, we’ll look at using pipes and named pipes in Linux.
2. What Is a Pipe?
A pipe is an important mechanism in Unix-based systems that allows us to communicate data from one process to another without storing anything on the disk.
In Linux, we have two types of pipes: pipes (also known as anonymous or unnamed pipes) and FIFO’s (also known as named pipes).
3. Pipes
Pipes are used by stringing together commands, separated by the pipe character, ‘*|*‘. This is often referred to as a pipeline, and each shell defines its behavior.
The shell executes each command in a separate process running in the background, starting with the far left command.
Then, the standard output of the command on the left side is connected to the command’s standard input on the right side. This offers a unidirectionality of the stream.
This mechanism lasts until all of the processes in the pipeline have been completed.
Shells like Bash and Zsh use the token “*|&*” to refer to a pipeline, connecting both the standard output and the standard error of the command on the left side with the standard input of the command on the right side.
Let’s say we want to use the netstat command to see what processes are running using the localhost and filter with the grep utility:
$ netstat -tlpn | grep 127.0.0.1
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -
In this example output, we see both the stdout and stderr of our script in netstat regardless of the filter.
Now, let’s merge the stderr into the stdout and pass it to the stdin of grep:
$ netstat -tlpn |& grep 127.0.0.1
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -
And we’ve suppressed it the warning message.
3.1. Pipelines in Bash
Bash has a variable called PIPESTATUS, which contains a list of exit status from the processes in the most recently executed pipeline:
$ exit 1 | exit 2 | exit 3 | exit 4 | exit 5
$ echo ${PIPESTATUS[@]}
1 2 3 4 5
The return status of the execution of the whole pipeline will depend on the pipefail variable’s status.
If this variable is set, the return status of the pipeline will be the exit status of the rightmost command with a non-zero status or will be zero if all commands exit successfully:
$ set -o pipefail
$ exit 1 | exit 2 | exit 3| exit 4 | exit 0
$ echo $?
4
With the pipefail option disabled, the return status of the pipe will be the exit status of the last command:
$ set +o pipefail
$ exit 1 | exit 2 | exit 3| exit 4 | exit 0
$ echo $?
0
Bash also has the lastpipe option that instructs the shell to execute the last command in the foreground of the current environment.
3.2. Pipelines in Zsh
Zsh has similar control over pipelines as Bash but with a few differences. For example, Zsh has the pipestatus command, which is similar to the PIPESTATUS variable in Bash.
Additionally, Zsh executes the commands in each pipeline in separate processes, except for the last command, which is executed in the current shell environment.
4. Named Pipes
A FIFO, also known as a named pipe, is a special file similar to a pipe but with a name on the filesystem. Multiple processes can access this special file for reading and writing like any ordinary file.
Thus, the name works only as a reference point for processes that need to use a name in the filesystem.
A FIFO has the same characteristics as any other file. For example, it has ownership, permissions, and metadata.
Another important feature that a FIFO has is that it provides bidirectional communication.
In Linux, we can create a FIFO with the commands mknod (using the letter “p” to indicate the FIFO type) and mkfifo:
$ mkfifo pipe1
$ mknod pipe2 p
$ ls -l
prw-r--r-- 1 cuau cuau 0 Oct 7 21:17 pipe1
prw-r--r-- 1 cuau cuau 0 Oct 7 21:17 pipe2
Here, we can see that our FIFO’s file type is indicated with the letter “p”.
This mechanism allows us to create more complex applications using our shell.
Named and anonymous pipes can be used together. Let’s create a reverse shell combining both FIFOs and pipes.
We’ll use the nc utility to create a client/server application, in which the “server” side will provide its shell, and the “client” side will be able to access it.
First, let’s install the netcat-openbsd package*.* We can install it on any Ubuntu/Debian system using:
$ sudo apt install netcat-openbsd
Next, let’s create a FIFO called fifo_reverse typing mkfifo fifo_reverse.
Then, let’s log in with two different users that will each act as a “client” (let’s say, “user1”) and as a “server” (let’s say, “user2”). Let’s run this pipeline on the user2 shell:
user2_prompt$ bash -i < fifo_reverse |& nc -l 127.0.0.1 1234 > fifo_reverse
In this one-liner, the shell reads the content of our FIFO and passes it to an interactive Bash shell.
Next, both the stdout and the stderr of the interactive shell will be passed to the nc command, which will be listening at port 1234 of address 127.0.0.1.
And finally, when the “client” establishes a connection successfully, nc will write what is received to our FIFO, and the interactive shell will be able to execute what is received.
Now, using the user1 shell, let’s type:
user1_prompt$ nc 127.0.0.1 1234
user2_prompt$
And we’ve obtained the user2 prompt but using the user1 shell combining anonymous and named pipes.
5. Temporary Named Pipes
Some shells have a feature called process substitution which connects the input or output of a list of commands to a FIFO. The commands will then use the name of this FIFO.
The notation for this mechanism in Bash and Zsh is <(command list)* to pass the result of the list to the standard input of the actual command, or *>(command list) to pass the standard output of the actual command to the standard input of the list.
Let’s use what we’ve seen to pass the output of multiple commands to the wc command:
$ wc -l \
<(find / -mindepth 1 -maxdepth 1 -type d) \
<(find /opt -mindepth 1 -maxdepth 1 -type d)
20 /proc/self/fd/11
2 /proc/self/fd/12
22 total
In this example output, we use the find command to get the number of directories inside the / and /opt directories.
6. When to Use Named or Anonymous Pipes?
Using an anonymous pipe instead of a named pipe depends on the characteristics we’re looking for. Some of them can be persistence, two-way communication, having a filename, creating a filter, and restricting access permissions among others.
For example, if we want to filter the output of a command multiple times, using an anonymous pipe seems the most appropriate option. Let’s also remember that the shell we’re using will play a central role when we’re working with anonymous pipes.
On the other hand, if we need a filename and we don’t want to store data on disk, what we’re looking for is a FIFO. If we only need a name as a reference, with content that comes directly from another process.
Also, let’s consider that while an anonymous pipe can look like a plumbing type pipe, a FIFO can create more complex diagrams.
7. Conclusion
Pipes and FIFOs are very useful when working on the Linux command line. In this tutorial, we’ve reviewed some features of both and discussed when to use one instead of the other.