1. Overview
When we work with Linux command-lines, we often pipe the output of one command to another command.
However, sometimes, we may face the problem of sending the output of a command to multiple commands. In this tutorial, we’ll discuss how to solve this problem.
2. Introduction to the Problem
First of all, let’s see a concrete example so that we can understand the problem easier.
Let’s say we have an input file scores.txt, which is holding the names of students and their scores of an exam:
$ cat scores.txt
Mark 98.3
Kent 99.7
Amanda 88.8
Eric 95
John 70
Timo 42
Jerry 93
Further, we have created three shell script files:
- avg.sh – Calculate the average score among all students and save to a file: avg.result
- max.sh – Save the highest score with the student name to a file: max.result
- min.sh – Save the lowest score with the student name to a file: min.result
Our problem is sending the output of the command cat scores.txt to the three scripts above to get the average, highest, and lowest scores.
Now, let’s take a closer look at the scripts. Each script contains a short awk command to do the calculation and then save the result in a file:
$ head *.sh
==> avg.sh <==
#!/bin/bash
if [ -p /dev/stdin ]; then
awk '{ sum+=$2 } END{ printf "The average score: %.2f\n", sum/NR }' /dev/stdin > avg.result
else
echo "Error Occured: No input was found on stdin!"
fi
==> max.sh <==
#!/bin/bash
if [ -p /dev/stdin ]; then
awk 'max < $2{ max=$2; max_row=$0 } \
END{ printf "The highest score: %s\n", max_row }' /dev/stdin > max.result
else
echo "Error Occured: No input was found on stdin!"
fi
==> min.sh <==
#!/bin/bash
if [ -p /dev/stdin ]; then
awk 'NR==1 || $2 < min{ min=$2; min_row=$0 } \
END{ printf "The lowest score: %s\n", min_row }' /dev/stdin > min.result
else
echo "Error Occured: No input was found on stdin!"
fi
The three scripts above are pretty straightforward. However, a couple of points are worthwhile to mention:
- All scripts will only read input from stdin (/dev/stdin)
- We need the [ -p /dev/stdin ] checking in each script. Otherwise, it will hang until something is typed if we run a script without input on stdin
In this tutorial, we’ll address how to solve the problem in three different ways:
- First, using the tee command and process substitution
- Next, using the tee command and named pipes
- Finally, using the tee command and file descriptors
3. Using the tee Command and Process Substitution
Using the tee command and process substitutions, we can feed stdin directly into processes:
$ tee >(process1) >(process2) >(process3)....
Therefore, our problem can be solved in this way:
$ cat scores.txt | tee >(./min.sh) >(./max.sh) | ./avg.sh
$ head *.result
==> avg.result <==
The average score: 82.30
==> max.result <==
The highest score: Kent 99.7
==> min.result <==
The lowest score: Timo 42
This solution is pretty straightforward. However, not all shells support the process substitution feature. We can use the process substitution if our shell is Bash, Zsh, or Ksh93.
Next, let’s take a look at some more portable solutions to the problem.
4. Using the tee Command and Named Pipes
Using named pipes is a portable method to solve the problem since it is supported by all *nix systems.
It’s worth spending a few minutes to understand what a named pipe is and how to use it.
4.1. Named Pipe in a Nutshell
We are familiar with the unnamed pipe already, and we use it quite often in the Linux command-line.
The unnamed pipe is a handy technique to allow us to transfer data between different processes, such as command1 | command2. However, the unnamed pipe exists only in the kernel and cannot be accessed by processes other than the command2.
A named pipe is similar to an unnamed pipe. Also, it exists in the filesystem and can be opened by multiple processes for reading and writing.
Setting up and using a named pipe is convenient. Let’s have a look at the standard operations of using a named pipe:
- mkfifo myPipe – Create a named pipe with the name of “myPipe“. A special file myPipe will be created
- command1 > myPipe – Redirect the output of command1 to the named pipe myPipe
- command2 < myPipe – The command2 reads data from myPipe as the input
- rm myPipe – Close the named pipe, just as deleting a regular file
4.2. A Little Example Using Named Pipes
After understanding what a named pipe is, let’s show how to use named pipes through an example:
Firstly, let’s create a named pipe using the mkfifo command:
$ mkfifo myPipe
$ ls -l myPipe
prw-r--r-- 1 kent kent 0 Jul 12 16:38 myPipe
In the output of the ls command above, the p in the most left column indicates the file myPipe is a named pipe.
After that, we’ll tell a short awk command to read data from the named pipe we just created:
$ awk '$0 = NR ":" $0' > withLineNumber.txt < myPipe &
[1] 467439
The awk command will read the data from the named pipe. Further, it adds a line number as the prefix on each line and saves the result in a file called withLineNumber.txt.
We may notice the command is ending with a & character. This is because we still want to type other commands in the current shell, and the & operator allows us to run the awk command in the background.
Now, let’s feed the named pipe using our scores.txt file:
$ cat scores.txt > myPipe
After the data has been written to the named pipe, the background job is done:
$ jobs
[1]+ Done awk '$0 = NR ":" $0' > withLineNumber.txt < myPipe
Next, let’s check if the withLineNumber.txt has been created and filled with expected data:
$ cat withLineNumber.txt
1:Mark 98.3
2:Kent 99.7
3:Amanda 88.8
4:Eric 95
5:John 70
6:Timo 42
7:Jerry 93
Finally, we should close the named pipe using the rm command:
$ rm myPipe
4.3. Solve the Problem Using tee and Named Pipes
Now, let’s have a look at how to solve our problem using the tee command and named pipes:
$ mkfifo myPipe1 myPipe2
$ ./min.sh < myPipe1 &
[1] 482525
$ ./max.sh < myPipe2 &
[2] 482657
$ cat scores.txt | tee myPipe1 myPipe2 |./avg.sh
Further, let’s check if the result files were created:
$ head *.result
==> avg.result <==
The average score: 82.30
==> max.result <==
The highest score: Kent 99.7
==> min.result <==
The lowest score: Timo 42
Great! The result files were generated.
Finally, we shouldn’t forget closing the pipes using the rm command:
$ rm myPipe1 myPipe2
In this solution, the tee command helps us to redirect the content of the scores.txt file to both named pipes. It won’t be a problem to understand the solution if we know how a named pipe works.
5. Using the tee Command and File Descriptors
As we can use multiple file descriptors in any POSIX shell, the solution using file descriptors is also portable.
Firstly, Let’s start by looking at how to solve the problem using file descriptors:
$ { { cat scores.txt| tee /dev/fd/3 /dev/fd/4 | ./avg.sh \
} 3>&1 | ./min.sh \
} 4>&1 | ./max.sh \
After we execute the command above, let’s check the result files to see if they contain the expected results:
$ head *.result
==> avg.result <==
The average score: 82.30
==> max.result <==
The highest score: Kent 99.7
==> min.result <==
The lowest score: Timo 42
Now, let’s understand how does the solution work:
- Curly braces {…} – We use curly braces to wrap commands to run those commands in the current shell instead of subshells
- tee /dev/fd/3 /dev/fd/4 – The tee command redirects the file content to two file descriptors (FD) 3 and 4
- tee …| ./avg.sh; – Also, the tee command pipes the file content to the avg.sh script
- {cat.. | tee..} 3>&1 | ./min.sh; – We redirect FD 3 to FD 1, which is the stdout. Then we pipe the stdout to the min.sh as the input
- {…} 4>&1 | ./max.sh; – Similarly, we redirect FD 4 to stdout and then pipe to the max.sh. The max.sh script will take this as input and find the highest score
6. Conclusion
In this article, we solved the problem of sending the output of a command to multiple commands. And we’ve discussed three different solutions through examples.
All three methods use the tee command in the middle to redirect the output of the previous command.
Firstly, we’ve addressed a way of using tee and process substitutions. This is a straightforward solution if our shell supports the feature.
Otherwise, if we are looking for more portable solutions, we can consider using tee with named pipes or file descriptors.