1. Overview
When we work in the Linux command line, we can use the MY_CMD | wc -l command to count the lines of MY_CMD‘s output. However, if we want to see MY_CMD‘s output, we need to execute the command once again.
In this tutorial, we’ll explore how to get a command’s output and count lines in one shot.
2. Introduction to the Problem
As usual, let’s understand the problem by an example. Let’s simulate the command using a simple shell script tech.sh:
$ cat ./tech.sh
#!/bin/bash
cat <<EOF
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
EOF
As the cat output above shows, the shell script merely outputs a few lines using here document. Therefore, if we execute the script, we see the output:
$ ./tech.sh
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
If we want to know how many lines the output is, of course, we can pipe the script’s result to wc -l:
$ ./tech.sh | wc -l
9
As we can see, the output tells us the total number of lines: 9. If we want to view tech.sh‘s output again, we need to rerun the command.
However, in some circumstances, the commands may produce dynamic output. Then, the number we counted in a separate process (wc -l) might be incorrect.
Therefore, our goal is to *get tech.sh‘s output together with the line number in one shot*.
So next, let’s see how it gets done.
3. Using the tee Command
First, let’s solve the problem using the tee command.
The tee command can pipe input to multiple processes. Therefore, we can pipe tech.sh‘s output to /dev/stderr and the wc -l:
$ ./tech.sh | tee /dev/stderr | wc -l
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
9
As the output above shows, we’ve got the command’s output and the line count.
Curious eyes may spot that *the tee command copies the tech.sh‘s output to /dev/stderr instead of /dev/stdout*. This is because ./tech.sh | tee /dev/stdout will feed stdout twice with the same data. One time is from the script’s output. The other is from the tee command.
Further, tee pipes the data to processes in turn. So, the later “*| wc -l* ” consumes all data in the stdout. Therefore, we will only get a doubled line count without the script’s output:
$ ./tech.sh | tee /dev/stdout | wc -l
18
Humans can read the data outputted to stderr since they get printed on the terminal anyway. However, data in stderr won’t be passed to the next program through pipes. For instance, if we want to pipe the output further to the grep command, grep won’t read stderr:
$ ./tech.sh | tee /dev/stderr | wc -l | grep 'Kotlin'
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
$ echo $?
1
As the output above shows, grep ‘Kotlin’ didn’t filter the lines matching the given pattern. Also, the later “*echo $?*“‘ command’s output (1) indicates the grep command didn’t find any matches.
We can use tee together with process substitution to feed the data received by tee to different processes:
$ ./tech.sh | tee >(wc -l)
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
9
This time, if we pipe the result to the grep command, it works as expected. This is because all outputs are written to stdout:
$ ./tech.sh | tee >(wc -l) | grep 'Kotlin'
Kotlin is awesome!
4. Using the sed Command
We’ve seen how to solve the problem using the tee command. Alternatively, we can pipe tech.sh‘s output to the sed command, and ask sed to print the output and count lines.
Similar to “wc -l“, sed has provided the “*$=*” command to count lines:
$ ./tech.sh | sed -n 'p;$='
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
9
Some of us may wonder why we’ve used sed -n ‘p;$=’ instead of sed ‘$=’ in the example above, as sed will print all lines by default. This is because when the last line is read, “*$=*” counts the lines and outputs the total line number. After that, sed performs the default action: printing the last line:
$ ./tech.sh | sed '$='
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
9
- Linux kernel is open source software.
As we can see, sed prints the number 9 before the last line. This can be confusing if the last line in the original output is also a number. Therefore, to avoid that, we must tell sed to execute “p” before “*$=*“.
5. Using the awk Command
The awk command is another powerful weapon when we need to process text.
Adding a total line number at the end of the output using awk is an easy task. We can print the current NR value in the END{…} block to achieve that:
$ ./tech.sh | awk '1; END{print "The output has " NR " lines."}'
Programming languages:
- Java is powerful.
- Python is nice!
- Kotlin is awesome!
Operating systems:
- MacOS is a product of Apple.
- Windows is a product of Microsoft.
- Linux kernel is open source software.
The output has 9 lines
It’s worth mentioning that the awk approach allows us to customize the number’s output format easily. For example, the command above prints “The output has x lines” at the very end of the output.
6. Conclusion
In this article, we’ve learned three ways to print a command’s output and the total number of lines in one shot.
First, we’ve seen how to use the tee command to solve the problem and control the data stream (stderr and stdout) where the output goes. Then, we learned how to solve the problem using sed and awk straightforwardly.