1. Introduction
In Bash, we often run commands with a long execution time and continuous output. At times, we’d like to capture the output of a long-running command line-by-line and then do something with each line. In this tutorial, we’ll see how to accomplish that.
2. The Long Running Command
For this tutorial, we’ll be using the ping command as the long-running command for the sake of demonstration. The ping command checks if a remote computer is reachable at regular intervals and prints the output continuously.
$ ping -c 10 baeldung.com
PING baeldung.com (172.66.43.8) 56(84) bytes of data.
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=1 ttl=57 time=23.4 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=2 ttl=57 time=29.8 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=3 ttl=57 time=42.6 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=4 ttl=57 time=24.3 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=5 ttl=57 time=26.2 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=6 ttl=57 time=56.8 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=7 ttl=57 time=27.3 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=8 ttl=57 time=36.2 ms
64 bytes from 172.66.43.8 (172.66.43.8): icmp_seq=9 ttl=57 time=33.0 ms
--- baeldung.com ping statistics ---
10 packets transmitted, 9 received, 10% packet loss, time 9012ms
rtt min/avg/max/mdev = 23.446/33.294/56.810/10.132 ms
We used the -c argument to ping baeldung.com 10 times, or the command will keep pinging indefinitely.
3. Basic Setup
The most basic way to capture the output of a command line-by-line is as follows:
$ ping -c 10 baeldung.com |
while IFS= read -r line
do
echo $(date +%H:%m:%S) $line
done
In the above snippet, we piped the output into a while loop and used a read command on stdout, which essentially reads the output byte-by-byte and splits it into chunks based on the IFS variable we’ve set. Then, it stores the chunk in the line variable, which we process inside the loop. For the sake of demonstration, we are reprinting the line with a timestamp at the start of the line. We’ll see the following output when we run the above snippet:
11:09:34 PING baeldung.com (172.66.40.248) 56(84) bytes of data.
11:09:34 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=1 ttl=57 time=20.2 ms
11:09:35 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=2 ttl=57 time=21.3 ms
...
11:09:43 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=10 ttl=57 time=17.7 ms
11:09:43
11:09:43 --- baeldung.com ping statistics ---
11:09:43 10 packets transmitted, 10 received, 0% packet loss, time 9003ms
11:09:43 rtt min/avg/max/mdev = 17.747/26.898/46.550/9.512 ms
We observe that our defined timestamp precedes every line in the output.
4. Accessing Variables Outside the Loop
One problem with the above approach is that any variables we set inside the loop are removed from memory when the loop is completed. This happens because the loop runs in a separate subshell, and that shell finishes when the loop is done. To define and access variables from outside the loop, say a counter that counts the number of lines, we need to use parentheses:
$ ping -c 10 baeldung.com | {
i=0
while IFS= read -r line
do
i=$((i+1))
echo $i $line
done
echo "TOTAL LINES: $i"
}
This will give us the following output:
1 PING baeldung.com (172.66.40.248) 56(84) bytes of data.
2 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=1 ttl=57 time=27.3 ms
3 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=2 ttl=57 time=18.9 ms
...
9 64 bytes from 172.66.40.248 (172.66.40.248): icmp_seq=8 ttl=57 time=20.4 ms
10
11 --- baeldung.com ping statistics ---
12 10 packets transmitted, 8 received, 20% packet loss, time 9032ms
13 rtt min/avg/max/mdev = 18.872/25.420/49.099/9.629 ms
TOTAL LINES: 13
We see that we are able to define and access the variable i outside as well as inside the loop.
5. Conclusion
In this article, we looked at how we can capture and process the output of a long-running command line-by-line. We can use these methods in various scenarios where we have to log, transform, or send the output of commands to another service.