1. Overview
When we redirect the output from an application through a pipe, the output sometimes appears all at once, rather than line by line. This happens because there is a buffer in the pipe, or specifically in the standard streams.
We might want to get the output right away, instead of waiting for the buffer to be filled.
In this tutorial, we’ll look at how to use stdbuf to configure the buffer size in the standard streams.
2. Problem
Before we explore the usage of stdbuf, let’s look at an example.
Let’s create hello_baeldung.c and add the following content to it:
#include <stdio.h>
#include <unistd.h>
int main()
{
printf("Hello Baeldung\n");
sleep(1);
printf("Hello Baeldung\n");
sleep(1);
printf("Error happened!\n");
sleep(1);
printf("Hello Baeldung\n");
return 0;
}
Now, let’s compile this simple program written in C. On Ubuntu, to install the compiler, we must install build-essentials:
$ sudo apt install build-essentials
On Fedora, we can do this instead:
$ sudo dnf install gcc-c++
After installing the compiler package and writing the program, we should compile it:
$ gcc hello_baeldung.c -o hello_baeldung
The command above compiles hello_baeldung.c to an executable named hello_baeldung. Let’s run the program:
$ ./hello_baeldung
Hello Baeldung
<pause for 1 second>
Hello Baeldung
<pause for 1 second>
Error happened!
<pause for 1 second>
Hello Baeldung
We’ll see that there was a short pause between every line of output because we called sleep() in the program.
Now, to illustrate the problem of buffering in a pipe, let’s run it through a pipe:
$ ./hello_baeldung | cat
Hello Baeldung
Hello Baeldung
Error happened!
Hello Baeldung
The cat command above just displays the text as it is.
Nothing happened until three seconds elapsed. Then we got the full output in one go.
This shows that there’s a buffer in the pipe. That’s why we couldn’t get the output line by line.
Let’s say that in this example it’s crucial to get the output as fast as possible because there’s an error in the third line. We want to discover it immediately instead of waiting until the buffer is filled. We’ll see how to do this next.
3. General Usage of stdbuf
stdbuf can manipulate buffers in the standard input, standard output, and standard error streams. So let’s use stdbuf to change the buffer in the standard output stream buffering.
3.1. Standard Output Stream Buffering
We want a line buffer by line buffer, or even no buffers at all. So let’s execute stdbuf with the -o option:
$ stdbuf -oL ./hello_baeldung | cat
Hello Baeldung
<pause for 1 second>
Hello Baeldung
<pause for 1 second>
Error happened!
<pause for 1 second>
Hello Baeldung
We got the output line by line every second, just like the original command.
The -o option receives the L argument which means we use the line buffered setting. After the option, we passed the program as the argument. In this case, it’s hello_baeldung. So stdbuf manipulated the buffer of the standard output of hello_baeldung.
We could achieve the same thing with the 0 value. It means no buffering:
$ stdbuf -o0 ./hello_baeldung | cat
Hello Baeldung
<pause for 1 second>
Hello Baeldung
<pause for 1 second>
Error happened!
<pause for 1 second>
Hello Baeldung
There are other values that the -o option can accept, which are the numbers combined with the byte size mode:
- KB (1000 bytes)
- K (1024 bytes)
- MB (1,000,000 bytes)
- M (1,048,576 bytes)
- G (1,000,000,000 bytes)
- T (1,000,000,000,000 bytes)
- and so on for P, E, Z, and Y.
So if we want to buffer for 5 KB, here’s how we can use execute stdbuf:
$ stdbuf -o5KB ./hello_baeldung | cat
It means we use the buffer with the 5 KB size. Of course, in our case, the output of the problem doesn’t even reach 1 KB. So the full output would still be buffered.
3.2. Standard Input Stream Buffering
We can also adjust the buffer size in the standard input stream. To illustrate the example, let’s download a text file that’s bigger than 1 MB in size. We can download the classical e-book, Moby Dick:
$ wget https://www.gutenberg.org/files/2701/2701-0.txt -O moby.txt
$ ls -s --block-size=KB moby.txt
1278kB moby.txt
The size of the e-book is 1278 KB. Now, let’s send its contents to a pipe before being received by the cut application on the other end:
$ time cat moby.txt | stdbuf -i0 cut -d ' ' -f 2
<omitted>...
real 0m1,947s
user 0m0,769s
sys 0m2,183s
The command above measured the time required to send the content of the e-book to the pipe before being received by the cut application. But, in the middle, stdbuf sets no buffers in the standard input stream of the cut application as can be seen with the -i option that receives the 0 value. The real time needed is almost two seconds.
Now, let’s use a buffer of size 1M:
$ time cat moby.txt | stdbuf -i1M cut -d ' ' -f 2
<omitted>...
real 0m0,090s
user 0m0,036s
sys 0m0,056s
So when we used the buffer, the process ran faster. The buffer held the content until it reached 1M in size. But when we didn’t use buffer, the process ran slower because every time there’s new content in the standard input stream, the cut application took it and processed it.
We should note that in the standard input stream, there’s no line buffered setting.
3.3. Standard Error Stream Buffering
stdbuf can also set the buffer in the standard error stream. Let’s create a program in C to print something in stderr and name it hello_baeldung_stderr.c:
#include <stdio.h>
#include <unistd.h>
int main()
{
fprintf(stderr, "Hello Baeldung\n");
sleep(1);
fprintf(stderr, "Hello Baeldung\n");
sleep(1);
fprintf(stderr, "Error happened!\n");
sleep(1);
fprintf(stderr, "Hello Baeldung\n");
return 0;
}
We should compile it before running it:
$ gcc hello_baeldung_stderr.c -o hello_baeldung_stderr
$ ./hello_baeldung_stderr | cat
Hello Baeldung
<pause for 1 second>
Hello Baeldung
<pause for 1 second>
Error happened!
<pause for 1 second>
Hello Baeldung
This time, the output was displayed line by line every one second, unlike in the case with the standard output stream. After three seconds, we got the full output.
But we can set the buffer so the full output will be displayed in one go:
$ stdbuf -e1KB ./hello_baeldung_stderr | cat
<pause for 3 seconds>
Hello Baeldung
Hello Baeldung
Error happened!
Hello Baeldung
To set the buffer in the standard error stream, we use the -e option and choose the number and the byte size. In the previous example, we chose 1 KB.
4. When stdbuf Cannot Help
We’ve seen that we can set the buffer in the three standard streams. But there’s an exception. Some programs have their own buffer setting that stdbuf cannot override.
Let’s create a Python program that does a similar thing. Let’s create a file called hello_baeldung.py and add the following content to it:
import time
print("Hello Baeldung")
time.sleep(1)
print("Hello Baeldung")
time.sleep(1)
print("Error happened!")
time.sleep(1)
print("Hello Baeldung")
We can run it without stdbuf:
$ python3 hello_baeldung.py | cat
<pause for 3 seconds>
Hello Baeldung
Hello Baeldung
Error happened!
Hello Baeldung
So the Python program used a buffer. Let’s use stdbuf to turn off the buffer:
$ stdbuf -o0 python3 hello_baeldung.py | cat
<pause for 3 seconds>
Hello Baeldung
Hello Baeldung
Error happened!
Hello Baeldung
The same thing happened! stdbuf is useless in this case. Some programs adjust the buffering of their standard streams and in this case, there’s nothing stdbuf can do.
However, we can change the buffer setting of Python itself by using the -u option:
$ stdbuf -o0 python3 -u hello_baeldung.py | cat
Hello Baeldung
<pause for 1 second>
Hello Baeldung
<pause for 1 second>
Error happened!
<pause for 1 second>
Hello Baeldung
The -u option of Python turns off the buffering in the standard output stream and the standard error stream.
5. Conclusion
In this article, we used stdbuf to turn off and set the size of the buffer in the standard streams. We used the -o option to change the buffer setting into the line buffered in the standard output stream. Then we used the -i option to adjust the buffer size in the standard input stream.
After that, we changed the buffer size in the standard error stream using the -e option.
Finally, we’ve also learned that some programs have their own buffer setting and stdbuf cannot change this.