1. Overview
When we write shell scripts using the command substitution technique, we can save a command’s output in a variable.
Sometimes, the output of a command can have multiple lines. We may lose all the linebreaks if we don’t use the shell variable in a proper way.
In this tutorial, we’re going to understand why linebreaks in a variable may disappear in the output. Further, we’ll discuss what’s the right way to use shell variables in commands.
When we talk about the shell in this tutorial, we’re focusing on Bash.
2. Introduction to the Problem
An example may explain the problem quickly. Let’s say we have a text file input.txt:
$ cat input.txt
1, GNU Linux
2, Apple MacOS
3, MS Windows
As the output above shows, the input.txt file contains three lines.
Now, let’s assign the output of the cat command above to a shell variable using command substitution:
VAR=$(cat input.txt)
Let’s have a look at the VAR variable to see if it has our expected value:
$ echo $VAR
1, GNU Linux 2, Apple MacOS 3, MS Windows
The output shows the three lines joined together. That is, it appears that we’ve lost all linebreaks.
Next, we’ll discuss and look for answers to a few questions:
- Do we really lose the linebreaks in the $VAR variable?
- Why are the linebreaks gone in the echo command’s output?
- What’s the proper way to preserve linebreaks?
3. Do We Really Lose the Linebreaks in the $VAR Variable?
When we look at the output of the echo command, we may infer that we’ve lost all linebreaks.
But does it mean the $VAR variable contains no linebreaks? Let’s have a look at another test:
$ cat <<<$VAR
1, GNU Linux
2, Apple MacOS
3, MS Windows
This time, we used a here-string to feed the stdin of the cat command using the value of the $VAR variable. And we can see the linebreaks are there.
That is to say, *we won’t lose the linebreaks after we assign variable by command substitution, VAR=$(command).*
Naturally, the question comes up: Why does the echo command print the three lines as one?
Next, let’s find out the answer.
4. Why Are the Linebreaks Gone in the echo Command’s Output?
So far, we’ve learned that our $VAR variable contains the linebreaks.
To answer why the linebreaks disappear in the output of echo $VAR, let’s break down and understand what happens when we execute echo $VAR.
First, the shell will read the value of the $VAR variable and split it into arguments, using the IFS environment variable to feed the echo command.
The default value of the IFS variable is whitespace, which contains spaces, tabs, and newlines.
Therefore, our echo $VAR becomes:
echo "1," "GNU" "Linux" "2," "Apple" "MacOS" "3," "MS" "Windows"
Obviously, the output of the command above won’t contain linebreaks.
To verify that, we can create a simple shell script:
$ cat ifs_test.sh
#!/bin/bash
VAR=$(cat input.txt)
OLD_IFS="$IFS"
IFS=""
echo $VAR
IFS="$OLD_IFS"
The script is pretty straightforward. Before we invoke echo $VAR, we set the IFS variable to an empty string to tell the shell not to perform splitting.
After the execution of the echo command, we restore the IFS variable.
Now, let’s see what it’s going to print if we execute the script:
$ ./ifs_test.sh
1, GNU Linux
2, Apple Mac OS
3, MS Windows
Yes, this time, we see the expected linebreaks.
But changing and restoring IFS isn’t so convenient. We cannot wrap all command calls with shell variables in shell scripts.
Next, let’s take a look at the proper way to preserve linebreaks in the output.
5. The Proper Way to Preserve Linebreaks
Let’s compare two commands before we introduce the solution to the problem:
$ echo $VAR
1, GNU Linux 2, Apple MacOS 3, MS Windows
$ echo "$VAR"
1, GNU Linux
2, Apple MacOS
3, MS Windows
The right way to use a variable is simple: quote the variable.
This is because if we quote a variable, the shell will treat it as one single argument to the command, no matter if the variable contains linebreaks or not.
It’s worth mentioning that when we quote a shell variable, we should use double-quotes because the variable won’t be expanded within single-quotes:
$ echo '$VAR'
$VAR
In fact, we should always quote variables if we use them in commands. Sometimes, using shell variables without quoting can be dangerous.
Let’s take a look at another example. Assume we’ve assigned the $FILENAME variable using:
FILENAME=$(some_command)
And then, let’s execute:
$ rm $FILENAME
The command above looks fine. It’ll remove the file stored in the $FILENAME variable. For example, if we have $FILENAME=”not_important.txt”, the file not_important.txt will be deleted.
However, most Linux filesystems allow spaces in filenames. One day, “some_command” may return a filename “not important.txt“. Then, our rm command becomes:
rm not important.txt
The problem here is obvious. If we have files named “important.txt” or “not” in the current directory, our command will delete them unexpectedly.
And the fix to the problem is simply quoting the shell variable:
rm "$FILENAME"
6. Conclusion
In this article, we’ve analyzed why the linebreaks in a shell variable can disappear in echo‘s output.
Also, we’ve introduced the right way to use shell variables in commands: always quote shell variables.