1. Overview
When we write shell scripts, we often call a command and save the output into a variable for further processing. Sometimes, we want to save a multi-line output into a Bash array.
In this tutorial, we’ll discuss some common pitfalls of doing this and address how to do it in the right way.
2. Common Pitfalls
First of all, let’s define our problem. We’re going to execute a command and save its multi-line output into a Bash array. Each line should be an element of the array.
At first glance, the problem looks simple. We can put a command substitution between parentheses to initialize an array:
my_array=( $(command) )
Let’s take the seq command as an example and try if the above approach works:
$ seq 5
1
2
3
4
5
$ my_array=( $(seq 5) )
$ declare -p my_array
declare -a my_array=([0]="1" [1]="2" [2]="3" [3]="4" [4]="5")
Great, it works!
We use the Bash built-in declare with the -p option to examine the array. It shows that the array has been initialized as we expected.
Well, so far, so good. However, this is not a stable solution. Let’s see what’s wrong with it.
2.1. Output May Contain Spaces
The output of a command can often include spaces. Let’s change the seq command a little bit and check if our solution still works:
$ seq -f 'Num %g' 5
Num 1
Num 2
Num 3
Num 4
Num 5
$ my_array=( $(seq -f 'Num %g' 5) )
$ declare -p my_array
declare -a my_array=([0]="Num" [1]="1" [2]="Num" [3]="2" [4]="Num" [5]="3" [6]="Num" [7]="4" [8]="Num" [9]="5")
The spaces in the output break our solution. The output above tells us, the my_array now has ten elements, instead of five.
The fix may come to mind immediately: set the IFS to a newline character, so that a whole line can be assigned to an array element.
Let’s try if it can fix the problem:
$ IFS=$'\n'
$ my_array=( $(seq -f 'Num %g' 5) )
$ declare -p my_array
declare -a my_array=([0]="Num 1" [1]="Num 2" [2]="Num 3" [3]="Num 4" [4]="Num 5")
Yes! That fixed it!
Unfortunately, the solution is still fragile, even though it handled spaces correctly.
Let’s see what problem it still has.
2.2. Output May Contain Wildcard Characters
Some output of a command may contain wildcard characters such as *, […] or ?, and so on.
Let’s change the seq command once again and create a couple of files under our working directory:
$ seq -f 'Num*%g' 5
Num*1
Num*2
Num*3
Num*4
Num*5
$ touch Number.app.log.{4..5}
$ ls -1
Number.app.log.4
Number.app.log.5
Now, let’s check if our solution can still convert the output into an array correctly:
$ my_array=( $(seq -f 'Num*%g' 5) )
$ declare -p my_array
declare -a my_array=([0]="Num*1" [1]="Num*2" [2]="Num*3" [3]="Number.app.log.4" [4]="Number.app.log.5")
Oops! The last two elements are filled by the two filenames instead of the expected “Num*4″ and “Num*5”. This is because if the wildcard characters match some filenames in our working directory, the filename will be picked instead of the original string.
Well, we can do a quick fix to disable the filename globbing by set -f. However, it’s not wise to fix a fragile technique by changing the IFS and set -f.
Next, let’s take a look at more proper ways to solve the problem.
3. Using the readarray Command
readarray is a built-in Bash command. It was introduced in Bash ver.4.
We can use the readarray built-in to solve the problem:
$ readarray -t my_array < <(seq 5)
$ declare -p my_array
declare -a my_array=([0]="1" [1]="2" [2]="3" [3]="4" [4]="5")
$ readarray -t my_array < <(seq -f 'Num %g' 5)
$ declare -p my_array
declare -a my_array=([0]="Num 1" [1]="Num 2" [2]="Num 3" [3]="Num 4" [4]="Num 5")
$ ls -1
Number.app.log.4
Number.app.log.5
$ readarray -t my_array < <(seq -f 'Num*%g' 5)
$ declare -p my_array
declare -a my_array=([0]="Num*1" [1]="Num*2" [2]="Num*3" [3]="Num*4" [4]="Num*5")
The output above shows that readarray -t my_array < <(COMMAND) can always convert the output of the COMMAND into the my_array correctly. This works no matter if the COMMAND output contains spaces or wildcard characters.
Now, let’s understand why it works.
The readarray reads lines from the standard input into an array variable: my_array. The -t option will remove the trailing newlines from each line.
We used the < <(COMMAND) trick to redirect the COMMAND output to the standard input. The <(COMMAND) is called process substitution. It makes the output of the COMMAND appear like a file. Then, we redirect the file to standard input using the < FILE.
Thus, the readarray command can read the output of the COMMAND and save it to our my_array.
4. Using the read Command
We’ve seen that by using the readarray command, we can conveniently solve this problem. Since the readarray command was introduced in Bash ver.4, it is not available if we are working with an older Bash version.
The Bash shell has another built-in command: read, it reads a line of text from the standard input and splits it into words.
We can solve the problem using the read command:
IFS=$'\n' read -r -d '' -a my_array < <( COMMAND && printf '\0' )
Let’s test it and see if it will work on different cases:
$ IFS=$'\n' read -r -d '' -a my_array < <( seq 5 && printf '\0' )
$ declare -p my_array
declare -a my_array=([0]="1" [1]="2" [2]="3" [3]="4" [4]="5")
$ IFS=$'\n' read -r -d '' -a my_array < <( seq -f 'Num %g' 5 && printf '\0' )
$ declare -p my_array
declare -a my_array=([0]="Num 1" [1]="Num 2" [2]="Num 3" [3]="Num 4" [4]="Num 5")
$ IFS=$'\n' read -r -d '' -a my_array < <( seq -f 'Num*%g' 5 && printf '\0' )
$ declare -p my_array
declare -a my_array=([0]="Num*1" [1]="Num*2" [2]="Num*3" [3]="Num*4" [4]="Num*5")
The output shows it works with our examples as well. The command looks a little bit longer than the readarray one, but it’s not hard to understand either.
Let’s break it down to explain what it does:
- COMMAND && printf ‘\0’: Here we append a null byte ‘\0’ to the output of the COMMAND so that later read will stop reading here
- < <(COMMAND && printf ‘\0’): This is not new to us. We redirect the COMMAND output together with the trailing null byte to the standard input
- IFS=$’\n’: We’ve learned this as well, we set the IFS to a newline character so that the read command will read a whole line from the stream
- read -r: The -r option tells the read command not to interpret the backslashes as escape sequences
- -d ”: We let the read command stop reading at a null byte
- -a my_array: This is straightforward, we tell the read command to populate the array my_array while reading
It’s worthwhile to mention that the IFS variable change will only set the variable for the read statement. It won’t interfere with the current shell environment.
5. Conclusion
In this article, we’ve solved the problem: How to save the output of a command into a Bash array.
The readarray command will be the most straightforward solution to that problem if we’re working with a Bash newer than Ver. 4.
If we have to work with an older Bash, we can still solve the problem using the read command.
Apart from that, we’ve also seen some common pitfalls, which we should pay attention to when we write shell scripts.