1. Overview
Many Linux command-line tools allow us to process text conveniently, such as cut, sed, and awk. Usually, we use these utilities to process text files.
In this tutorial, we’ll take GNU sed as an example to address how to use an external command to process shell variables.
2. Introduction to the Problem
Bash has provided parameter expansions to manipulate string variables. Let’s see an example:
$ echo $VAR
Eric and Kent are good friends.
As the output above shows, the VAR variable holds a string. Using Bash’s parameter expansion, we can do some simple manipulations. For example, let’s say we want to replace “Kent” in the variable with the string “Saajan“:
$ RESULT="${VAR//Kent/Saajan}"
$ echo $RESULT
Eric and Saajan are good friends.
However, if the requirement is more complex, we may need to write multiple expansion expressions, or even a shell script function, to process the variable.
We still take the $VAR variable as an example. This time, we want to exchange the two names in the string to get “*Kent and Eric are good friends.*”
We can achieve that using sed‘s powerful ‘s‘ (substitution) command. However, we cannot pass the variable directly to the sed command:
$ echo $VAR
Eric and Kent are good friends.
$ sed -r 's/(\w+) and (\w+)(.*)/\2 and \1\3/' "$VAR"
sed: can't read Eric and Kent are good friends.: No such file or directory
As the error message above shows, the sed command refuses to perform the substitution. This is because sed treats the variable’s value as a file name if we pass it directly to sed.
Next, let’s see how we can solve the problem using the sed command.
3. Using the echo “$VAR” | sed Approach
Like many Linux commands, sed supports reading input from Stdin. Therefore, we can pipe the variable’s value to the sed command to perform string processing:
$ echo $VAR
Eric and Kent are good friends.
$ RESULT=$(echo "$VAR" | sed -r 's/(\w+) and (\w+)(.*)/\2 and \1\3/')
$ echo $RESULT
Kent and Eric are good friends.
As we can see in the example above, we assign the processed result to another variable RESULT using command substitution $(echo “$VAR” | sed …). Then, the RESULT variable holds the expected value.
4. Using Here-String (<<<)
Here-string can feed a command’s Stdin from a string. Further, if we use a variable in the here-string, it will get expanded, for example, COMMAND <<< “$MY_VAR”.
So, next, let’s use here-string to process VAR‘s value and assign the result to RESULT:
$ echo $VAR
Eric and Kent are good friends.
$ RESULT=$(sed -r 's/(\w+) and (\w+)(.*)/\2 and \1\3/' <<< "$VAR")
$ echo $RESULT
Kent and Eric are good friends.
As the output above shows, the RESULT variable contains the expected value.
5. Using Process Substitution
So far, we’ve solved the problem by feeding the sed command’s Stdin. We’ve mentioned sed treats its parameters as filenames.
Process substitution can save one process’s output in a temp file and pass the file to another process:
Process_A <(Process_B)
So, we can also solve the problem using process substitution. In our case, Process_A is the sed command, and we can simply use echo $VAR as Process_B to provide the variable’s value:
$ echo $VAR
Eric and Kent are good friends.
$ RESULT=$(sed -r 's/(\w+) and (\w+)(.*)/\2 and \1\3/' <( echo "$VAR" ))
$ echo $RESULT
Kent and Eric are good friends.
As we can see, process substitution does the job.
6. Conclusion
In this article, we’ve taken sed as an example and learned three approaches to using an external command to process shell variables:
- Pipe: echo $VAR | sed ‘…’
- Here-string: sed ‘…’ <<< $VAR
- Process substitution: sed ‘…’ <(echo $VAR)