1. Overview
sed is a powerful text processing tool in the Linux command-line. We often do text substitution using compact sed one-liners. They’re pretty convenient.
However, when we execute sed substitution with shell variables, there are some pitfalls we should be aware of.
In this tutorial, we’ll take a closer look at some common mistakes made using sed substitution with shell variables, and we’ll offer some solutions.
2. The Example Problem
To more easily reproduce those common mistakes and discuss how to fix them, let’s make an example problem.
Suppose we have a file test.txt:
$ cat test.txt
CURRENT_TIME = # fill the current date and time
JAVA_HOME = # fill the JAVA_HOME path
We want to write a shell script to fill the current time and the JAVA_HOME path of the current system in the file above.
The task looks easy. However, there are some potential problems.
Let’s write the script together using GNU sed.
3. Which Quotes Should We Use?
As the problem requires, we need to perform two substitutions: the current time and the JAVA_HOME path.
First, let’s fill the current time in the right place. We can use the date command to get the current time:
$ cat solution.sh
#!/bin/sh
MY_DATE=$(date)
sed -i -r 's/^(CURRENT_TIME =).*/\1 $MY_DATE/' test.txt
The script above isn’t hard to understand. Let’s walk through it quickly.
We first get the current date and time from a command substitution and save it in a variable MY_DATE.
After we get the date, we fill it in the file by using a sed substitution. We’ve used the GNU sed command’s -i option to do an in-place edit.
Let’s execute our script and check if it works as we expect:
$ ./solution.sh
$ cat test.txt
CURRENT_TIME = $MY_DATE
JAVA_HOME = # fill the JAVA_HOME path
As the output above shows, the line with “CURRENT_TIME =” has been substituted. However, instead of the current date and time, the literal “$MY_DATE” is filled.
This happened because we used single quotes in our sed command. Shell variables will not get expanded within single quotes.
Therefore, the quick fix would be using double quotes in the sed command to allow shell variable expansion:
$ cat solution.sh
#!/bin/sh
MY_DATE=$(date)
sed -i -r "s/^(CURRENT_TIME =).*/\1 $MY_DATE/" test.txt
Now, let’s test the solution.sh script once again:
$ ./solution.sh
$ cat test.txt
CURRENT_TIME = Wed Jan 27 10:02:05 PM CET 2021
JAVA_HOME = # fill the JAVA_HOME path
Good! We’ve filled the date and time in the right place.
Next, let’s fill the JAVA_HOME path in the file.
4. Which Delimiter Should We Use?
Now that we have the substitution of the current time working, we may think that the JAVA_HOME part is more or less a copy-and-paste job.
Is it really that simple? Let’s add one more sed command in our solution.sh script:
$ cat solution.sh
...
sed -i -r "s/^(CURRENT_TIME =).*/\1 $MY_DATE/" test.txt
sed -i -r "s/^(JAVA_HOME =).*/\1 $JAVA_HOME/" test.txt
It’s time to test the script:
$ ./solution.sh
sed: -e expression #1, char 24: unknown option to `s'
Oops! The newly added sed command doesn’t work. If we double-check it, it is pretty similar to the other working sed command, and only the variable is different.
What’s going on?
4.1. Choosing a Delimiter Not Contained in the Variable
To understand what has happened, let’s first check what’s in the environment variable $JAVA_HOME:
$ echo $JAVA_HOME
/usr/lib/jvm/default
We’ve learned that shell variables will get expanded within double-quotes. Therefore, after the variable expansion, our second sed command becomes:
sed -i -r "s/^(JAVA_HOME =).*/\1 /usr/lib/jvm/default/" test.txt
Well, the above sed command obviously won’t work because the slashes (/) in the variable’s value interfere with the ‘s‘ command (s/pattern/replacement/).
Fortunately, we can choose other characters as the delimiter of the ‘s’ command.
Let’s modify the second sed command a little bit and use ‘#’ as the delimiter of the s command:
sed -i -r "s#^(JAVA_HOME =).*#\1 $JAVA_HOME#" test.txt
Now, let’s test the script again:
$ ./solution.sh
$ cat test.txt
CURRENT_TIME = Wed Jan 27 10:36:57 PM CET 2021
JAVA_HOME = /usr/lib/jvm/default
Great! The problem is solved — or is it?
4.2. A Better Solution
Indeed, our solution.sh will work in most cases. However, it’s worthwhile to mention that ‘#’ is a valid character in filenames on most *nix filesystems.
That means, if one day we execute our script on a system with JAVA_HOME set to /opt/#jvm#, for example, the script will fail again.
To make our script work for all cases, we can:
- First, choose a delimiter for sed’s s command. Let’s say we take ‘#’ as the delimiter for better readability
- Second, escape all delimiter characters in the content of the variables
- Finally, assemble the escaped content in the sed command
We can use Bash substitution to escape delimiters. For example, we can escape all ‘#’ characters in the variable $VAR:
$ VAR="foo#bar#blah"
$ echo "${VAR//#/\\#}"
foo\#bar\#blah
Let’s apply it to our script:
$ cat solution.sh
#!/bin/sh
MY_DATE=$(date)
sed -i -r "s/^(CURRENT_TIME =).*/\1 $MY_DATE/" test.txt
sed -i -r "s#^(JAVA_HOME =).*#\1 ${JAVA_HOME//#/\\#}#" test.txt
Next, let’s execute our script with a simulated JAVA_HOME variable and check if it works as we expect:
$ JAVA_HOME=/opt/#/:/@/-/_/$/jvm ./solution.sh
$ cat test.txt
CURRENT_TIME = Thu Jan 28 11:23:07 AM CET 2021
JAVA_HOME = /opt/#/:/@/-/_/$/jvm
As the output shows, our script works even if our JAVA_HOME variable contains many special characters.
5. Conclusion
This article discussed some common mistakes we may make when we write sed substitution with shell variables.
We’ve also addressed how to solve those problems through examples.