1. Overview
GNU sed is a powerful stream editor for processing textual data. However, unlike many other languages, it doesn’t support functions in the typical sense, which is its weak area. So, we need a mechanism to execute operations externally and use their output within a sed expression.
In this tutorial, we’ll learn how to embed a shell command into a sed expression. As a result, we’ll equip ourselves to solve complex problems with much more powerful and flexible sed expressions at our disposal.
2. Using the /e Flag
First, let’s start with a simple sed expression that prints the date command without executing it:
$ echo 'date' | sed -n 'p'
date
Next, let’s say we want to call this shell command within a sed expression. For this purpose, we can use the substitute command with the /e flag to execute the replaced string as a shell command:
$ echo 'date' | sed -n -E -e 's/(.*)/&/ep'
Sun Apr 30 20:18:18 UTC 2023
Herein, we must note that we’re using .* as the search pattern and & as the replacement string. So, we’re replacing the search pattern with itself. Then after, the /e flag executes the final replaced string as a shell command, and the /p flag prints the output after execution.
Moreover, we can also use this to execute multiple shell commands present in the replaced string:
$ echo 'date; sleep 1; date' | sed -n -E -e 's/(.*)/&/ep'
Sun Apr 30 22:43:07 UTC 2023
Sun Apr 30 22:43:08 UTC 2023
Moving on, let’s take a look at a scenario when the replaced string contains multiple shell commands, but not all of them are valid:
$ echo 'date; sleep 1; random1; random2; date' | sed -n -E -e 's/(.*)/&/ep'
sh: 1: random1: not found
sh: 1: random2: not found
Sun Apr 30 22:43:32 UTC 2023
Sun Apr 30 22:43:33 UTC 2023
In such a scenario, we can notice that the shell executed the commands partially, as two of the commands were invalid. Further, such errors are reported all at once at the very start.
Finally, let’s see the effect of swapping the order of the /e and /p flags in the sed’s substitution (s) command:
$ echo 'date > ~/sample-file' | sed -n -E -e 's/(.*)/&/pe'
date > ~/sample-file
It’s interesting to note that when we use the /p flag before the /e flag, it may seem that the /p flag prints the replaced string, but the /e flag doesn’t execute the replaced string as a shell command. However, that’s not true. In reality, both the flags did their job sequentially, and since /e is the last flag in the sequence, it executes, but the output doesn’t show up.
Additionally, we can verify our understanding by checking the presence of ~/sample-file:
$ cat ~/sample-file
Sun Apr 30 22:58:56 UTC 2023
That’s it. It seems that we have a thorough understanding of this concept now.
3. Using Double Quotes and Backticks
We can use the backticks to execute shell commands and return the output:
$ echo date
date
$ `echo date`
Sun Apr 30 23:10:24 UTC 2023
When writing sed expressions within double quotes, we can use the concept of command substitution to run the shell command(s) at runtime:
$ echo "date is: " | sed -n -E -e "s/(.*)/&`date`/p"
date is: Sun Apr 30 20:37:28 UTC 2023
Perfect! The result looks as expected.
4. Using Single Quotes and Backticks
sed lets us write the expressions within single quotes too. So, let’s go ahead and see if we can continue to use backticks for running a shell command during runtime:
$ echo "date is: " | sed -n -E -e 's/(.*)/&`date`/p'
date is: `date`
Unfortunately, the shell command didn’t execute successfully. That’s because *sed treats*`date` as a literal string when the expression is within the single quotes**. However, if we pass the output of the sed command to the echo command, then we’d solve our use case:
$ echo date is: `date`
date is: Sun Apr 30 23:23:10 UTC 2023
Next, let’s see how we can pass the output from the execution of the sed command to the echo command using xargs:
$ echo "Today date is: " | sed -n -E -e 's/(.*)/&`date`/p' | xargs -I{} bash -c 'echo {}'
date is: Sun Apr 30 21:11:46 UTC 2023
Great! It looks like we’ve got this right.
5. Using Double Quotes and $()
Although Bash still supports command substitution using backticks, it’s an outdated syntax. Further, for any new code, we must use the $() syntax for command substitution.
Now, let’s see how we can embed the date command into a sed expression that’s enclosed within double quotes:
$ echo "date is: " | sed -n -E -e "s/(.*)/&$(date)/p"
date is: Sun Apr 30 20:36:53 UTC 2023
Perfect! This was quite convenient.
6. Using Single Quotes and $()
In typical scenarios, the shell treats variables and commands within single quotes as literal strings. So, it won’t execute them, even if we’re using the modern $() syntax for command substitution:
# echo "date is: " | sed -n -E -e 's/(.*)/&$(date)/p'
date is: $(date)
Nonetheless, we can make it work by passing this output to another sed expression that uses the /e flag to execute this command via the echo command:
$ echo "Today date is: " | sed -n -E -e 's/(.*)/&$(date)/p' | sed -n 's/.*/echo &/ep'
Today date is: Sun Apr 30 21:25:45 UTC 2023
Fortunately, it worked this time.
Alternatively, we could also use the xargs command to pass the sed output as an argument to the echo command:
# echo "Today date is: " | sed -n -E -e 's/(.*)/&$(date)/p' | xargs -I{} bash -c 'echo {}'
Today date is: Sun Apr 30 21:28:15 UTC 2023
Fantastic! We’ve learned one more approach to solve our use case.
7. Using Double Quotes and ${}
First, let’s start by looking at a single sed expression that prints the input data without any filtering:
$ printf "Title\nContent-line1\nContent-line2\nContent-line3\nFooter\n" | sed -n -e 'p'
Title
Sub-Title
Content-line1
Content-line2
Content-line3
Footer
Now, let’s say we want to print the Title-specific data only. Since we know that this data corresponds to the first two lines from the input, we could use the <start, end>p command to solve our use case:
$ printf "Title\nSubTitle\nContent\nFooter\n" | sed -n -e "1,2p"
Title
SubTitle
However, if we were interested in the Title-only data excluding the subtitle, we’d need to change the sed expression. So, let’s decouple this by introducing the title_lines_cmd variable that will contain the logic about title-specific line numbers, and the sed expression will use it:
$ title_lines_cmd='echo 1,2'
$ printf "Title\nSubTitle\nContent\nFooter\n" | sed -n -e "$(${title_lines_cmd})p"
Title
SubTitle
$ title_lines_cmd='echo 1'
$ printf "Title\nSubTitle\nContent\nFooter\n" | sed -n -e "$(${title_lines_cmd})p"
Title
Great! The result looks as expected, and our approach made the sed expression more flexible for reuse.
8. Using Single Quotes and ${}
Let’s begin by taking a look at a scenario where we want to print the lines based on pattern matching:
$ echo 'this is sample data' | sed -n -e '/sample/p'
this is sample data
Herein, we used the sample string as a pattern, so there was a successful match.
Now, we can see a hard-coded value in the sed expressions, and that’s an opportunity for us to make it more flexible. So let’s go ahead and introduce the match variable with a default value:
# echo 'this is sample data' | sed -n -e '/'${match:-sample}'/p'
this is sample data
We can observe that we used the :- operator to define the default value for the match variable.
Lastly, let’s also verify a negative scenario where the search pattern doesn’t match the value in the match variable:
$ match='random'
$ echo 'this is sample data' | sed -n -e '/'${match:-sample}'/p'
# no output because pattern doesn't match
9. Using the Here Document
In this section, we’ll learn one last strategy where we’ll use here document for embedding a shell command into a sed expression.
Let’s start by taking a look at the data.log file that contains text data for our scenario:
$ cat data.log
log-line-1
log-line-2
Next, let’s say our goal is to prepend the operating system’s name before each log line that we can get from the /etc/os-release file:
$ cat /etc/os-release | awk -F'=' '/^NAME/{print $2}'
"Ubuntu"
Now, let’s get ready to write the sed_script_generator.sh Bash script that uses the tee command and a here document for generating the main.sed script dynamically:
$ cat sed_script_generator.sh
#!/bin/bash
osname=$(cat /etc/os-release | awk -F'=' '/^NAME/{print $2}')
tee main.sed 1>/dev/null <<END_SCRIPT
s/.*/${osname}: &/p
END_SCRIPT
Over here, we’ve used END_SCRIPT as a delimiter string for the here document. Further, we can directly reference the ${osname} variable within the same script.
Moving on, let’s see the script-generator.sh script in action to generate the main.sed script:
$ chmod u+x sed_script_generator.sh && ./sed_script_generator.sh
$ cat main.sed
s/.*/"Ubuntu": &/p
Finally, let’s use the sed command to run the commands from the main.sed script:
$ sed -n -E -f main.sed data.log
Mon May 1 03:17:28 IST 2023: log-line-1
Mon May 1 03:17:28 IST 2023: log-line-2
The output looks perfect.
10. Conclusion
In this article, we learned how to embed a shell command into a sed expression. Additionally, we explored several strategies to solve the use case, such as the /e substitution flag, command substitution with “ and $() , and parameter expansion with ${}.
More interestingly, we analyzed two widely used execution scenarios for sed expressions where they are enclosed within single or double quotes. Lastly, we wrote the script-generator.sh Bash script to generate a sed script dynamically using a here document.