1. Overview
While manipulating text files, we might need to insert a multi-line block of text after a specified pattern. In this tutorial, we’ll look at some of the Bash utilities that we can use for this purpose.
2. Setup
First, let’s set up our input file, which we’ll use in most of the tutorial:
$ cat stocks.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
Also, let’s create another file that contains the multi-line text that we want to insert into our original file:
$ cat missing_data.csv
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
Now, let’s look at various solutions to insert the contents from the missing_data.csv file to the stocks.csv file*,* after the line matching the pattern “08-08-2021,SCRIP5,340”.
3. Inserting Lines in a File Using the sed Editor
Let’s start our exploration with one of the most powerful text manipulation tools available, the sed editor.
3.1. When the Data Is in Another File
Let’s take a look at how to insert the data from the missing_data.csv file to the stocks.csv file:
$ sed -i '/08-08-2021,SCRIP5,340/r missing_data.csv' stocks.csv
$ cat stocks_new.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
In this case, we’ve specified the pattern to be matched in the /../ block of the sed command. Additionally, we’re using the r flag to read the data from the missing_data.csv file.
Alongside that, we’re also using the -i option to modify our file in place.
3.2. When the Data Is Specified Within the sed Command
In the previous section, we saw how to insert data from a file. However, we can also provide the data to be added within the sed command itself. Let’s take a look:
$ sed -i "/08-08-2021,SCRIP5,340/a\08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85" stocks.csv
$ cat stocks_new.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
Here, we’re specifying the lines to be appended after the matching pattern using the “a” flag of the sed command. Note that we’re using the newline characters (\n) within the input data to highlight line breaks.
3.3. Reading the Data From stdin
In the previous section, we saw an approach to insert data directly, without writing it to a file first. However, as we saw, it can make the command lengthy and prone to errors. Alternatively, we can also read the stdin using heredoc:
$ cat << EOF | sed -i '/08-08-2021,SCRIP5,340/r /dev/stdin' stocks.csv
> 08-08-2021,SCRIP6,49
> 08-08-2021,SCRIP7,76
> 08-08-2021,SCRIP8,85
> EOF
$ cat stocks_new.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
In this case, our delimiter token is EOF, and we’re using the cat command to print the user input. Subsequently, we’re piping this information to the sed command. Hence, the /dev/stdin file effectively contains the data piped to the sed command.
4. Inserting Lines in a File Using the awk Utility
In the section, we’ll check a few ways to insert a multi-line block of text in a file, after a specified pattern, using the awk utility.
4.1. When the Data Is Specified Within the awk Command
Let’s examine a way to insert the lines by directly specifying them within the awk command:
$ awk '/08-08-2021,SCRIP5,340/{print $0 "\n08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85";next}1'
stocks.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
The output is in line with our expectations. Further, let’s break this command into parts to discuss how it works.
We can specify the matching pattern in the awk command within the /../ block. In our case, the “/08-08-2021,SCRIP5,340/” becomes the matching pattern. We’re using the “print $0” syntax to print the matching line. Also, we’re printing the lines be added, each separated by the \n (newline) character:
$ awk '/08-08-2021,SCRIP5,340/{print $0 "\n08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85"}' stocks.csv
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
Now that we’ve got the matching line and the appended data, let’s also print the remaining lines:
$ awk '/08-08-2021,SCRIP5,340/{print $0 "\n08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85"}1' stocks.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP5,340
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
Here, the “1” at the end of the command always evaluates to true. Hence, it prints every line of the file. Notably, the matching line is printed twice. To prevent this, we’ll use the next command, which tells awk to skip the lines that match our pattern:
$ awk '/08-08-2021,SCRIP5,340/{print $0 "\n08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85";next}1' stocks.csv
Note that the awk command hasn’t made any changes in the original file. However, we can modify our command to make changes to a temporary file, which we then use to replace the original file:
$ awk '/08-08-2021,SCRIP5,340/{print $0 "\n08-08-2021,SCRIP6,49\n08-08-2021,SCRIP7,76\n08-08-2021,SCRIP8,85";next}1' stocks.csv > tempfile.csv && mv tempfile.csv stocks.csv
$ cat stocks.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
4.2. When the Data Is in Another File
Again, let’s look at a case when the data we want to insert is in another file:
$ missing_lines=$(cat missing_data.csv)
$ echo "$missing_lines"
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
$ awk -v addpattern="$missing_lines" '/08-08-2021,SCRIP5,340/{print $0 "\n" addpattern;next}1' stocks.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
In this approach, we’re first storing the contents of the file in a Bash variable. Then, we’re passing this variable to the awk command using the -v option for further processing.
5. Creating the Bash Script
Finally, let’s also write a Bash script solution using loops and the grep command:
$ cat insert_pattern.sh
#!/bin/bash
read -p "Enter the pattern: " pattern
>new_file.csv
while read line
do
echo $line >>new.csv
echo $line | grep -q "$pattern"
[ $? -eq 0 ] && cat missing_data.csv >> new.csv
done < stocks.csv
Here, we’re reading records from a file and writing them to a file new.csv. Then, we’re checking the pattern using the grep command. If the pattern matches, we’re appending the data of the missing_data.csv file in the new.csv file. Let’s execute the script and check the contents of the file:
$ ./insert_pattern.sh
Enter the pattern: 08-08-2021,SCRIP5,340
$ cat new.csv
Date,TICKR,Price
08-08-2021,SCRIP1,104
08-08-2021,SCRIP2,107
08-08-2021,SCRIP3,105
08-08-2021,SCRIP4,110
08-08-2021,SCRIP5,340
08-08-2021,SCRIP6,49
08-08-2021,SCRIP7,76
08-08-2021,SCRIP8,85
08-08-2021,SCRIP9,100
08-08-2021,SCRIP10,35
If required, we can copy the contents of the newly generated file to the original file.
6. Conclusion
In this article, we looked at ways to insert multiple lines into a file after a specified pattern using the Bash shell.
We started by discussing a few solutions using the sed command. In doing so, we explored approaches for handling different ways of providing the data to be added.
Then, we implemented some solutions using the awk command. Finally, we presented a solution by writing a shell script to insert data in a file using the Bash loops and the grep command.