1. Overview
Sometimes, we encounter files that contain specific lines in a uniform interval we wish to remove. For instance, when dealing with files containing repetitive or unnecessary lines that need removal. In that case, we can rely on the Linux core utilities, which handle this task gracefully.
In this article, we’ll use several utilities to remove every nth line from text content. For demonstration, we’ll write a simple text file that contains random Linux quotes:
1. "Linux: where file deletions need names, not mysteries."
2. "Linux: no windows, no gates, just Apache in the wigwam!"
3. "Linux unites, stays strong, and conquers together."
4. "Linux is friendly, but picks its friends wisely."
5. "Treat Linux like a pet, not a wild animal."
6. "Linux: control your destiny, not just costs."
7. "In Linux, Tux rules the OS kingdom."
8. "Linux: where you become one with the terminal."
9. "Linux: beyond an OS, it's a way of life."
10. "Linux wears the crown of computer freedom."
2. sed
In sed, the solution is by far the easiest. We simply specify the expression followed by the file:
$ sed '<start>:<lines>d' <filename>
Using this expression, let’s delete every second line of the file:
$ sed '0~2d' quotes.txt
1. "Linux: where file deletions need names, not mysteries."
3. "Linux unites, stays strong, and conquers together."
5. "Treat Linux like a pet, not a wild animal."
7. "In Linux, Tux rules the OS kingdom."
9. "Linux: beyond an OS, it's a way of life."
0~2d is a range address expression:
- 0~2 tells sed to start from line 0 and then every second line after that
- d is the operator that deletes the specified lines
Essentially, it deletes every second line starting from the first line. Similarly, we can achieve the same result in another way:
$ sed -n 'p;n' quotes.txt
1. "Linux: where file deletions need names, not mysteries."
3. "Linux unites, stays strong, and conquers together."
5. "Treat Linux like a pet, not a wild animal."
7. "In Linux, Tux rules the OS kingdom."
9. "Linux: beyond an OS, it's a way of life."
Let’s dig into this:
- -n tells sed not to print anything by default
- ‘p;n’ is a command script:
- p prints the current line
- n reads the next line into the current pattern space, replacing the current content
3. awk
In addition to sed, we can also use awk for the same task:
$ awk 'NR % 2 != 0' quotes.txt
1. "Linux: where file deletions need names, not mysteries."
3. "Linux unites, stays strong, and conquers together."
5. "Treat Linux like a pet, not a wild animal."
7. "In Linux, Tux rules the OS kingdom."
9. "Linux: beyond an OS, it's a way of life."
Let’s dive into the NR % 2 != 0 expression:
- NR is the built-in variable representing the “Number of Records” or the current line number
- NR % 2 calculates the remainder by dividing the line number by 2
- != 0 checks if the remainder is equal to zero, which concludes whether the number is odd
When the condition evaluates to true, the line is printed immediately. In the same way, we can print every third line and discard the rest:
$ awk 'NR % 3 == 0' quotes.txt
3. "Linux unites, stays strong, and conquers together."
6. "Linux: control your destiny, not just costs."
9. "Linux: beyond an OS, it's a way of life."
In the expression, we check if the line number is entirely divisible by 3. When it yields true, it prints that line.
4. Perl
Perl is the Swiss Army knife for text processing. We can easily write a one-liner to delete lines at regular intervals:
$ perl -ne 'print if $. % 3' <quotes.txt
1. "Linux: where file deletions need names, not mysteries."
2. "Linux: no windows, no gates, just Apache in the wigwam!"
4. "Linux is friendly, but picks its friends wisely."
5. "Treat Linux like a pet, not a wild animal."
7. "In Linux, Tux rules the OS kingdom."
8. "Linux: where you become one with the terminal."
10. "Linux wears the crown of computer freedom."
Let’s dissect this:
- -n starts a loop that goes over each line in the text
- -e allows us to run Perl code in the command-line
- ‘print if $. % 3’ is the actual Perl code:
- $. refers to the current line number
- $. % 3 calculates the remainder
- print prints the line if its condition evaluates to true
- <quotes.txt redirects the content of the file to Perl
We can change the number inside the condition accordingly.
5. Vi/Vim
In addition to the other tools, we can also use Vim to delete every next line:
:g/^/+d
Let’s dig into this:
- g applies the expression globally
- /$/ matches every line
- +d deletes the next line
Alternatively, we can use macros if we need to delete every nth line. We start recording a macro by pressing “q“, followed by a register to store the macro:
ggqa
gg takes us to the first line in the buffer, and qa records the macro into a. Next, we carry out simple deletion. For instance, let’s go down three lines from the current one and delete it:
jjjdd
Finally, press q to stop the macro. Now, we can repeat this macro for all the lines:
10000qa
It effectively repeats the macro for the specified lines. However, we can increase this number when required.
6. Conclusion
In this tutorial, we learned how to remove lines that are repeated every nth interval in a file. We used several different utilities for this task. Finally, we also discussed how to carry out this task in Vim with macros.