1. Overview
We know the grep command is a handy utility for file content search in the Linux command line. Also, grep allows us to recursively search a directory to find all files matching the given pattern in the directory.
In this tutorial, let’s explore how to use grep to search files and move the found files to a directory in one shot.
2. Introduction to the Problem
Let’s say some files are located in different sub-directories. First, we’ll search for a pattern in all files in the parent directory recursively. Then, we’d like to move the matched files to a new place.
Since the files are in different directories, we should note that the found files may have the same filenames, such as*/parent/dir1/file.txt* and /parent/dir2/file.txt.
In this tutorial, we’ll assume that the found files don’t have duplicated filenames, as our goal is to address approaches to solve a general problem. However, in the filename collision situation, we may have specific requirements to handle the collision, such as overwriting or renaming.
Usually, an example can explain a problem quickly. Let’s first have a look at some log files under the logs directory:
$ head logs/**/*.log
==> logs/app1/app1.log <==
2022-01-20 15:21:10 application started
2022-01-20 17:07:14 [Warn] Security alert: 10 Permission Denied Requests from the same IP ...
2022-01-20 17:08:14 [Warn] High RAM usage: 90%
2022-01-20 17:14:10 RAM usage is back to normal
==> logs/app1/app1_user.log <==
2022-01-20 19:22:10 user Kevin created
2022-01-20 20:21:10 user Kevin login
2022-01-20 22:18:10 security alert: 10 times failed login from the same IP ...
==> logs/app2/app2.log <==
2021-11-20 15:21:10 application started
2021-11-20 17:08:14 [Warn] High CPU usage: 80%
2021-11-20 17:14:10 CPU usage is back to normal
==> logs/app2/app2_user.log <==
2021-11-20 19:21:10 user Eric login
2021-11-20 22:08:14 security alert: 10 times failed login from the same IP ...
2021-11-20 23:44:10 user Eric logout
As we can see in the output above, we have four log files for two applications – app1 and app2.
These log files are lying in different directories. Also, some log files are case-insensitively containing “security alert” logs. So, our requirement is to find those log files containing “security alert” entries and move them to a new directory in one single command.
Today, we’ll address three approaches to achieve that:
- Using the grep command and a while loop
- Using the grep command and the xargs command
- Using Command Substitution
Next, let’s see them in action.
3. Using the grep Command and a while Loop
First, we examine the log files’ content to find the files to move. The grep command is a good choice for this task:
$ grep -lir 'security alert' logs
logs/app2/app2_user.log
logs/app1/app1_user.log
logs/app1/app1.log
As grep‘s output shows, we’ve found the three log files containing “security alert“. Here, we’ve used three of grep‘s options:
- -l: Output the matched filenames only
- -i: Case-insensitive match
- -r: Read all files under each directory recursively
In the next step, *a while loop will take over the control. It’ll read each filename from grep‘s result and move the file to the target directory*. Of course, before we move the files, we should create the target directory:
$ mkdir logs/security_logs_loop && grep -lir 'security alert' logs | while read log; do mv "$log" logs/security_logs_loop; done
$ tree logs
logs
├── app1
├── app2
│ └── app2.log
└── security_logs_loop
├── app1.log
├── app1_user.log
└── app2_user.log
3 directories, 4 files
As we can see in the tree output above, after the command’s execution, the three log files have been moved to the target directory.
4. Using the grep and xargs Commands
We’ve learned to use grep to get the file list we want to move. Then, we piped the file list to a while loop to solve the problem.
Alternatively, we can pipe grep‘s output to the xargs command to do the job:
$ mkdir logs/security_logs_xargs && grep -lir 'security alert' logs | xargs -I'{}' mv '{}' logs/security_logs_xargs
$ tree logs
logs
├── app1
├── app2
│ └── app2.log
└── security_logs_xargs
├── app1.log
├── app1_user.log
└── app2_user.log
3 directories, 4 files
The output above shows we’ve moved the required log files to the target directory.
In the xargs command, we’ve used the -I{} placeholder to build the mv FOUND_LOGS TARGET_DIR command.
Since the mv command supports the -t option, we can write the command in a shorter form:
mkdir logs/security_logs_xargs && grep -lir 'security alert' logs | xargs mv -t logs/security_logs_xargs
5. grep | while vs. grep | xargs
We’ve addressed two solutions following the pattern grep … | while or xargs. We may ask, what’s the difference between the two approaches? Further, which one is preferred?
First, “while” is a shell statement, which is “built-in”, whereas the xargs command is an external command from the findutils package. Therefore, xargs is not universally available, although most modern Linux Distros have it installed by default.
Next, let’s talk about the performance.
Apparently, the while loop will read a file in each step and execute the mv command until all files have been read. On the other hand, the xargs command doesn’t work similarly. xargs will build found files into bundles until it reaches a system-defined limit and then run them through the command, which is mv in this case. In other words, xargs executes mv as few times as possible.
Therefore, the “grep | while” approach will execute three mv commands in our example. But “grep | xargs” runs mv only one single time. That is to say, “grep | xargs” has better performance than “grep | while”, especially when moving many files.
6. Using Command Substitution
So far, we’ve learned two methods to solve the problem. However, if we take a closer look at the two solutions, both start from the grep command and pipe grep‘s output to the while loop or xargs.
Alternatively, we can interpret the problem as:
mv [Required_Files] targetDir
Now, the problem is converted into how to find the required files and fill the command above. Finding those files isn’t a challenge for us. We’ve done it a couple of times already using the grep command.
Further, command substitution allows us to quickly obtain the command’s output and use it in further operations.
Next, let’s try to solve the problem using command substitution:
$ mkdir logs/security_logs_cmdsub && mv $(grep -lir 'security alert' logs) logs/security_logs_cmdsub
$ tree logs
logs
├── app1
├── app2
│ └── app2.log
└── security_logs_cmdsub
├── app1.log
├── app1_user.log
└── app2_user.log
3 directories, 4 files
Good! Our command works.
7. Conclusion
In this article, we’ve discussed using grep to search files and move the found files to a directory in one shot. We’ve addressed three different approaches through examples.
Also, we’ve talked about the difference between the “grep | xargs” and “grep | while” methods.
Although our example’s goal is to mv files, in practice, we can adjust the solutions to adapt other file manipulation requirements, such as cp or rm.