1. Overview
When we want to copy directories recursively in the Linux command-line, cp -r could be the first command to come up.
We know that dotfiles are treated as hidden files in Linux. Sometimes, when we copy directories recursively, we want to exclude hidden files and directories under the directories.
In this quick tutorial, we’ll learn how to achieve that.
2. Introduction to the Problem
First of all, let’s build up a directory tree as an example:
$ tree -a parentDir
parentDir
├── file1
├── file2
├── .hidden_dir
│ ├── .hidden_sub2
│ ├── sub2_file1
│ └── sub2_file2
├── .hidden_parent
└── sub1
├── .hidden_sub1
├── sub1_file1
└── sub1_file2
2 directories, 9 files
We’ve used the handy tree command to present the content under the parentDir directory.
Since the directory and its subdirectories contain hidden files and directories, we’ve passed the -a option to the tree command to show the hidden objects.
Now, let’s copy the parentDir recursively to a new directory using the cp -r command, say parentDir2:
$ cp -r parentDir parentDir2
$ tree -a parentDir2
As the output above shows, the directory is successfully copied. However, we’ve copied those hidden files to the new directory as well.
Our goal is to copy directories recursively, excluding hidden directories and files.
Next, let’s see how to solve the problem.
3. Copying Everything Then Removing Hidden Files
One idea may come up to solve the problem: first copying everything recursively and then removing hidden objects from the target directory.
Now, let’s translate this idea into a command and give it a try:
$ cp -r parentDir parentDir2 && find parentDir2 -name '.*' | xargs rm -rf
$ tree -a parentDir2
parentDir2
├── file1
├── file2
└── sub1
├── sub1_file1
└── sub1_file2
1 directory, 4 files
We’ve combined the cp -r command and the “find and delete” command with “&&” so that the find command starts only if the cp -r command runs successfully.
As we can see in the tree output, all hidden objects are not in the parentDir2 directory. So the command works for the example.
This approach is straightforward and works for most cases. However, it may have some problems. So next, let’s take a closer look at them.
First, a hidden file or directory can be huge. If this is the case, this approach will unnecessarily copy the large objects and then delete them. That is to say, it may lead to a performance problem.
In a worse case, if the hidden objects are large enough, the target device may not have enough space to store them. Thus, the command may fail.
Second, sometimes, hidden files or directories contain sensitive data. That’s one reason we want to exclude them during copying.
If we copy everything to the target, the sensitive data will be automatically transferred to the target devices. We may think that would be ok since we delete them immediately after the cp command is done.
Once the copy command fails or takes a long time, the sensitive data can stay on an insecure device for a pretty long time. Further, if the target device is a remote filesystem, we don’t know if file-synchronization services are running.
Therefore, if the hidden objects contain sensitive data, this approach may create security holes.
4. Using the rsync Command
rsync is a powerful and convenient file-copying utility. It accepts two valuable options that allow us to include or exclude files during the copy: –include=PATTERN and –exclude=PATTERN.
To skip hidden files and directories, we can pass the “.*” pattern to the –exclude option. We should note that here, the “.*” pattern is not a regex. Instead, it indicates any filename or directory name beginning with a dot.
Now, let’s give it a try:
$ rsync -av --exclude=".*" parentDir/* parentDir3
sending incremental file list
created directory parentDir3
file1
file2
sub1/
sub1/sub1_file1
sub1/sub1_file2
sent 315 bytes received 137 bytes 904.00 bytes/sec
total size is 0 speedup is 0.00
$ tree -a parentDir3
parentDir3
├── file1
├── file2
└── sub1
├── sub1_file1
└── sub1_file2
1 directory, 4 files
As the output above shows, all hidden objects are not copied to the parentDir3 directory. So, the command does the job.
The rsync command supports the filter option -f to make including/excluding definitions easier.
For example, we can write our command in a shorter form:
rsync -av -f "- .*" parentDir/* parentDir3
In the command above, the -f option defines a filter, and a rule follows. When rsync works, it’s going to apply the filters on files it needs to copy.
A plus “+” indicates “include”, while a minus “-” means “exclude”. Thus, in our command, we exclude all hidden files and directories via “- .*”.
5. Conclusion
In this article, we’ve addressed how to copy directories recursively, excluding hidden files and directories.
The “copy everything then remove hidden objects” approach is straightforward. However, it may bring performance problems and security flaws.
When rsync is available on the system, we can consider using this file-copying tool to do the job.