1. Overview
In Linux, hidden files are prefixed with a dot (.) character, for example, ‘*.hiddenfile.txt*‘. By default, these files aren’t visible when browsing through a terminal or GUI file manager. However, there is typically a configuration option or menu to display them.
The same behavior applies to file operations, such as copying files. The command to copy files usually does not include hidden files by default.
In this tutorial, we’ll learn how to ensure we copy hidden files.
All commands mentioned in this guide have been tested on Debian Bullseye, running GNU bash 5.1.4, cp 8.32, find 4.8.0, rsync 3.2.3, and tar 1.34.
2. Setting Up a Sample Directory Structure
First, let’s create a sample directory structure:
$ mkdir -p source/a/b source/c/d target
$ touch source/a/file01.txt source/a/file02.txt source/a/.hidden01.txt source/a/.hidden02.txt
$ touch source/a/b/file03.txt source/a/b/file04.txt source/a/b/.hidden03.txt source/a/b/.hidden04.txt
$ touch source/c/file05.txt source/c/file06.txt source/c/.hidden05.txt source/c/.hidden06.txt
$ touch source/c/d/file07.txt source/c/d/file08.txt source/c/d/.hidden07.txt source/c/d/.hidden08.txt
$ tree -a
.
├── source
│ ├── a
│ │ ├── b
│ │ │ ├── file03.txt
│ │ │ ├── file04.txt
│ │ │ ├── .hidden03.txt
│ │ │ └── .hidden04.txt
│ │ ├── file01.txt
│ │ ├── file02.txt
│ │ ├── .hidden01.txt
│ │ └── .hidden02.txt
│ └── c
│ ├── d
│ │ ├── file07.txt
│ │ ├── file08.txt
│ │ ├── .hidden07.txt
│ │ └── .hidden08.txt
│ ├── file05.txt
│ ├── file06.txt
│ ├── .hidden05.txt
│ └── .hidden06.txt
└── target
6 directories, 16 files
The -p option in the mkdir command stands for parents, where it’ll make parent directories if needed and show no error if the directory exists. Meanwhile, the -a option in the tree command shows all files, including hidden ones.
We now have a simple directory structure that has a couple of hidden files in each directory.
3. Using cp
The standard cp command can be used on its own or in conjunction with find.
3.1. Using cp Standalone
Let’s use the cp command to copy the hidden files recursively to the target directory:
$ cp -r source/* target
$ tree -a
.
├── source
...
...
└── target
├── a
│ ├── b
│ │ ├── file03.txt
│ │ ├── file04.txt
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── file01.txt
│ ├── file02.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── file07.txt
│ ├── file08.txt
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── file05.txt
├── file06.txt
├── .hidden05.txt
└── .hidden06.txt
10 directories, 32 files
The -r option means copy directories recursively.
As we can see above, the cp command copied all files, including the hidden files.
If we want to copy only the hidden files:
$ cp -R --target-directory=target source/*/.[^.]* source/*/*/.[^.]*
$ tree -a
.
├── source
...
...
└── target
├── .hidden01.txt
├── .hidden02.txt
├── .hidden03.txt
├── .hidden04.txt
├── .hidden05.txt
├── .hidden06.txt
├── .hidden07.txt
└── .hidden08.txt
6 directories, 24 files
Let’s break down the command. The -r option means copy directories recursively. Then, –target-directory provides the target directory. Lastly, source/*/.[^.]* and source/*/*/.[^.]* are the source directories for each directory level.
If we want to preserve the directory structure:
$ cp -R --parents --target-directory=target source/*/.[^.]* source/*/*/.[^.]*
$ tree -a
.
├── source
...
...
└── target
└── source
├── a
│ ├── b
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── .hidden05.txt
└── .hidden06.txt
11 directories, 24 files
The –parents option preserves the source directory structure in the target directory.
3.2. Using cp with find
In the previous section, we had to define the source directory for each directory level, which was extra work.
As an alternative, we can utilize the find command to list all the files that we want to copy, in this case, the hidden files, then pipe them to cp.
Let’s first list the hidden files with find:
$ find -H source -name '.*' -a \( -type d -o -type f -o -type l \)
source/a/.hidden01.txt
source/a/b/.hidden04.txt
source/a/b/.hidden03.txt
source/a/.hidden02.txt
source/c/.hidden06.txt
source/c/.hidden05.txt
source/c/d/.hidden07.txt
source/c/d/.hidden08.txt
Then we pass the find output to cp:
$ find -H source -name '.*' -a \( -type d -o -type f -o -type l \) -exec cp -a '{}' target/ \;
$ tree -a
.
├── source
...
...
└── target
├── .hidden01.txt
├── .hidden02.txt
├── .hidden03.txt
├── .hidden04.txt
├── .hidden05.txt
├── .hidden06.txt
├── .hidden07.txt
└── .hidden08.txt
6 directories, 24 files
As we can see from the output above, the hidden files were all copied to the target directory.
Let’s break down the command:
- find – the command to search for files
- -H – do not follow symlinks
- source – the source directory
- -name ‘.*’ – the pattern of the filename that we want to find
- a \( … \) – an AND operator followed by a condition in the parentheses
- -type d -o -type f -o -type l – the conditions with the OR ‘-o’ operator; -type d matches directories, -type f matches files, and -type l matches symlinks
- -exec cp -a ‘{}’ target/ \; – tells find to execute the cp command every time a match is encountered. The -a means preserves file attributes, and {} is where each output from find will end up
If we want to have the same source directory structure in the target directory, we can pass the –parents option to cp:
$ find -H source -name '.*' -a \( -type d -o -type f -o -type l \) -exec cp -a --parents '{}' target/ \;
$ tree -a
.
├── source
...
...
└── target
└── source
├── a
│ ├── b
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── .hidden05.txt
└── .hidden06.txt
11 directories, 24 files
As we can see from the above output, the hidden files were copied to the target directory along with the source directory structure.
4. Using rsync
rsync is a common command for copying files between locations:
$ rsync -a --include=".*" --include="*/" --exclude="*" source/ target/
$ tree -a
.
├── source
...
...
└── target
├── a
│ ├── b
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── .hidden05.txt
└── .hidden06.txt
10 directories, 24 files
Let’s run through the options that we used:
- -a or –archive – preserves various file attributes, including permissions, timestamps, and recursively copies directories
- –include=’.*’ – includes hidden files
- –include=’*/’ – includes directories so that the recursion can continue inside them
- –exclude=’*’ – excludes all other files and directories
- source/ – the source directory, followed by a trailing slash (/) to copy the contents rather than the directory itself
- target/ – the target directory, followed by a trailing slash (/) to ensure that the files are copied into the directory instead of creating a new sub directory with the source directory name
5. Using tar
We can also utilize the tar command to copy the hidden files:
$ tar cf - source/*/.[^.]* source/*/*/.[^.]* | (cd target && tar xf -)
tenzin@modigeko:~/apps/jobs/baeldung/reccp$ tree -a
.
├── source
...
...
└── target
└── source
├── a
│ ├── b
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── .hidden05.txt
└── .hidden06.txt
11 directories, 24 files
The tar command compressed the hidden files (tar cf – source/*/.[^.]* source/*/*/.[^.]*) and then uncompressed it at the target directory (cd target && tar xf –).
The c option creates a new archive, and f specifies the archive filename or destination. We set the destination to dash (–) to send the output to standard output (stdout), which we then piped (|) to tar xf –, which then extracted the contents of the tar archive from the standard input (stdin).
6. Using cpio with find
cpio is another alternative. However, it’s less commonly used.
First, we need to get the list of the hidden files using find:
$ find -H source -depth -name '.*' -a \( -type d -o -type f -o -type l \) -print0
source/a/.hidden01.txtsource/a/b/.hidden04.txtsource/a/b/.hidden03.txtsource/a/.hidden02.txtsource/c/.hidden06.txtsource/c/.hidden05.txtsource/c/d/.hidden07.txtsource/c/d/.hidden08.txt
The output of find might look like a very long string, but it’s actually a null-separated list of files and directories.
We reused the find command from earlier and added -depth and -print0 options.
The -depth option tells find to process directories in a depth-first manner, meaning it will descend into sub directories before processing the current directory.
The -print0 option tells find to print the found files and directories separated by a null character instead of a newline character. This ensures the correct handling of filenames with spaces or special characters. It also allows programs that process the find output to correctly interpret filenames containing newlines or other types of whitespace.
Next, let’s pipe the find output to cpio:
$ find -H source -depth -name '.*' -a \( -type d -o -type f -o -type l \) -print0 | cpio --null -pd target/
0 blocks
$ tree -a
.
├── source
...
...
└── target
└── source
├── a
│ ├── b
│ │ ├── .hidden03.txt
│ │ └── .hidden04.txt
│ ├── .hidden01.txt
│ └── .hidden02.txt
└── c
├── d
│ ├── .hidden07.txt
│ └── .hidden08.txt
├── .hidden05.txt
└── .hidden06.txt
11 directories, 24 files
The target directory now contains all the hidden files from the source directory, preserving the directory structure.
Let’s break down the cpio command.
The cpio command receives the input from find. The –null option tells cpio that the filenames in the input list are delimited by null characters instead of newlines. The -p option instructs cpio to read a list of file names from the standard input and copy them to the target directory. Finally, the -d option is to create directories if necessary.
7. Conclusion
In this article, we learned how to recursively copy hidden files using various copy commands such as rsync, cp, tar, and cpio.
However, despite having several options available, it’s probably best to use the tools specifically built for the operation, such as rsync and cp.
That said, tar and cpio can also achieve this, and these techniques may be useful when the aim is to use those tools while including hidden files in the archives they produce.