1. Overview
In this tutorial, we discuss what we have to keep in mind when moving data from one directory to another, for which one helpful tool is the rsync command.
A typical use case, for example, is when we develop a website staging from development to live system.
2. Problem With rsync and dotfiles
Let’s assume a typical scenario. We have two folders with web content. One is the development path. The other one is the production path:
$ ls -las
total 16
4 drwxrwxr-x 4 tracer tracer 4096 Feb 28 13:05 .
4 drwxr-xr-x 39 tracer tracer 4096 Feb 28 13:11 ..
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 12:57 dev
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 13:05 prod
The content of our dev environment:
$ ls -las dev
total 8
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 12:57 .
4 drwxrwxr-x 4 tracer tracer 4096 Feb 28 13:05 ..
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 12:57 .htaccess
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 12:57 index.html
When we try to use the rsync command in this way, it will give us an unexpected result:
$ rsync -av dev/* prod
sending incremental file list
index.html
As we can see, rsync omits the .htaccess file, or at least, it seems to be the case.
3. Solution
When we leave out the asterisk and run the command again:
rsync -av dev/ prod
sending incremental file list
./
.htaccess
index.html
We can see that even the hidden file .htacces is copied.
4. Explanation
In this case, our problem is related to the way the shell works. There is a mechanism called globbing, which expands asterisks to filenames before the rsync program is actually called.
Let’s assume we have a few more files:
ls -las dev
total 8
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 13:22 .
4 drwxrwxr-x 4 tracer tracer 4096 Feb 28 13:05 ..
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 12:57 .htaccess
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 13:22 about.html
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 13:22 home.html
0 -rw-rw-r-- 1 tracer tracer 0 Feb 28 12:57 index.html
Let’s issue the command:
rsync -av dev/* prod
The shell expands the content from the directory before sending it to the rsync command. In reality, the call looks like this:
rsync -av about.html home.html index.html prod
So, the globbing of the shell ignores dotfiles, which are also called hidden files. If we just add the directory itself as a parameter, no globbing takes place, and rsync even copies dotfiles.
5. Conclusion
In this article, we looked at how to include dotfiles when syncing folders with the rsync command.