1. Overview
Syncronizing Linux directories in both directions ensures changes made in one directory are mirrored or updated in another, and vice versa.
In this tutorial, we’ll test several commands or programs to synchronize directories in both directions on a single machine, and then compare the results.
All commands mentioned in this guide have been tested on 64-bit Debian 10.10 (Buster), running GNU Bash 5.1.4, rsync 3.1.3, 64-bit Unison 2.51.4 with OCaml 4.05.0 binary, Zaloha2 Bash script, and FreeFileSync 11.16.
2. Using the rsync Command
2.1. Installation
rsync is commonly found on Unix-like operating systems. Its source code or binary can also be downloaded from its homepage.
2.2. Usage
rsync works in one direction, so we need to run it twice to sync directories in both directions.
To illustrate the sync operation, let’s first create a sample directory structure:
$ mkdir source target
$ touch source/file01.bin source/file02.bin source/file03.bin
$ touch target/file0A.bin target/file0B.bin target/file0C.bin
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── file03.bin
└── target
├── file0A.bin
├── file0B.bin
└── file0C.bin
2 directories, 6 files
Now that we have a sample directory, let’s sync from source to target:
$ rsync -Pcauv source/ target/
sending incremental file list
./
file01.bin
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=2/4)
file02.bin
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=1/4)
file03.bin
0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=0/4)
sent 297 bytes received 76 bytes 746.00 bytes/sec
total size is 0 speedup is 0.00
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── file03.bin
└── target
├── file01.bin
├── file02.bin
├── file03.bin
├── file0A.bin
├── file0B.bin
└── file0C.bin
2 directories, 9 files
Then, let’s sync from target to source:
$ rsync -Pcauv target/ source/
sending incremental file list
file0A.bin
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=2/7)
file0B.bin
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=1/7)
file0C.bin
0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=0/7)
sent 401 bytes received 73 bytes 948.00 bytes/sec
total size is 0 speedup is 0.00
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ ├── file03.bin
│ ├── file0A.bin
│ ├── file0B.bin
│ └── file0C.bin
└── target
├── file01.bin
├── file02.bin
├── file03.bin
├── file0A.bin
├── file0B.bin
└── file0C.bin
2 directories, 12 files
As a result of the rsync command that we ran twice in both directions, both directories are now identical.
Let’s review the options that we used:
- -P: display statistics of the whole transfer rather than individual files
- -c: skip object based on checksum, not on modified time and size
- -a: enable archive mode, equals to:
- -r: recurse into directories
- -l: copy symlinks as symlinks
- -p: preserve permissions
- -t: preserve modification times
- -g: preserve group
- -o: preserve owner
- -D: preserve device files and special files
- -u: skip newer files on the target
- -v: display detailed information of what is being done
Additionally, we can also do a dry run by passing the -n option, which will simulate the sync with no changes made:
$ rsync -Pcauvn source/ target/
sending incremental file list
./
file01.bin
file02.bin
file03.bin
sent 189 bytes received 28 bytes 434.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
The output in the last line, (DRY RUN), indicates that it’s a dry run. This is particularly useful if we need to verify that the command will succeed without any errors, such as incorrect options, or insufficient disk space, or any other potential issues.
3. Using Unison Program
3.1. Installation
Unison has its own homepage, however, it hosts the source code and binaries on a GitHub repository.
We’re using 64-bit Unison 2.51.4 with OCaml 4.05.0 binary, which gives us this package:
$ tree
.
├── bin
│ ├── unison
│ ├── unison-fsmonitor
│ └── unison-gtk2
├── LICENSE
└── README.md
1 directory, 5 files
The package provides 3 executable files: text-based (unison), GUI-based (unison-gtk2), and file system monitoring (unison-fsmonitor). However, for this article, we’ll use the text-based Unison.
3.2. Usage
To demonstrate its usage, let’s say we have the following structure of source and target directories:
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── file03.bin
└── target
├── file0A.bin
├── file0B.bin
└── file0C.bin
2 directories, 6 files
Whenever we sync a new source and target directory for the first time, which Unison refers to as a profile, we’ll get this interactive prompt:
$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
~/source
~/target
This can happen either because this is the first time you have synchronized these roots, or because you have upgraded Unison to a new version with a different archive format.
...
Donations to the Unison project are gratefully accepted: http://www.cis.upenn.edu/~bcpierce/unison
Press return to continue.[<spc>]
We can press Enter to continue, or q to quit. If we press Enter, it will ask for confirmation for each change that we want to do.
We press f or Enter to confirm each change. We can press > or < to force the change to be propagated from left to right, or right to left, or press / to skip the file and leave both objects alone.
When we reach the end of the list, Unison will ask one more time if we want to proceed with the changes, and we can press y to proceed, or q to terminate.
Every time Unison asks for input, we can press ? to list all possible responses and their meanings:
Reconciling changes
source target
file ----> file01.bin [f] f
file ----> file02.bin [f] f
file ----> file03.bin [f] f
<---- file file0A.bin [f] f
<---- file file0B.bin [f] f
<---- file file0C.bin [f] f
Proceed with propagating updates? [] y
Propagating updates
Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 15:29:09.85 on 06 Jan 2022
[BGN] Copying file01.bin from ~/source to ~/target
[END] Copying file01.bin
[BGN] Copying file02.bin from ~/source to ~/target
[END] Copying file02.bin
[BGN] Copying file03.bin from ~/source to ~/target
[END] Copying file03.bin
[BGN] Copying file0A.bin from ~/target to ~/source
[END] Copying file0A.bin
[BGN] Copying file0B.bin from ~/target to ~/source
[END] Copying file0B.bin
[BGN] Copying file0C.bin from ~/target to ~/source
[END] Copying file0C.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 15:29:09.86 on 06 Jan 2022, 0.012 s
Saving synchronizer state
Synchronization complete at 15:29:09 (6 items transferred, 0 skipped, 0 failed)
Once it’s done, if we try to sync again, it’ll say nothing to sync:
$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes
Nothing to do: replicas have not changed since last sync.
Let’s delete and modify some files, then sync again:
$ rm source/file0B.bin
$ echo "edit file" > source/file0C.bin
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ ├── file03.bin
│ ├── file0A.bin
│ └── file0C.bin
└── target
├── file01.bin
├── file02.bin
├── file03.bin
├── file0A.bin
├── file0B.bin
└── file0C.bin
2 directories, 11 files
$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes
source target
deleted ----> file0B.bin [f]
changed ----> file0C.bin [f]
Proceed with propagating updates? [] y
Propagating updates
Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 15:50:16.93 on 06 Jan 2022
[BGN] Updating file file0C.bin from ~/source to ~/target
[END] Updating file file0C.bin
[BGN] Deleting file0B.bin from ~/target
[END] Deleting file0B.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 15:50:16.93 on 06 Jan 2022, 0.003 s
Saving synchronizer state
Both directories are now identical.
3.3. Non-interactive Mode
If we want to immediately proceed to sync with default Unison settings without any interactive prompt, where non-conflicting changes will be propagated and conflicts will be skipped, we can pass -batch=true as an option:
$ ./unison -batch=true ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes
changed ----> file0C.bin
source : changed file modified on 2022-01-06 at 16:40:13 size 11 rw-r--r--
target : unchanged file modified on 2022-01-06 at 16:28:13 size 11 rw-r--r--
Propagating updates
Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 16:40:35.93 on 06 Jan 2022
[BGN] Updating file file0C.bin from ~/source to ~/target
[END] Updating file file0C.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 16:40:35.93 on 06 Jan 2022, 0.000 s
Saving synchronizer state
Synchronization complete at 16:40:35 (1 item transferred, 0 skipped, 0 failed)
By using the batch option, we can create a script or a cron job to do the sync automatically and periodically.
4. Using Zaloha2 Bash Script
4.1. Installation
Zaloha2 is a small and simple directory synchronizer in a form of a Bash script. We can download it from its GitHub repository.
Once we have downloaded the project, we’ll get the following package:
$ tree
.
├── DOCUMENTATION.md
├── LICENSE
├── README.md
├── Simple_Demo_screenshot.png
├── Simple_Demo_step1.sh
├── Simple_Demo_step2.sh
├── Simple_Demo_step3.sh
├── Simple_Demo_step4.sh
├── Simple_Demo_step5.sh
├── Simple_Demo_step6.sh
├── Simple_Demo_step7.sh
└── Zaloha2.sh
0 directories, 12 files
We may need to make Zaloha2.sh executable to be able to use it:
$ sudo chmod u+x Zaloha2.sh
$ ls -la
total 528
drwxr-xr-x 4 baeldung baeldung 4096 Jan 23 14:58 .
drwxr-xr-x 4 baeldung baeldung 4096 Jan 23 13:44 ..
...
-rwxr--r-- 1 baeldung baeldung 222899 Jan 23 11:08 Zaloha2.sh
As we can see for the above ls output, Zaloha2.sh now has the x permission. As a result, we can execute the script file as a program or script.
4.2. Usage
Using Zaloha2 is pretty straightforward. First, let’s say we have the following directory structure:
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ ├── file03.bin
│ ├── subdir1
│ │ └── file04.bin
│ └── subdir2
│ └── file05.bin
└── target
├── file0A.bin
├── file0B.bin
├── file0C.bin
├── subdir3
│ └── file0D.bin
└── subdir4
└── file0E.bin
6 directories, 10 files
We then run the sync process:
$ ./Zaloha2.sh --sourceDir="/home/baeldung/source" --backupDir="/home/baeldung/target" --color --revUp --revNew --noRemove --sha256 --findSourceOps='( -type d -a -name subdir ) -prune -o' --findSourceOps='( -type d -a -name .svn ) -prune -o' --findSourceOps='( -type d -a -name .cache ) -prune -o' --findGeneralOps=
...
Execute above listed copies to /home/baeldung/target/ ?
[Y/y=Yes, S/s=do nothing and show further, other=do nothing and abort]: y
...
$
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ ├── file03.bin
│ ├── subdir1
│ │ └── file04.bin
│ └── subdir2
│ └── file05.bin
└── target
├── file01.bin
├── file02.bin
├── file03.bin
├── file0A.bin
├── file0B.bin
├── file0C.bin
├── subdir1
│ └── file04.bin
├── subdir2
│ └── file05.bin
├── subdir3
│ └── file0D.bin
└── subdir4
└── file0E.bin
8 directories, 15 files
Let’s examine the options that we use:
- –sourceDir: source directory
- –backupDir: target directory
- –color: to colorize the changes during interactive prompt
- –revUp: reverse-update; if there are newer objects in target, copy them to the source
- –revNew: reverse-new; if there are objects in the target that don’t exist in the source and they are newer than the last run of Zaloha2, copy them to the source
- –noRemove: do not remove objects in target; only add or update existing objects in the target
- –sha256: besides comparing file size and modified date, check file SHA256 checksum
- –noProgress: do not display messages from the analysis phase
- –findSourceOps: exclude certain sub directories from sync. If such sub directory exists in the target, it will be deleted
- –findGeneralOps: if it is empty, it will disable internally defined defaults that exclude certain directories (“trash” or “lost+found”). It can also be used instead of findSourceOps to generally exclude files or directory
Unlike rsync or Unison, Zaloha2 doesn’t reverse copy files or directories from target to source initially, unless those files or directories are newer than the last run of Zaloha2.
4.3. Reverse Copy
Let’s update some files in the target directory, then run Zaloha2 again to trigger reverse copy:
$ echo "edit" > target/file0B.bin
$ echo "edit" > target/file0C.bin
$ echo "edit" > target/subdir3/file0D.bin
$
$ ./Zaloha2.sh --sourceDir="/home/baeldung/source" --backupDir="/home/baeldung/target" --color --revUp --revNew --noRemove --sha256 --findSourceOps='( -type d -a -name subdir ) -prune -o' --findSourceOps='( -type d -a -name .svn ) -prune -o' --findSourceOps='( -type d -a -name .cache ) -prune -o' --findGeneralOps=
...
Execute above listed reverse-copies to /home/baeldung/source/ ?
[Y/y=Yes, S/s=do nothing and show further, other=do nothing and abort]: y
...
$
$ tree
.
├── source
│ ├── file01.bin
│ ├── file02.bin
│ ├── file03.bin
│ ├── file0B.bin
│ ├── file0C.bin
│ ├── subdir1
│ │ └── file04.bin
│ ├── subdir2
│ │ └── file05.bin
│ └── subdir3
│ └── file0D.bin
└── target
├── file01.bin
├── file02.bin
├── file03.bin
├── file0A.bin
├── file0B.bin
├── file0C.bin
├── subdir1
│ └── file04.bin
├── subdir2
│ └── file05.bin
├── subdir3
│ └── file0D.bin
└── subdir4
└── file0E.bin
9 directories, 18 files
As we can see from the output of the tree command above, the changes in the target directory were copied to the source directory, though both directories are still not identical. This is because those files or directories in the target directory aren’t newer than the last run of Zaloha2.
4.4. Non-interactive Mode
If we want to set up automatic/non-interactive mode, there are steps that are a bit complicated, albeit flexible, to implement, which the author suggests in the documentation.
Firstly, we need to run the command with an extra option –noExec, which will only generate the script files in the .Zalora_metadata directory in the target directory. Then, we need to write a wrapper script to execute those scripts. Please refer to the documentation in the Automatic Operations section.
Zaloha2 project is still in active development — use with caution.
5. Using FreeFileSync Program
5.1. Installation
FreeFileSync is an open-source, multi-platform software. It has a GUI interface, but also provides a feature to do batch/non-interactive/text-based mode.
We can download the FreeFileSync installer script from its homepage and install it. The installer will also create the Desktop shortcut which we can use to run the program.
5.2. Usage
This is the main UI of FreeFileSync:
The interface is quite intuitive:
- Change comparison settings, such as compare based on the file modified time and size, or file content, or file size.
- Start comparison between the two directories.
- Filter: we can include or exclude certain files or directories based on the file name, extension, size, or modified time.
- Change synchronization settings: we can synchronize in two directions (Two way), one-way (Mirror), update only (Update), or custom.
- Start synchronization.
- Source directory.
- Target directory.
5.3. Non-interactive Mode
If we want to create a batch/non-interactive mode, first we need to select the source and target directories from the main UI, then click on menu File > Save as batch job…, which will show the following dialog:
We need to disable the Progress dialog option and enable the Ignore errors option. Otherwise, it’ll display the interactive/error message box and will wait for interaction indefinitely.
After we created the batch job file, e.g. BatchRun.ffs_batch, we need to set FreeFileSync to run it.
Let’s create a cron job:
$ crontab -e
0 1 * * * <FreeFileSync installation path>/FreeFileSync <batch file path>/BatchRun.ffs_batch
The cron job will run the sync every day at 1AM.
6. Comparison
6.1. Features
Here is a list of common features shared by the tools:
==============================================================================================================================================
Tool Language License Bidirectional Delta Copying Detect Conflict Propagate Deletes Revision Control Scheduling or Service
----------------------------------------------------------------------------------------------------------------------------------------------
rsync C GPL v3 Yes Yes Yes Yes Yes Yes via OS
Unison Mainly OCaml GPL Yes Yes Yes Yes Yes Yes via OS
Zaloha2 Bash MIT Yes Yes No Yes Yes Yes via OS
FreeFileSync C++ GPL Yes Yes Yes Yes Yes Yes via OS
==============================================================================================================================================
6.2. Benchmark
We measured the performance of each command on the following system and data:
- Dell Latitude Intel Core i7-6600U CPU @ 2.60GHz × 4, 16GB RAM
- Source directory: Seagate external HDD 5TB, 7200rpm, SATA III, ext4
- Target directory: Seagate external HDD 1.5TB, 7200rpm, SATA III, NTFS
- Data synchronized: 10GB, 11,780 image files
- OS: Debian Buster
The number of files and total size of the source directory:
$ find source -type f | wc -l
11780
$
$ du source -hc --max-depth=0
1.3G source/subdir1
3.3G source/subdir2
5.5G source/subdir3
10G source
10G total
For the test, we synced both directories three times. We ran each command for three times, except for rsync. We ran rsync six times – twice for each sync – because it is not bidirectional:
- for the initial sync, we synced/mirrored data between source and target directories
- for the second sync, all tools returned immediately as there’s nothing to sync
- finally for the third sync, first we deleted some data in source (delete subdir1), then added some data in target (copy subdir1 and name it subdir4), then we synced the directories.
The options that we used for each command:
- rsync
$ rsync -PScauv --info=progress2 source/ /media/baeldung/target/ $ rsync -PScauv --info=progress2 target/ /media/baeldung/source/ - Unison
$ ./unison -batch=true /media/baeldung/source/ /media/baeldung/target/ - Zaloha2
$ ./Zaloha2.sh --sourceDir="/media/baeldung/source" --backupDir="/media/baeldung/target" --color --revUp --revNew --noRemove --noRestore --mawk --findGeneralOps= - FreeFileSync
$ which FreeFileSync /home/baeldung/.local/bin/FreeFileSync $ ./home/baeldung/.local/bin/FreeFileSync BatchRun.ffs_batch
Results:
=================================================================
Tool First Sync Second Sync Third Sync
-----------------------------------------------------------------
rsync 15:44 00:01 15:48
Unison 07:31 00:01 00:04
Zaloha2 05:22 00:01 Failed (no reverse update)
FreeFileSync 05:10 00:01 00:13
=================================================================
7. Conclusion
Of all four tools that we tested, we found that Unison is the best for synchronizing directories in both directions.
Unison may take a bit longer when we run it for the first time, but the subsequent run was the fastest compared to the other tools in terms of performance. It supports conflict detection, is resilient to failure (it has revision control, useful for rollback), is bidirectional, and has a command-line as well as a GUI interface.
Coming in a close second is FreeFileSync, which is also great with its intuitive GUI.