1. Overview
In this tutorial, we’ll look at several methods to sync files one-way periodically or when we detect some changes in the files or directories.
We’ll be using a cron job to trigger the sync process with rsync/unison/FreeFileSync periodically and inotifywait or lsyncd to sync on change.
2. Sync Files Periodically
Let’s start with syncing files periodically with a cron job. First, we create a sync script. Then we set up a cron job to run the script every minute.
2.1. Creating a Sync Script
Let’s create a simple script and save it in sync-script.sh:
#!/bin/sh
date >> sync-script.log
The script writes the system date and time to sync-script.log file in append mode (notice the double greater-than sign >>, where the single > would overwrite the file). Later on, we’ll add some file synchronization to this script, but for now, let’s focus on getting it to work and scheduling it to run.
Next, we update our script file permissions to be executable, then run it:
$ sudo chmod 775 sync-script.sh
$ ./sync-script.sh
The script that we ran wrote the system date and time to sync-script.log:
$ cat sync-script.log
Thu 30 Jun 2022 02:14:45 PM
2.2. Setting up a Cron Job
Next, let’s create a cron job to run the script file every minute:
$ crontab -e
*/1 * * * * /bin/sh ~/sync-script.sh >> ~/sync-job.log 2>&1
We can check the script log to ensure the job is running as expected:
$ cat sync-script.log
Thu 30 Jun 2022 02:14:45 PM
Thu 30 Jun 2022 02:25:01 PM
Thu 30 Jun 2022 02:25:02 PM
Thu 30 Jun 2022 02:25:03 PM
...
Thu 30 Jun 2022 02:38:01 PM
We store any error message during script execution in the job log file:
$ cat sync-job.log
At this point, we now have a working cron job that runs the script file every minute.
3. Sync Periodically Using rsync
Now, let’s update our script to sync files periodically using rsync.
3.1. Updating the Script File
We’ll replace the date command with rsync:
#!/bin/sh
rsync -acu --delete ~/sample/source/ ~/sample/target/ >> ~/sync-script.log
Let’s review the rsync options that we used:
- -a: enable archive mode; equivalent to –rlptgoD (no –H,-A,-X)
- -c: skip object based on checksum, not on modified time and size
- -u: skip newer files on the target
- –delete: delete files from target not exist in source
3.2. Testing the Script File
We’ll now run the script to test it:
$ pwd
/home/baeldung
$ tree sample
sample
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file04.bin
└── target
3 directories, 3 files
$
$ ./sync-script.sh
$
$ tree sample
sample
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file04.bin
└── target
├── file01.bin
├── file02.bin
└── subdir1
└── file04.bin
4 directories, 6 files
Let’s delete a file (~/sample/source/file01.bin) from the source to see if it will also delete the same file from the target:
$ rm sample/source/file01.bin
$ tree sample
sample
├── source
│ ├── file02.bin
│ └── subdir1
│ └── file04.bin
└── target
├── file01.bin
├── file02.bin
└── subdir1
└── file04.bin
4 directories, 5 files
$ ./sync-script.sh
$ tree sample
sample
├── source
│ ├── file02.bin
│ └── subdir1
│ └── file04.bin
└── target
├── file02.bin
└── subdir1
└── file04.bin
4 directories, 4 files
The script removed the files in the target directory successfully. Our ~/sample/source directory will now sync to ~/sample/target directory every minute.
4. Sync Periodically Using Unison
Unison is another powerful file-synchronization tool. However, unlike rsync, besides having a one-way (mirror) mode, it also has a two-way synchronization mode.
Let’s update our sync script to use Unison command in mirror mode:
#!/bin/sh
unison -batch=true ~/sample/source/ ~/sample/target/ \
-force ~/sample/source/ >> /home/baeldung/sync-script.log
The Unison command options that we use:
- -batch=true: batch/non-interactive mode. Unison will use default settings without any interactive prompt
- -force: sync in mirror mode
Our cron job will still run our script every minute. Only this time, the script will be using Unison command.
5. Sync Periodically Using FreeFileSync
FreeFileSync is a GUI-based file-synchronization tool. We need to generate the ffs_batch file, which contains our synchronization settings, in order to use it in batch/non-interactive mode for our script:
$ which FreeFileSync
/home/baeldung/.local/bin/FreeFileSync
$ vi sync-script.sh
#!/bin/sh
$ ./home/baeldung/.local/bin/FreeFileSync BatchRun.ffs_batch >> /home/baeldung/sync-script.log
We store our sync settings in the BatchRun.ffs_batch file and use it as a parameter to the FreeFileSync binary file.
6. Sync on Change Using inotifywait
Instead of using a cron job to run the sync command periodically, which may cause memory or processor overhead, let’s now look at using inotifywait to do file sync only when we make some changes to the file or directory.
6.1. Installing inotifywait
inotifywait is part of the inotify-tools package, which is available on many Linux core repositories. For example, on Debian/Ubuntu, we can run:
$ apt install inotify-tools
On Fedora:
$ dnf -y install inotify-tools
On Arch and CentOS/RHEL:
$ yum install -y epel-release && yum update
$ yum install inotify-tools
The inotify repository has a complete list of Linux distros that have the inotify-tools package.
6.2. Using inotifywait
Let’s create a script to use inotifywait to run the sync command only when there’s a change to the file or directory:
#!/bin/bash
if [ -z "$(which inotifywait)" ]; then
echo "inotifywait not installed."
echo "In most distros, it is available in the inotify-tools package."
exit 1
fi
counter=0;
function execute() {
counter=$((counter+1))
echo "Detected change n. $counter"
eval "$@"
}
inotifywait --recursive --monitor --format "%e %w%f" \
--event modify,move,create,delete /home/baeldung/sample/source \
| while read changed; do
echo $changed
execute "$@"
done
The script above waits for filesystem events (CREATE, MODIFY, MOVE, DELETE) and acts accordingly by running the execute function.
Let’s update the execute function to run the rsync/Unison/FreeFileSync command from the previous section to sync our directories:
function execute() {
counter=$((counter+1))
echo "Detected change n. $counter"
eval "$@"
if test $counter -gt 100
then
rsync -cau /home/baeldung/sample/source/ /home/baeldung/sample/target/ \
>> /home/baeldung/sync-script.log 2>&1
fi
}
Since we don’t want to run the sync command every time we make some changes, we configured the script to run the sync command after there have been over 100 events.
6.3. Troubleshooting a Lack of Events
Our script should work just fine, but what will happen if we rarely change the file or directory and inotifywait receives less than 100 events? Our directories won’t be in sync until we make enough changes.
To handle this case, we have a few options:
- update our script, so we only wait for fewer events
- add a timer to sync the directories every few minutes
- combine the script with a cron job that runs every 30 minutes or hourly
7. Sync on Change Using lsyncd
lsyncd is a daemon that uses the filesystem event interface (inotify or fsevents) to detect changes to local files or directories. It uses rsync as the default synchronization method.
7.1. Installing lsyncd
lsyncd is available on many Linux core repositories. For example, on Debian or Ubuntu, we can run:
$ apt install lsyncd
On Fedora:
$ dnf -y install lsyncd
On CentOS/RHEL 7:
$ yum install lsyncd
We can also download lsyncd from their website directly. We should note that lsyncd 2.2.1 or later requires rsync >= 3.1 on all source and target machines.
7.2. Using lsyncd
Let’s set up a local lsync:
$ lsyncd -rsync /home/baeldung/sample/source /home/baeldung/sample/target
If the target directory is on a different machine:
$ lsyncd -rsyncssh /home/baeldung/sample/source/ remotehost.org target-path/
The command above will copy/mirror the source directory recursively to the target directory:
$ tree sample
sample
└── source
├── file01.bin
├── file02.bin
└── subdir1
└── file03.bin
2 directories, 3 files
$ lsyncd -rsync /home/baeldung/sample/source /home/baeldung/sample/target
15:41:03 Normal: --- Startup, daemonizing ---
$ tree sample
sample
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file03.bin
└── target
├── file01.bin
├── file02.bin
└── subdir1
└── file03.bin
4 directories, 6 files
If we edit files in the source directory, lsyncd will automatically reflect it in the target directory:
$ cat sample/source/file01.bin
This is line 1
$ cat sample/target/file01.bin
This is line 1
$ echo "This is line 2" >> sample/source/file01.bin
$ cat sample/source/file01.bin
This is line 1
This is line 2
$ cat sample/target/file01.bin
This is line 1
...
$ cat sample/target/file01.bin
This is line 1
$ cat sample/target/file01.bin
This is line 1
This is line 2
lsyncd aggregates events up to 1000 separate events, or a 15-second delay before synchronizing, whichever happens first, so our changes may not be synced immediately.
7.3. lsyncd Options
Let’s try configuring lsyncd by creating its config file lsyncd-config.cfg:
settings{
logfile = "/home/baeldung/lsyncd-log.log",
statusFile = "/home/baeldung/lsyncd-status.stat",
}
sync{
default.rsync,
source="/home/baeldung/sample/source",
target="/home/baeldung/sample/target",
}
We pass the config file as a parameter to the lsyncd command:
$ lsyncd lsyncd_config.cfg
15:56:54 Normal: --- Startup, daemonizing ---
$ ls -l lsyncd*
-rw-r--r-- 1 baeldung baeldung 275 Jul 2 15:55 lsyncd_config.cfg
-rw-r--r-- 1 baeldung baeldung 356 Jul 2 15:56 lsyncd-log.log
-rw-r--r-- 1 baeldung baeldung 287 Jul 2 15:56 lsyncd-status.stat
Let’s review the options that we used in the config file:
- settings: global config options which will be used by all syncs
- logfile: log file path
- statusFile: status file path
- sync: config options for each sync process
- default.rsync: file synchronization tool
- source: source directory
- target: target directory
7.4. Sync From One Source to Multiple Targets
lsyncd has the feature of syncing from one source to multiple targets. Let’s update our config file:
settings{
logfile = "/home/baeldung/lsyncd-log.log",
statusFile = "/home/baeldung/lsyncd-status.stat",
}
sync{
default.rsync,
source="/home/baeldung/sample/source",
target="/home/baeldung/sample/target",
}
sync{
default.rsync,
source="/home/baeldung/sample/source",
target="/home/baeldung/sample/target2",
}
sync{
default.rsync,
source="/home/baeldung/sample/source",
target="/home/baeldung/sample/target3",
}
Then we run lsyncd with the new config file:
$ tree sample
sample
└── source
├── file01.bin
├── file02.bin
└── subdir1
└── file03.bin
2 directories, 3 files
$ lsyncd lsyncd-config.cfg
16:08:58 Normal: --- Startup, daemonizing ---
$ tree sample
sample
├── source
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file03.bin
├── target
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file03.bin
├── target2
│ ├── file01.bin
│ ├── file02.bin
│ └── subdir1
│ └── file03.bin
└── target3
├── file01.bin
├── file02.bin
└── subdir1
└── file03.bin
8 directories, 12 files
The lsyncd website has a complete list of config options with detailed explanations and examples.
8. Conclusion
In this article, we saw how to continuously sync files one-way periodically or on change.
We first utilized a cron job to periodically run file sync commands, such as rsync/Unison/FreeFileSync.
Then we used inotifywait to detect filesystem events as a trigger to run the file sync command.
Lastly, we used the lsyncd tool, which uses the filesystem event interface (inotify or fsevents) to detect changes, as well as a periodic sync, which provided the best combination of all techniques.