1. Overview
Sometimes we want to transfer files from one Linux machine to another securely. Perhaps we want to upload some files to a production server or take a backup.
In this article, we’ll look at different tools for transferring files between Linux machines over ssh, the most popular protocol for remote connection between Linux machines.
We’ll look at the two most popular file transfer tools: scp and rsync.
2. Basic Tool Usage
2.1. A Quick Example
Let’s imagine we want to copy the file text.log from our local system to the path /var/log on a remote Linux server at IP address 111.111.111.111, and we want to use root user credentials.
This can be achieved with scp or rsync using the same basic syntax:
$ scp text.log [email protected]:/var/log
$ rsync text.log [email protected]:/var/log
Let’s understand what these commands have in common before we learn their variations and differences.
2.2. The General Syntax
The general syntax for scp or rsync is:
$ scp/rsync [OPTION] [user@][SRC_HOST:]Source_File [user@][DEST_HOST:]Destination_file
The OPTION section allows us to use command line switches to modify the behavior of the copy operation. The two paths for source and destination allow us to specify the path to the file at both sides of the copy operation, as well as the user account(s) to use.
In the above example, the local file is copied to a remote location — [email protected]. As we’ll see later, we can use the remote format to describe source or target, allowing these commands to both upload files to a server and download files from a server.
2.3. Important Things to Note
There are few points to remember when we’re transferring files over ssh using any tool:
- The scp/rsync command will ask for the user password to login to the remote server if we don’t have auto-login setup by exchanging RSA keys
- We don’t need to provide our username or server IP for our local server
- We can either provide an absolute path or relative path to the source/destination file, where relative paths are relative to the current working directory or the remote user’s home directory
2.4. Specifying a Filename
It’s not compulsory to provide a filename at the destination path. If we don’t provide a filename, the source file’s name will be used.
In our first example, the file will be created in the target directory with the name text.log. If we’d wanted the file to be stored with a different name, we could’ve provided a filename as part of the destination path.
Using the implicit filename, we can copy a file from remote Linux server into the current directory with its original name:
$ scp [email protected]:/var/log/text.log .
$ rsync [email protected]:/var/log/text.log .
Here, the “.” parameter represents the current working directory. We can also provide a different path if want:
$ scp [email protected]:/var/log/text.log /opt/path/mytextlog.log
$ rsync [email protected]:/var/log/text.log /opt/path/mytextlog.log
2.5. Copying Between Servers
We can even use these tools to copy between two remote servers. For example, let’s copy the text.log file from server 111.111.111.111. to server 222.222.222.222 for the root user:
$ scp [email protected]:/var/log/text.log [email protected]:/var/log
$ rsync [email protected]:/var/log/text.log [email protected]:/var/log
Now that we have a basic understanding of scp and rsync, let’s look at them in more depth.
3. scp (Secure Copy)
scp stands for Secure Copy and is used to transfer files over an ssh connection. It is a raw copy, meaning it will just read the data from the source folder and write it in the destination folder. So, if we are looking for a quick copy that shouldn’t take more than a few minutes, then we should go for scp.
If we’re transferring a large file, we should make sure that we have a stable internet connection. As scp is not resumable, time can be lost if the connection drops during a large transfer, and the entire copy operation will have to be restarted.
3.1. Installation
Most Linux distributions come with scp pre-installed. We can easily check if scp is installed or not:
$ which scp
We’ll see the path to the scp command if it’s already installed. An empty response means that scp is not installed.
If we don’t already have it, then to install scp, we need to install the OpenSSH client package from the official package repository:
$ sudo apt-get install openssh-clients # For Debian based systems like Ubuntu
$ sudo yum install openssh-clients # For Red Hat based systems like CentOS and Fedora
Let’s go through a few common scenarios for using scp.
3.2. Transfer Multiple Files
We can provide a list of files separated by space to transfer multiple files:
$ scp text.log text1.log text2.log [email protected]:/var/log
We can also use the -r option to copy the whole directory recursively. For example, to copy all files from the /home/app/log/ directory:
$ scp -r /home/app/log [email protected]:/var/log
We can use * to send files matching a particular naming pattern. For example, we have four files text.log, text2.log, app.log, and app2.log, and we want to send only files starting with app:
$ scp /home/app/log/app* [email protected]:/var/log
3.3. Transfer Files Using Identity Files
Key-based authentication is preferred in most Linux distributions. In the scp command, we can specify the identity file using the -i option:
$ scp -i key.pem text.log [email protected]:/var/log
Many system administrators use key-based authentication to transfer files since it doesn’t require the user credentials on every transfer.
3.4. On-the-Fly Compression
We can reduce transfer time by using on-the-fly compression with the -C option. This will compress and decompress the file at source and destination paths respectively:
$ scp -C text.log [email protected]:/var/log
3.5. Limit Bandwidth During Transfer
We can specify bandwidth in Kbit/s with the -l option:
$ scp -l 500 text.log [email protected]:/var/log
3.6. Verbose Mode for Debugging
We can enable verbose mode with the -v option:
$ scp -v text.log [email protected]:/var/log
The output would look something like:
Executing: program /usr/bin/ssh host prod-reporting-b, user jain, command scp -v -t /tmp
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to prod-reporting-b port 22.
debug1: Connection established.
------
Transferred: sent 2856, received 2932 bytes, in 2.1 seconds
Bytes per second: sent 1366.8, received 1403.1
debug1: Exit status 0
Verbose (debug) output prints the exact steps scp is executing, which is useful for debugging connections, authentication, and configuration problems.
4. rsync (Remote Synchronization)
rsync is fast and versatile. It’s good for synchronizing and smartly transferring files because it keeps track of how much data has been copied and how much is left.
We should use rsync when we have big files to transfer — even if there are network connection problems, data copied up to that point won’t be lost, and when the connection is restored, the copy will resume from that point.
rsync is faster than the scp command when we have to transfer the same file multiple times as it uses a remote-update protocol, which allows transferring just the differences between two sets of files.
4.1. Installation
Let’s check if rsync is installed by invoking it to show its version:
$ rsync --version
If we get an error, then we can install it:
$ sudo apt-get install rsync # For Debian based systems like Ubuntu
$ sudo yum install rsync # For Red Hat based systems like CentOS and Fedora
Let’s go through some common scenarios where we can use rsync.
4.2. Do a Dry Run
Since rsync is so powerful, it can be dangerous to use it without checking what it’s about to do. Fortunately, rsync provides the –dry-run option, which performs a trial run without actually transferring files:
$ rsync --dry-run text.log [email protected]:/var/log
4.3. Copy Attributes and Metadata
rsync provides the -a option, which copies symbolic links, modification times, group, ownership, and permissions along with the file content.
Another important option is –h, which will print output in a human-readable form. This can be especially helpful when running along with the –dry-run option:
$ rsync -ah --dry-run text.log [email protected]:/var/log
4.4. Transfer Multiple Files in a Directory
We can provide a list of files separated by a space to transfer multiple files:
$ rsync text.log text1.log text2.log [email protected]:/var/log
The -r option recursively copies all the files within the directory:
$ rsync -r /home/app/log [email protected]:/var/log
The * operator sends files matching a particular pattern:
$ rsync /home/app/log/app* [email protected]:/var/log
We can use the –include option and the –exclude option to include and exclude files from the list of files inside the directory. For example, to send all files from a directory excluding those starting with ‘R‘:
$ rsync --include '*' --exclude 'R*' text.log [email protected]:/urs/log
4.5. On-The-Fly Compression
We can reduce transfer time by using on-the-fly compression with the -z option. This will compress and decompress file at source and destination paths respectively:
$ rsync -z text.log [email protected]:/var/log
4.6. Delete Existing Files at Destination Path Before Transfer
The –delete option deletes existing files from the destination before transferring new files:
$ rsync --delete text.log [email protected]:/var/log
4.7. Automatically Remove Source Files After Transfer
Let’s say we want to take a backup of logs from the production server to a backup server. In this scenario, we would probably like to remove files from the production server after transferring all the files to our backup server.
rsync server provides the –remove-source-files option to remove source files after the transfer is complete:
$ rsync --remove-source-files text.log [email protected]:/var/log
5. Which Tool to Choose?
We can use both of these tools to transfer files between Linux machines. However, we should consider scp when the file size is small — it’s a simple copy tool and is widely available. In our day-to-day use, scp is the easy choice for small, one-off transfers.
We would benefit most from rsync when file size is big or when we need more complex and efficient synchronization. It’s also the better choice for recurring tasks like cron jobs and scripts.
6. Conclusion
In this article, we looked at two file transfer tools: scp and rsync.
We also looked at the differences between them, and which tools match certain use cases.