1. Overview
xargs (short for “eXtended ARGuments”) is a command on Unix and most Unix-like operating systems. It converts input from standard input (STDIN) into arguments to a command.
In this brief tutorial, we’ll see how we can use xargs to build and execute commands from standard input. We’ll be using BASH for our examples, so there could be slight differences with other shells.
2. A Typical Example
Sometimes, we may need to pass the output of one command as input for another. Some commands can take input either as command-line arguments or from the standard input.
However, there are others like cp, rm, and echo that can only take input as arguments. In such situations, we can use xargs to convert input coming from standard input to arguments.
Let’s now see an example that illustrates this.
Let’s say we need to remove all log files, older than seven days, from this log directory:
$ ls -l ./log
-rw-rw-rw- 1 someone someone 3645 Nov 19 2019 file1.log
-rw-rw-rw- 1 someone someone 22176 Oct 25 2019 file2.log
-rw-rw-rw- 1 someone someone 82051 Oct 25 2019 file3.log
-rw-rw-rw- 1 someone someone 1932647 Nov 6 2019 file4.log
-rw-rw-rw- 1 someone someone 225464 Jul 4 09:09 file5.log
-rw-rw-rw- 1 someone someone 3733 Jul 4 09:09 file6.log
To find files older than seven days, we’ll use the find command with -mtime option:
$ find ./log -type f -mtime +7
./log/file1.log
./log/file2.log
./log/file3.log
./log/file4.log
Now, let’s use the pipe operator in order to send the output of find as the input for rm:
$find . -type f -mtime +7 | rm
rm: missing operand
Try 'rm --help' for more information.
This prints an error message since rm expects arguments and can’t read them from STDIN.
Listing the log directory still shows the same number of files, as the rm command failed:
$ ls -l ./log
-rw-rw-rw- 1 someone someone 3645 Nov 19 2019 file1.log
-rw-rw-rw- 1 someone someone 22176 Oct 25 2019 file2.log
-rw-rw-rw- 1 someone someone 82051 Oct 25 2019 file3.log
-rw-rw-rw- 1 someone someone 1932647 Nov 6 2019 file4.log
-rw-rw-rw- 1 someone someone 225464 Jul 4 09:09 file5.log
-rw-rw-rw- 1 someone someone 3733 Jul 4 09:09 file6.log
Instead, let’s use xargs* along with *rm:
$ find . -type f -mtime +7 | xargs rm
$ ls -l ./log
-rw-rw-rw- 1 someone someone 225464 Jul 4 09:09 file5.log
-rw-rw-rw- 1 someone someone 3733 Jul 4 09:09 file6.log
Now rm removes files older than seven days.
Let’s now see other common scenarios for xargs.
3. Other Common Uses
3.1. Limit Output per Line
To find out what xargs does with the output from the find command, let’s add echo before the rm command:
$ find . -type f -name "*.log" | xargs -n 1 echo rm
rm ./log/file5.log
rm ./log/file6.log
Because we added the -n 1 argument, xargs turns each line into a command of its own.
3.2. Enable User Prompt Before Execution
To prompt the user with y (yes) and n (no) options before execution, let’s use the -p option:
$ find ./log -type f -name "*.log" | xargs -p -n 1 rm
rm ./log/file5.log ?...y
rm ./log/file6.log ?...n
Since we have provided the ‘y’ option, file5.log has now been removed:
$ ls -l ./log
-rw-rw-rw- 1 someone someone 3733 Jul 4 09:09 file6.log
3.3. Insert Arguments at a Particular Position
The xargs command offers options to insert the listed arguments at some arbitrary position other than the end of the command line.
The -I option takes a string that gets replaced with the supplied input before the command executes. Although this string can be any set of characters, a common choice for it is “%”.
Let’s move file6.log to the backup directory:
$ find ./log -type f -name "*.log" | xargs -I % mv % backup/
$ ls -l ./log
total 0
$ ls -l ./backup/
-rw-rw-rw- 1 someone someone 3733 Jul 4 09:09 file6.log
3.4. Enable Multiple Process Usage
To parallelize operations, we can use the -P option to specify the number of parallel processes used in executing the commands over the input argument list.
Let’s use it to parallel encode a series of wav files to mp3 format:
$find . -type f -name '*.wav' -print0 |xargs -0 -P 3 -n 1 mp3 -V8
The encoding process executes using three processes, since -P 3 is specified.
4. Handling Files with Special Characters in the Names
A filename can contain special characters like single quotes, double quotes, or spaces.
Let’s see how to handle those files when passed to xargs.
4.1. Filename Contains Spaces
Let’s check if xargs passes the file file 1.log, to the rm command, despite the space in the name:
$ ls -l ./log
-rw-rw-rw- 1 someone someone 3645 Nov 19 2019 file 1.log
$ find ./log -type f | xargs rm
rm: cannot remove ‘./log/file’: No such file or directory
rm: cannot remove ‘1.log’: No such file or directory
As we can see, xargs sends two arguments ./log/file and 1.log to the rm command instead of a single argument ./log/file 1.log. Of course, there are no files with name file and 1.log, so the rm command gives an error message No such file or directory.
To make xargs recognize the files with spaces in their names, again, we will use the -I option. But we must quote the placeholder:
$ find ./log -type f | xargs -I % rm "%"
$ ls -l ./log
total 0
The file 1.log has been deleted now.
4.2. Filename Contains Quotes
Let’s start with another example:
$ find ./log -type f
./log/file"2.log
./log/file'1.log
We have created two log files. The file file’1.log has a single quote in the name, while the file”2.log has a double quote character in the name.
Next, let’s pass the output of find to the xargs command and try to remove the two files:
$ find ./log -type f | xargs rm
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Let’s see if quoting the placeholder of the xargs‘s -I option helps:
$ find ./log -type f | xargs -I % rm '%'
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
$ find ./log -type f | xargs -I % rm "%"
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
The test shows, if a file contains a single or double quote character in its name, xargs cannot work by quoting the filename.
To deal with this problem, both xargs and the command passing the output to xargs must use a null character as the record separator.
We can pass the option -print0 to the find command to ask it to output filenames separated by a null character instead of the default newline.
On the xargs command side, since the default record separator is a newline character as well, we need the -0 option to use a null character as a record separator:
$ find ./log -type f -print0 | xargs -0 rm
$ ls -l ./log
total 0
The output above shows the two files have been deleted.
Note, however, that some commands cannot use a null character as a separator (for example, head, tail, ls, echo, sed, and wc). In these cases, xargs cannot handle the output of such commands if the output contains quote marks.
5. Conclusion
In this brief tutorial, we saw the use of xargs to build and execute commands from standard input.
It is used in situations when a command can take input from other commands, only in the form of arguments.
It is also particularly useful when we have a list of file paths on STDIN and want to do something with them.
Finally, we saw that the xargs command has its own limitation with files having a single quote or double quote in the name.