1. Overview
When working with the Linux command line, we use many commands that receive important data as arguments.
In this tutorial, we’ll explore some scenarios on how we can use the output of programs as arguments to others.
2. Preparing Example Files
First, let’s create an environment where we can test our strategies.
Let’s create two folders – one that contains files created with certain specifications, and another that we’ll use as the destination for our tests:
$ mkdir dir_example target
Next, let’s create two sets of files with sizes of ten and four bytes, respectively:
$ truncate -s 10 dir_example/file{1..3}.this
$ truncate -s 4 dir_example/file{1..2}.not
Here, we’ve used the truncate command to create two sets of files, with a specific size, filled with the null character.
We added a suffix in the form .this or .not to the files, to visually differentiate them by their sizes for our examples.
Note: after we go through each example, we must empty the destination folder:
$ rm target/*
3. The read Builtin
Some shells, like Bash and Zsh, have the read builtin that reads content from the input and stores it in a variable.
Let’s copy the files whose size is greater than five bytes from the dir_example folder to the target folder:
$ find dir_example/ -type f -size +5c | while read file; do cp "$file" target/; done
Here, we’re using the find command with the -type f argument to search for regular files. The -size +5c argument is used to find only files with sizes greater than five bytes.
Next, we’re using the output of the find command to feed the while loop. The read builtin then reads from the input and assigns the first word to the file variable.
Finally, using the file variable as a parameter in the cp command, we can copy the file with the same name into the destination folder:
$ ls -l target/
total 0
-rwxrwxrwx 1 cuau cuau 10 Nov 19 11:37 file1.this
-rwxrwxrwx 1 cuau cuau 10 Nov 19 11:37 file2.this
-rwxrwxrwx 1 cuau cuau 10 Nov 19 11:37 file3.this
4. Command Substitution
Several shells have a mechanism called command substitution that allows the output of a command to replace the command name.
Let’s use this capability to copy the files greater than five bytes to the target folder:
$ cp $(find dir_example/ -type f -size +5c) target/
In this scenario, we’re using command substitution to feed the list of source files as parameters of the cp command. Then, we copy them to the destination folder indicated in the last argument.
5. Process Substitution
Similar to our last scenario, several shells have another mechanism called process substitution, which connects the input or output of a list to a FIFO. Then, the commands use the name of this FIFO.
Since awk accepts only a filename as an argument instead of simple words, let’s use this capability within an awk script to count the size of our two sets of files:
$ awk '{hist[ARGIND]++}
END {
for (i in hist)
printf "%s records in process substitution %s\n", hist[i] , i
}' \
<(find dir_example/ -type f -size +5c) \
<(find dir_example/ -type f -size -5c)
Here, we’re separately listing the files greater and smaller than five bytes in two FIFOs.
Then, within the awk script, we format the output to show the count of the files:
3 records in process substitution 1
2 records in process substitution 2
6. The xargs Command
xargs is a powerful tool that builds and executes command lines using the standard input. Let’s create another example file to see what we can get with this command:
$ cat - << __EOF > dir_example/file4.this.xargs
This file example
will be check by
the following line:
"choose me"
__EOF
Now, let’s try feeding xargs with the find command:
$ find dir_example/ -type f -size +5c \
| xargs -I {} bash -c 'grep -q "choose me" {} && cp {} target'
Here, we use the -I replace-string argument that replaces the occurrences of the replace-string with the name read from the input.
Next, we use the bash command to create a command-string consisting of the grep command filtering the files that contain the string “choose me”.
Finally, cp will copy the file to the destination folder only if the grep exit status is successful.
In this way, we use xargs to pass the output of find as a parameter of the grep and cp commands.
Our target directory now only contains one file:
$ ls -l target/
total 0
-rwxrwxrwx 1 cuau cuau 67 Nov 19 12:40 file4.this.xargs
7. The GNU parallel Tool
GNU parallel is a command-line tool for executing jobs in parallel. A job can be a single command or a script that runs for each line of the input. In other words, it’s like xargs on steroids.
If we’re using a Debian-based distribution, we can use the apt command to install it:
$ sudo apt install -y parallel
Otherwise, if we’re using another distribution (or if we want to compile a specific version), we can download and install it manually:
$ mkdir -p /tmp/parallel_install \
&& curl http://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2 \
| tar xj -C /tmp/parallel_install --strip-components 1 \
&& sudo bash -c '
cd /tmp/parallel_install \
&& ./configure \
&& make \
&& make install'
This one-liner will download GNU Parallel into the /tmp/parallel_install directory.
Let’s test the installation:
$ parallel echo ::: hello
hello
Now we can run some of the past examples with this new tool.
Let’s combine parallel with the process and command substitution to copy the files into the destination folder:
$ parallel cp {1} {2} :::: <(find dir_example/ -type f -size +5c) \
::: $(echo target)
In this case, the output of the process and command substitution provided by parallel is being used as arguments to the cp command.
We use the tokens “::::” and “:::” since parallel uses what is on the right as a file-argument and argument respectively.
Additionally, we use the string {n} (where n is a number) as a positional replacement string, which will replace the nth input source or argument, similar to the {} replace-string from xargs.
Let’s see the content of the folder:
$ ls -l target/
total 0
-rwxrwxrwx 1 cuau cuau 10 Nov 19 21:21 file1.this
-rwxrwxrwx 1 cuau cuau 10 Nov 19 21:21 file2.this
-rwxrwxrwx 1 cuau cuau 10 Nov 19 21:21 file3.this
-rwxrwxrwx 1 cuau cuau 67 Nov 19 21:21 file4.this.xargs
Now, let’s try with the xargs scenario, but this time using parallel in place of xargs:
$ parallel 'grep -q "choose me" {1} && cp {1} {2}' \
:::: <(find dir_example/ -type f -size +5c) \
::: target
By surrounding with quotes, we can run a composed command.
As expected, this copies the file4.this.xargs file:
$ head -v target/*
==> target/file4.this.xargs <==
This file example
will be check by
the following line:
"choose me"
8. Conclusion
In this tutorial, we’ve addressed different ways to use the output of a command as an argument for others.
We’ve reviewed the read builtin, command and process substitution, xargs, and GNU parallel.