1. Introduction
In this tutorial, we’ll discuss how to use the lines of a file as arguments for a Linux command. For example, this can be useful when one program returns a list of filenames that we want to use as inputs for another command. We can also use the techniques here shown to preliminary automate commands with long lists of arguments before moving into scripts.
2. Setup
The working directory used for the example of the article contains different text files:
$ tree ./
./
├── arguments1
├── arguments2
├── file10
├── file15
├── file20
└── file5
0 directories, 6 files
The four files whose name starts with file followed by a number N have N words on them. That is, file5 contains five words. The file named arguments1 contains, line by line, the name of these four files. We discuss the content of file arguments2 later on. We can double-check the content of the first file with arguments using the cat command:
$ cat arguments1
file10
file15
file20
file5
The example that we’ll follow is based on the word counting command wc. This command may return, among other outputs depending on the specified flags, the number of words of a file. Based on the file naming and the folder structure chosen for this example, we can get the number of words of each file using bash globbing:
$ wc -w file*
10 file10
15 file15
20 file20
5 file5
50 total
Before moving on to the different approaches, we should clarify the two ways in which the shell interprets the lines of arguments.
First, we assume that we want all the lines of arguments1 to be used together in a single call to the command of interest. Therefore, the approaches presented at the beginning return the same information as bash globbing (check the latest code snippet).
Later, we call the command once for each line, using each line as the argument.
3. Proposed Solutions for Arguments Without Spaces
In this section, we’ll propose different solutions to be able to use the lines of a file as an argument for a command.
3.1. Using the Command cat: One Command Call With All File Lines
The first approach is to use the cat command, which was already used to check the contents of the file arguments1. The general command is:
commmand `cat file_with_arguments`
This yields the same result as above for the command example:
$ wc -w `cat arguments1`
10 file10
15 file15
20 file20
5 file5
50 total
The result from the cat command can also be expanded with the use of the dollar sign and parentheses, which is a preferred way by the POSIX standards, and it provides other benefits as better nesting and handling of backslashes:
commmand $(cat file_with_arguments)
This second approach of expanding the results of cat applied to the example of the article returns:
$ wc -w $(cat arguments1)
10 file10
15 file15
20 file20
5 file5
50 total
3.2. Using the Redirection Operator: One Command Call With All File Lines
The redirection operator is another solution to the problem, not requiring the use of the cat command. Nevertheless, its syntax is similar to the previous procedure:
commmand $(< file_with_arguments)
The obtained result matches the one before:
$ wc -w $(< arguments1)
10 file10
15 file15
20 file20
5 file5
50 total
The difference between the redirection operator < and the cat command is that with the redirection operator, the shell performs the redirection itself without executing the cat binary. The cat command works because it has redirection properties, but it incurs some overhead, as it requires a subshell.
3.3. Using for Loops: One Command Call per File Line
There are cases where the previous approaches won’t work because multiple calls to the command are desired, one for each line of file with arguments. There is also another case where the approach below shown is superior: multiple arguments want to be varied multiple times, having one file for each argument.
This approach relies on for loops to call the command once with each line of the arguments1 file individually:
for arg in $(< file_with_arguments); do command "$arg"; done
Even if the previous approaches worked with the wc command, let’s solve the same problem with for loops:
$ for arg in $(< arguments1); do wc -w "$arg"; done
10 file10
15 file15
20 file20
5 file5
With this approach, we explicitly call the command multiple times instead of once. This is why the for loop with the wc command doesn’t show the line “**50 total” that appeared in the previous approaches: we call the command with a single file four times (instead of one time with four files).
4. Proposed Solutions for Arguments With Spaces
In this section, we present the limitations of the previous approaches and the solution that can be taken to address them.
4.1. Limitations of the Previous Approaches
With all the previous approaches, the main limitation is how the shell handles the lines of the file with arguments: it will also split the spaces. Thus, let’s imagine that this file is composed of one string with spaces for the second argument:
arg1
"arg 2"
arg3
Even if we enclosed the string with the space between quotes, the shell doesn’t escape the sequence. With the presented approaches so far, either the function receives four arguments, or we call the function four times with one argument each:
arg1
"arg
2"
arg3
This may not be problematic in all cases, deeming the previous approaches valid and straightforward. However, we should keep it in mind to avoid issues with certain arguments.
We can see this by considering another file with the arguments: arguments2. This file contains the same arguments as before but ordered differently:
$ cat arguments2
file10
file15
file20 file5
Attending to the number of lines, the command wc has to be called three times. The first one with file10 as a unique argument, the second one only with file15, and the last time with two arguments: file20 and file5. This way, only in the last call, the total count of words should appear.
However, when using the approach based on for loops, the output shows that the function is called once for each file:
$ for arg in $(< arguments2); do wc -w "$arg"; done
10 file10
15 file15
20 file20
5 file5
4.2. Using the While-Read-Do Construction: One Command Call per File Line
To address this problem, there is yet another procedure to use the content of a file, line by line, as arguments to a command. It is a method that allows handling lines as full arguments, which may be necessary for some cases.
Moreover, if we want to vary multiple arguments simultaneously, this is advantageous. The following approach requires one file only (with the space-separated arguments for each command call in each line). Meanwhile, the for loop procedure requires one file for each argument position (having each file as many lines as command calls but only one argument per line).
The solution uses the while-read-do and the redirection operator used before:
while read arg; do command $arg; done < file_with_arguments
This approach used with the file arguments2 returns the expected output:
$ while read arg; do wc -w $arg; done < arguments2
10 file10
15 file15
20 file20
5 file5
25 total
In this last output, we see the total number of words of the last function call because it received two arguments at the same time: file20 and file5.
5. Conclusion
In this article, we have discussed different ways to use the lines of a file as arguments for a function. Some methods use all the lines together in a single call to the function. Others tackle the problem with as many calls to the function as lines in the file, using a single argument for each call.