1. Overview
cURL is a versatile command-line utility for transferring data. One of its many features is downloading multiple files with a single command using brace expansion. This can be particularly useful when downloading files that follow a predictable naming pattern.
In this tutorial, we’ll learn how to download multiple files with the curl command using brace syntax.
2. Understanding the Scenario
Let’s start by using brace syntax to make a curl request for getting user details for two users with different IDs:
$ curl -s -XGET https://reqres.in/api/users/{1,2} | jq .
{
"data": {
"id": 1,
"email": "[email protected]",
"first_name": "George",
"last_name": "Bluth",
"avatar": "https://reqres.in/img/faces/1-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
{
"data": {
"id": 2,
"email": "[email protected]",
"first_name": "Janet",
"last_name": "Weaver",
"avatar": "https://reqres.in/img/faces/2-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
We must note that we used jq to get the JSON output in a readable format. Further, the output for both requests is on stdout, and the user details aren’t downloaded locally.
Now, let’s download the response output in a file named as the remote file using the -O (–remote-name) option:
$ curl -s -O -XGET https://reqres.in/api/users/{1,2} | jq .
{
"data": {
"id": 2,
"email": "[email protected]",
"first_name": "Janet",
"last_name": "Weaver",
"avatar": "https://reqres.in/img/faces/2-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
Unfortunately, we still have the output for the second API call on stdout. However, we can verify that the response from the first API call is saved locally:
$ ls
1
$ cat 1 | jq .
{
"data": {
"id": 1,
"email": "[email protected]",
"first_name": "George",
"last_name": "Bluth",
"avatar": "https://reqres.in/img/faces/1-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
In the following sections, we’ll explore multiple ways to download multiple files with the curl command using brace expansion
3. Using –remote-name (-O) Option
In this section, we’ll solve the issue where we can’t download multiple files locally using a single instance of the -O option with curl.
3.1. With Multiple -O Options
We need to specify the -O option once for each file when downloading multiple files using the same curl request. So, let’s add it twice when making API calls to get user details for two users:
$ curl -s -OO -XGET https://reqres.in/api/users/{1,2}
With this approach, we didn’t get any output on the stdout. Further, we can verify that there are two files downloaded locally:
$ ls
1 2
The file names are extracted from the API endpoint for each user.
Next, let’s validate that the first file has the right content:
$ cat 1 | jq .
{
"data": {
"id": 1,
"email": "[email protected]",
"first_name": "George",
"last_name": "Bluth",
"avatar": "https://reqres.in/img/faces/1-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
Similarly, let’s verify the contents of the second file as well:
$ cat 2 | jq .
{
"data": {
"id": 2,
"email": "[email protected]",
"first_name": "Janet",
"last_name": "Weaver",
"avatar": "https://reqres.in/img/faces/2-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
Fantastic! It looks like we nailed this one.
3.2. With -O Option and printf
As the number of files increases, it’ll become increasingly inconvenient to specify the -O option separately for each file. However, we can use the printf command to automatically add the -O option for a given number of times:
$ printf 'O%.0s' {1..2}
OO
We used the %.0s format string with zero length to ensure that 1 and 2 don’t print. Moreover, the “O” string is printed for each argument in the range {1…2}.
Next, let’s make the API calls to download the user details for two users:
$ curl -s -$(printf 'O%.0s' {1..2}) -XGET https://reqres.in/api/users/{1,2}
Lastly, we can verify the correctness of our approach:
$ cat {1,2} | jq .
{
"data": {
"id": 1,
"email": "[email protected]",
"first_name": "George",
"last_name": "Bluth",
"avatar": "https://reqres.in/img/faces/1-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
{
"data": {
"id": 2,
"email": "[email protected]",
"first_name": "Janet",
"last_name": "Weaver",
"avatar": "https://reqres.in/img/faces/2-image.jpg"
},
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
Great! The files are downloaded correctly.
3.3. With -O Option in a Loop
We can solve our use case with a single -O option with a single curl command for each URL. For this purpose, we can write a for loop to iterate over the multiple user IDs using the brace expansion:
for user_id in {1,2};
do
curl -s -O -XGET https://reqres.in/api/users/$user_id
done
Further, let’s confirm that our solution is working correctly:
$ cat {1,2} | jq .
# same as earlier
It looks like we’ve got this one right.
4. Using –remote-name-all Option
We can use the –remote-name-all option over the multiple -O (–remote-name) options for downloading multiple files:
$ curl -s --remote-name-all -XGET https://reqres.in/api/users/{1,2}
However, we must note that the –remote-name-all option is only available with versions 7.19 and later. Further, there is no short notation for this option.
Now, let’s confirm that we’re able to download the files with this option:
$ cat {1,2} | jq .
# same as earlier
It works as expected!
5. wget as an Alternative
As an alternative tool, we can use the wget utility to download multiple files using the brace syntax:
$ wget --quiet https://reqres.in/api/users/{1,2}
We used the –quiet option to remove the debug output from the output. Other than that, we don’t require any special flags to download the files locally because it’s the default behavior for wget.
Like always, let’s verify the contents of the file:
$ cat {1,2} | jq .
# same as earlier
Perfect! We downloaded the files successfully by choosing wget over curl.
6. Conclusion
In this article, we learned how to download multiple files using brace syntax with the curl command. Furthermore, we explored the –remote-name and –remote-name-all options with curl.
Lastly, we learned the alternative solution of using the wget command to solve the exact use case without any special flags.