1. Overview
cURL is a popular software that provides a command line tool for transferring data using various network protocols.
In this tutorial, we’ll learn how to use the curl command to download a set of URLs listed in a file.
2. Sample File
Let’s start by creating the file-with-urls.txt file to contain a list of file URLs:
$ echo "file://$HOME/file1" >> $HOME/file-with-urls.txt; \
> echo "file://$HOME/file2" >> $HOME/file-with-urls.txt; \
> echo "file://$HOME/file3" >> $HOME/file-with-urls.txt;
We can note that we’ve specified the $HOME variable to use the user’s home directory as a parent directory for all the files.
Next, we know that the files file1, file2, and file3 don’t exist yet. So, let’s go ahead and create these files with some sample textual data:
$ echo "content for file1" >> $HOME/file1; \
> echo "content for file2" >> $HOME/file2; \
> echo "content for file3" >> $HOME/file3;
Finally, let’s verify the content within these sample files while using the -n option of the tail command so that we can see both the filenames and the content:
$ tail -n +1 file-with-urls.txt file1 file2 file3
==> file-with-urls.txt <==
file:///Users/tavasthi/file1
file:///Users/tavasthi/file2
file:///Users/tavasthi/file3
==> file1 <==
content for file1
content for file1
==> file2 <==
content for file2
==> file3 <==
content for file3
It looks like we’ve got this right!
3. curl and xargs
First, let’s see how we can download a single file using the -O option of the curl command:
$ curl -O file://$HOME/file1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18 100 18 0 0 32667 0 --:--:-- --:--:-- --:--:-- 32667
$ cat file1
content for file1
Next, our problem boils down to how we can read the file-with-urls.txt file one line at a time and send it to the curl command for downloading one file at a time. The -n option available with the xargs command solves this use case.
So, let’s go ahead and use the xargs command for passing a single file URL as an argument to the curl command for download:
$ xargs -n 1 curl -O < $HOME/file-with-urls.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18 100 18 0 0 23622 0 --:--:-- --:--:-- --:--:-- 23622
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18 100 18 0 0 32667 0 --:--:-- --:--:-- --:--:-- 32667
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18 100 18 0 0 40449 0 --:--:-- --:--:-- --:--:-- 40449
Finally, let’s verify that the files are downloaded correctly:
$ tail -n +1 ./file1 ./file2 ./file3
==> ./file1 <==
content for file1
==> ./file2 <==
content for file2
==> ./file3 <==
content for file3
We can see that the downloaded files are exactly what we were looking for!
4. curl With while Loop
We can also use the while loop for solving the problem of reading file-with-urls.txt line by line and then pass a single file URL as an argument to the curl command.
Let’s go ahead and write a one-liner solution for downloading all the files listed in the file-with-urls.txt file:
$ while read -r line; do curl -O $line; done < $HOME/file-with-urls.txt
We must note that the call to the read command gives us one file URL in the line variable at a time. Then, the curl command within the loop executes iteratively to download each file.
Lastly, let’s verify the downloaded files:
$ tail -n +1 ./file1 ./file2 ./file3
==> ./file1 <==
content for file1
==> ./file2 <==
content for file2
==> ./file3 <==
content for file3
The result looks perfect!
5. wget as an Alternative
In this section, let’s explore the alternative approach of using the wget command with the -i option to solve the same use case.
First, let’s see if we can get it working for downloading the files listed in the file-with-urls.txt file:
$ wget -i $HOME/file-with-urls.txt
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file1: Unsupported scheme ‘file’
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file2: Unsupported scheme ‘file’
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file3: Unsupported scheme ‘file’
No URLs found in /Users/tavasthi/file-with-urls.txt.
It looks like the attempt failed, and the error message indicates that the wget command doesn’t support the file scheme. This is a significant limitation of wget compared to the curl command that supports this scheme.
Moving on, let’s keep in mind that the wget command supports limited schemes such as http, https, and ftp. Henceforth, let’s go ahead and use a different sample file that contains a list of file URLs with the https scheme:
$ cat $HOME/file-with-http-urls.txt
https://filesamples.com/samples/document/txt/sample1.txt
https://filesamples.com/samples/document/txt/sample2.txt
Finally, let’s download these files using wget with the -i option:
$ wget -i $HOME/file-with-http-urls.txt
--2022-12-22 08:23:44-- https://filesamples.com/samples/document/txt/sample1.txt
Resolving filesamples.com (filesamples.com)... 172.67.178.244, 104.21.17.252
Connecting to filesamples.com (filesamples.com)|172.67.178.244|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 607 [text/plain]
Saving to: ‘sample1.txt’
sample1.txt 100%[========================================================================================>] 607 --.-KB/s in 0s
2022-12-22 08:23:45 (2.64 MB/s) - ‘sample1.txt’ saved [607/607]
--2022-12-22 08:23:45-- https://filesamples.com/samples/document/txt/sample2.txt
Reusing existing connection to filesamples.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘sample2.txt’
sample2.txt [ <=> ] 2.79K --.-KB/s in 0s
2022-12-22 08:23:45 (9.81 MB/s) - ‘sample2.txt’ saved [2859]
FINISHED --2022-12-22 08:23:45--
Total wall clock time: 1.2s
Downloaded: 2 files, 3.4K in 0s (6.65 MB/s)
We can see that this time, the download worked as expected. Additionally, we must realize that for the supported schemes, using the wget was more straightforward than using the curl command.
6. Conclusion
In this article, we learned a few ways to use the curl command for downloading a set of files listed in a file. Additionally, we learned the alternative of the wget command to solve the exact use case with ease.