1. Overview
Sometimes, when retrieving data from a URL via the command line, the original URL we provide is redirected, and we’d like to know the final URL that serves the content. In this article, we demonstrate getting the final URL after a requested URL is redirected using curl and wget. curl and wget are non-interactive command-line utilities for transferring data to or from a server.
2. URL and Final URL
URL stands for Uniform Resource Locator – it’s the unique address of a resource on the web. A URL may redirect or forward traffic from one page to another for various reasons. For example, perhaps the content we seek has moved to a new location, the domain name changed, or the page was removed from the site.
The final URL is the page where users land after redirection. For example, typing the URL, http://google.com, in the browser redirects to the final URL, https://www.google.com/.
3. Using the curl Command
We use a curl command to get the final URL after redirection:
$ curl -Ls -o /dev/null -w %{url_effective} http://google.com
http://www.google.com/
We pass in http://google.com, but the URL redirects to http://www.google.com/. Let’s break down the command.
The -L option tells the curl command to follow the URL redirections. The -s option is for silent mode, which means the command should not output anything to the terminal, while the -o option provides the path that it should send the output to instead of sending it to stdout. In the case above, we’re sending the output to /dev/null.
The -w option writes out information after the transfer is complete. Moreover, this option has a lot of variables that are set with values and information from the data transfer. The %{url_effective} variable displays the URL that was fetched last. Since we’ve provided the –L option to follow redirections, it’ll give us the final URL.
4. Using the wget Command
We can also use the wget command to get the final URL after redirection:
$ wget -O /dev/null google.com 2>&1 | grep -w 'Location'
Location: http://www.google.com/ [following]
Let’s break down the above wget command. The input to the wget command is google.com, but it redirects to http://www.google.com/.
The -O option sets the location to which the output of the command will be written. We’re writing the output to /dev/null. By default, the wget command directs the headers to the standard error, so 2>&1 redirects the standard error of the command to the standard output. Then, the output is piped to the grep command. The grep command searches for the line with the word “Location”. This line contains the value of the final URL after redirection.
5. Conclusion
In this tutorial, we’ve demonstrated how to use the curl and wget commands to get the final URL after the provided URL is redirected. Use curl –help or wget –help to see the complete list of options for each command. We can also refer to the man pages for more information.