1. Overview
cURL is a widely used Linux tool for sending HTTP requests and viewing the responses. For some cases, we may need to send requests that avoid cache and generate a fresh response from the server each time. Before we dig deeper into this, we need to understand that caching can happen either on the client-side (browser cache) or on the server-side (application-level cache or caching at a proxy layer such as Apache or NGINX).
When we use the cURL command, we must note that cURL is only an HTTP client, and it doesn’t cache any request on the client-side. Therefore, any caching while using this command happens on the server-side. To bypass the server-side cache, we can try some tweaks on the HTTP request we’re sending. However, these may or may not work based on how the server is configured to do caching. We’ll discuss these tweaks below.
2. Adding the Cache-Control HTTP Header
We may use the Cache-Control HTTP header in both HTTP requests and responses to control the caching behavior. We need to provide directives under this header, and some directives that may be of interest to us are:
- no-cache: This should cause the cache to be revalidated with the original resource before the response is sent back.
- no-store: The response for this request shouldn’t be cached at all.
We can use the curl command with the Cache-Control header and include our desired directives:
$ curl -H 'Cache-Control: no-cache, no-store' http://www.example.com
The server may or may not be configured to respect the Cache-Control header. Therefore, whether this method will work is dependent on the server or website we’re sending the HTTP request to.
3. Adding the Pragma HTTP Header
The Cache-Control HTTP header we discussed above was introduced only in HTTP 1.1. For backward compatibility with systems that still use HTTP 1.0, we can use the Pragma header:
$ curl -H 'Pragma: no-cache' http://www.example.com
As with the previous method, the server may or may not consider this directive to serve a fresh request, but it’s worth a try.
4. The Old School Way: Changing the URL
Most server-side caches (especially the ones that are set up at the proxy layer) work by caching a generated response against the URL that was used to access it. Therefore, we can bypass the cache by changing the URL in a way that won’t affect our response.
The most common way of doing this is by adding a query parameter or query string that we’re sure won’t alter the intended response in any way. To start with, we may add a string as a query parameter to the URL:
$ curl http://www.example.com/?xyzzyspoon
While this works the first time we try with the string appended to the URL, this new URL will also get cached and stop giving us a fresh response on subsequent requests. We can write a command that generates a different URL every time to overcome this.
One way of doing this is using the $RANDOM shell variable, which returns a random integer every time it’s accessed:
$ curl "http://www.example.com/?$RANDOM"
Another reasonably foolproof way of changing the URL every time the request is sent is to use a timestamp as a query parameter. To accomplish this, we can use the date command:
$ curl "http://www.example.com/?$(date +%s)"
5. Conclusion
First, we must note that the cURL command doesn’t do any caching on the client-side, and any caching that’s happening while using this command is happening on the server-side. To bypass the cache on the server-side, we may use HTTP headers such as Cache-Control and Pragma with applicable directives or tweak the query parameters of the URL to change the URL being accessed.
Depending on the server-side configuration, these methods may or may not work. Tweaking query parameters also carries the additional risk of modifying the intended response.
Hence, it’s always better we approach this problem from the server-end if we have control over the server-side code and deployment. Otherwise, we can consult the documentation provided or get in touch with the support team of the service we’re accessing. This is the only way to be entirely sure of how caching works for the system we’re dealing with.