Web Security Testing : Automating Specific Tasks with cURL - Fetching a Page with cURL

Problem

For starters we just need to fetch a web page. We want the whole page, unadulterated, exactly as it would be delivered to a browser. This is fundamentally what cURL was designed to do, so it should be no surprise that it’s simple. We will use a couple different options to send the output to files.

Solution

# basic invocationcurl -o example.html http://www.example.com/# fetch a secure web pagecurl -k -o example-secure.html https://www.example.com/# fetch a file by FTP. This time, have curl automatically# pick the output filenamecurl -O ftp://ftp.example.com/pub/download/file.zip

Discussion

In these basic fetch invocations, cURL simply writes the output to the designated file. The -o option specifies the file by name name. The -O (capital O) tells cURL to try to figure out the filename and save it. Note that you can’t do this if there is no obvious filename. For example, you cannot use -O with http://www.example.com/, but you can use it with http://www.example.com/default.asp (in which case your output would be saved to a file named to default.asp). In the final FTP example, cURL saves the downloaded file to file.zip.

Notice the -k option when we fetch a page via SSL/TLS. The -k option tells cURL to ignore the fact that it cannot verify the SSL certificate. Most of the time when we’re testing websites, we’re not testing the production site (which probably has a legitimate SSL certificate). Our QA systems often have self-signed or otherwise illegitimate certificates. If you try to fetch a page from such a system, cURL will stop dead when it fails to verify the certificate. You won’t get a page and cURL will complain with this error:

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedMore details here: http://curl.haxx.se/docs/sslcerts.html

The simple answer is to use the -k option to disable checking. There is a more complicated method of adding the certificate to cURL’s set of trusted certificates. Since we’re testing in a QA capacity, there’s usually little value in going to that extra trouble.