Web Security Testing : Automating Specific Tasks with cURL - Fetching a Page with cURL

1/31/2015 2:41:27 AM


For starters we just need to fetch a web page. We want the whole page, unadulterated, exactly as it would be delivered to a browser. This is fundamentally what cURL was designed to do, so it should be no surprise that it’s simple. We will use a couple different options to send the output to files.


# basic invocationcurl -o example.html fetch a secure web pagecurl -k -o example-secure.html fetch a file by FTP. This time, have curl automatically# pick the output filenamecurl -O


In these basic fetch invocations, cURL simply writes the output to the designated file. The -o option specifies the file by name name. The -O (capital O) tells cURL to try to figure out the filename and save it. Note that you can’t do this if there is no obvious filename. For example, you cannot use -O with, but you can use it with (in which case your output would be saved to a file named to default.asp). In the final FTP example, cURL saves the downloaded file to

Notice the -k option when we fetch a page via SSL/TLS. The -k option tells cURL to ignore the fact that it cannot verify the SSL certificate. Most of the time when we’re testing websites, we’re not testing the production site (which probably has a legitimate SSL certificate). Our QA systems often have self-signed or otherwise illegitimate certificates. If you try to fetch a page from such a system, cURL will stop dead when it fails to verify the certificate. You won’t get a page and cURL will complain with this error:

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedMore details here:

The simple answer is to use the -k option to disable checking. There is a more complicated method of adding the certificate to cURL’s set of trusted certificates. Since we’re testing in a QA capacity, there’s usually little value in going to that extra trouble.

  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 6) - Check In/Out, Versioning, and Content Approval
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 5) - Content Management - Putting It All Together
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 4) - Content Management - Master Pages,Page Layouts
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 3) - Content Management - Site Columns, Content Types
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 2) - Publishing Features
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 1) - A Publishing Site
  •  Show the whole car kingdom “What’ll you do?”, Porsche (Part 1)
  •  Sharepoint 2013 : List and library essentials - Organizing items by using folders
  •  Sharepoint 2013 : List and library essentials - Sorting or filtering a list view
  •  Sharepoint 2013 : List and library essentials - Creating and selecting a list view
    Most View
    Asus Transformer Pad Infinity
    The 10 Things To Know Before Buying A Laptop (Part 1)
    Can Microsoft Really Kill Off Windows XP? (Part 2)
    Top 10 Smartphones - Q1 2013
    Logitech Z553 Speaker System - The Final Frontier
    BlackBerry Push APIs (part 2) - Building an Application that Uses the BlackBerry Push APIs - Unsubscribing From the Push System, Pushing Data to a Subscriber
    4G Has Landed (Part 1)
    Nexus 4 – A Smartphone With Elegant Design (Part 7)
    What Is CEBP?
    Enterprise Storage Options: Local, Shared, Cloud-Based & Beyond (Part 1)
    Popular Tags
    Microsoft Access Microsoft Excel Microsoft OneNote Microsoft PowerPoint Microsoft Project Microsoft Visio Microsoft Word Active Directory Biztalk Exchange Server Microsoft LynC Server Microsoft Dynamic Sharepoint Sql Server Windows Server 2008 Windows Server 2012 Windows 7 Windows 8 Adobe Indesign Adobe Flash Professional Dreamweaver Adobe Illustrator Adobe After Effects Adobe Photoshop Adobe Fireworks Adobe Flash Catalyst Corel Painter X CorelDRAW X5 CorelDraw 10 QuarkXPress 8 windows Phone 7 windows Phone 8 BlackBerry Android Ipad Iphone iOS
    Top 10
    Review : Acer Aspire R13
    Review : Microsoft Lumia 535
    Review : Olympus OM-D E-M5 Mark II
    TomTom Runner + MultiSport Cardio
    Timex Ironman Run Trainer 2.0
    Suunto Ambit3 Peak Sapphire HR
    Polar M400
    Garmin Forerunner 920XT
    Sharepoint 2013 : Content Model and Managed Metadata - Publishing, Un-publishing, and Republishing
    Sharepoint 2013 : Content Model and Managed Metadata - Content Type Hubs