Web Security Testing : Automating Specific Tasks with cURL - Fetching Many Variations on a URL

1/31/2015 2:41:53 AM


There are often times when we want to fetch a variety of URLs that vary in only a small way. For example, you might want to fetch variations on a product page by varying the PRODUCTID parameter. You might need to fetch a variety of URLs that vary in a small part (e.g., news, careers, blog). cURL makes this all easy by not only allowing you to specify variations in a concise format, but also letting you name the corresponding output file according to the variation you specified.


# Fetch all the categories from 00 to 99.curl -o 'category-#1#2.html' '[0-9][0-9]'curl -o 'category-#1.html' '[0-99]'# Fetch several main pages and store them in files named accordinglycurl -o '#1.html' '{news,blog,careers,contact,sitemap}/'


Note the use of single quotes. In a Unix environment, that’s necessary to avoid problems where the shell interprets the #, ?, and brackets. In Windows, that’s less necessary, but it doesn’t hurt. The first example fetches pages where CATID must be exactly 2 digits, i.e., 00, 01, 02, ..., 97, 98, 99. The second example fetches the same sorts of pages, but where CATID is a single digit for values 0–9 and double digits thereafter, i.e., 0, 1, 2, ..., 97, 98, 99.

You can put many varying parameters into a single command. cURL will do all the permutations and combinations. Consider an item page that takes a product ID (0001–9999), a category (0–9), a color (red, yellow, blue, or green) and a size (S, M, L, or XL). The following single invocation of cURL will fetch every possible combination (all 1,599,840 of them!).

curl -o '#1-#2-#3-#4.html' \"[0001-9999]    &cat=[0-9]&color={red,yellow,blue,green}&size={s,m,l,xl}"

Of course, in security testing, we would test for weird values: alphabetic product IDs, numeric sizes, and interesting boundary cases like 65,537 for color.

  •  Web Security Testing : Automating Specific Tasks with cURL - Fetching a Page with cURL
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 6) - Check In/Out, Versioning, and Content Approval
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 5) - Content Management - Putting It All Together
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 4) - Content Management - Master Pages,Page Layouts
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 3) - Content Management - Site Columns, Content Types
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 2) - Publishing Features
  •  Sharepoint 2013 : SharePoint Publishing Infrastructure (part 1) - A Publishing Site
  •  Show the whole car kingdom “What’ll you do?”, Porsche (Part 1)
  •  Sharepoint 2013 : List and library essentials - Organizing items by using folders
  •  Sharepoint 2013 : List and library essentials - Sorting or filtering a list view
    Most View
    Windows 7 : Understanding Libraries (part 2) - Special Shell Folders...Now Just User Folders
    Windows Server 2003 : Designing a Server Cluster (part 1) - Planning a Server Cluster Hardware Configuration
    Ubuntu Phones To Hit Stores In October (Part 2)
    What To Look For When Buying A New Phone Or Tablet (Part 1)
    Network Audio Player : Lindermann Musicbook 25 (Part 2)
    Security Pros Get Caught Out By QR Codes
    Windows 8 Architecture from a Developer’s Point of View : Windows 8 Development Architecture
    Creative Edge iPhone Movie Makers - Small Screen, Big Picture (Part 2)
    Sony VAIO S Series 13 - The Best-Looking Laptop
    Microsoft Exchange Server 2007 : Components of a Secure Messaging Environment (part 1) - Hardening Windows Server 2003 - Auditing Policies
    Popular Tags
    Microsoft Access Microsoft Excel Microsoft OneNote Microsoft PowerPoint Microsoft Project Microsoft Visio Microsoft Word Active Directory Biztalk Exchange Server Microsoft LynC Server Microsoft Dynamic Sharepoint Sql Server Windows Server 2008 Windows Server 2012 Windows 7 Windows 8 Adobe Indesign Adobe Flash Professional Dreamweaver Adobe Illustrator Adobe After Effects Adobe Photoshop Adobe Fireworks Adobe Flash Catalyst Corel Painter X CorelDRAW X5 CorelDraw 10 QuarkXPress 8 windows Phone 7 windows Phone 8 BlackBerry Android Ipad Iphone iOS
    Top 10
    Review : Acer Aspire R13
    Review : Microsoft Lumia 535
    Review : Olympus OM-D E-M5 Mark II
    TomTom Runner + MultiSport Cardio
    Timex Ironman Run Trainer 2.0
    Suunto Ambit3 Peak Sapphire HR
    Polar M400
    Garmin Forerunner 920XT
    Sharepoint 2013 : Content Model and Managed Metadata - Publishing, Un-publishing, and Republishing
    Sharepoint 2013 : Content Model and Managed Metadata - Content Type Hubs