Web Security : Automating with LibWWWPerl - Writing a Basic Perl Script to Fetch a Page, Programmatically Changing Parameters

4/27/2013 12:55:45 AM

1. Writing a Basic Perl Script to Fetch a Page

1.1. Problem

For basic testing, or as a basis for something larger, you want a Perl script that fetches a page from an application and stores the response in a Perl data structure you can use. This is a basic GET request.

1.2. Solution

This is what LibWWWPerl (LWP) is all about. You need to install the following Perl modules :

  • LWP

  • HTTP::Request

Example 1 shows a basic script that issues a request for a page and checks the return value. If the return code is successful, it prints the response contents to standard output. If the return code indicates failure, just the return code and error message are printed.

Example 1. Basic Perl script to fetch a page
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);

$UA   = LWP::UserAgent->new();
$req  = HTTP::Request->new( GET => "" );
$resp = $UA->request($req);

# check for error. Print page if it's OK
if ( ( $resp->code() >= 200 ) && ( $resp->code() < 400 ) ) {
    print $resp->decoded_content;
} else {
    print "Error: " . $resp->status_line . "\n";

1.3. Discussion

This script is a fundamental building block for all kinds of basic web requests. 

There are many kinds of requests you might make. Example 8-1 shows a GET request. POST is the other common request type. Additional request types are defined in HTTP and are supported by LWP. They include PUT, DELETE, OPTIONS, and PROPFIND, among others. One interesting set of security tests would be to determine your application’s response to some of these less frequently used methods. You may be surprised to find that, instead of a simple “405 Method Not Allowed” response, you receive a response that a hacker can use, like an error 500 with debugging information.

Other Useful LWP Scripts

It’s worth noting here that, in true Perl style, “there’s more than one way to do it,” and in fact Example 8-1 is a bit redundant. There are pre-made scripts that come with the LWP library that do basic jobs like this. When you’re building a test case, you might be more interested in using one of these pre-built scripts unless you need some special behavior. So that you’re aware of them, here’s a brief list. Each has its own man page or online documentation for more detailed information.


Use lwp-download to simply fetch something using a GET request and store it to a file. Similar to curl , it takes the URL from the command line. Unlike curl (or lwp-request), it has no ability to do anything sophisticated like cookies, authentication, or following redirects.


If you want to download a local copy of a file, but only if you don’t have the latest version, lwp-mirror can do that. That’s really its purpose: to be like lwp-download, but to check the server for the modification date of the file and only download it if the file has been modified.


Perl’s answer to curl is lwp-request. It gives you many of the same options and controls that curl does: authentication, cookies, content-type, arbitrary headers, etc. It is not quite as powerful as curl, but it is a good midway between writing your own Perl program and using a very complicated curl invocation.


If you need a primitive spider, the lwp-rget tool can help. It will fetch a page, parse the page, find all the links, and then fetch any of the links that you want it to.

2. Programmatically Changing Parameters

2.1. Problem

You want to programmatically change the inputs on a GET request. This might be to get a range of possible values, or because you need to calculate some part of the value (like today’s date).

2.2. Solution

We assume we have some kind of website that has a search page. Frequently, search pages have a parameter to limit the maximum number of matches returned. In our example, we assume that max can be in the URL. The script in Example 2 changes the max parameter in the URL to a variety of interesting values.

Example 2. Basic Perl script to change parameters
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use URI;

use constant MAINPAGE => '';

$UA = LWP::UserAgent->new();
$req = HTTP::Request->new( GET => MAINPAGE );

# This array says test 8-bits, 16-bits, and 32-bits
my @testSizes = ( 8, 16, 32 );

foreach $numBits (@testSizes) {
    my @boundaryValues = (
        ( 2**( $numBits - 1 ) - 1 ),
        ( 2**( $numBits - 1 ) ),
        ( 2**( $numBits - 1 ) + 1 ),
        ( 2**$numBits - 1 ),
        ( 2**$numBits ),
        ( 2**$numBits + 1 ),
    foreach $testValue (@boundaryValues) {
        my $url = URI->new(MAINPAGE);
            'term' => 'Mac',
            'max'  => $testValue

		# do the fetching of pages inside a loop, where we change the
		# parameter we're tinkering with each time.
        $resp = $UA->request($req);

        # Report any errors
        if ( ( $resp->code() < 200 ) || ( $resp->code() >= 400 ) ) {
            print resp->status_line . $req=>as_string();


2.3. Discussion

Example 2 performs boundary case testing around byte values. That is, it considers the powers of 2 that might be significant boundary cases. We know that 28 is 256, so if the application had only 1 byte for storing the max parameter, boundary values like 255, 256, and 257 should sniff that out. Notice how easily this could be extended to 64 bits by simply putting a 64 into the line with 8, 16, and 32.

Top 10
Review : Sigma 24mm f/1.4 DG HSM Art
Review : Canon EF11-24mm f/4L USM
Review : Creative Sound Blaster Roar 2
Review : Philips Fidelio M2L
Review : Alienware 17 - Dell's Alienware laptops
Review Smartwatch : Wellograph
Review : Xiaomi Redmi 2
Extending LINQ to Objects : Writing a Single Element Operator (part 2) - Building the RandomElement Operator
Extending LINQ to Objects : Writing a Single Element Operator (part 1) - Building Our Own Last Operator
3 Tips for Maintaining Your Cell Phone Battery (part 2) - Discharge Smart, Use Smart
- First look: Apple Watch

- 3 Tips for Maintaining Your Cell Phone Battery (part 1)

- 3 Tips for Maintaining Your Cell Phone Battery (part 2)
- How to create your first Swimlane Diagram or Cross-Functional Flowchart Diagram by using Microsoft Visio 2010 (Part 1)

- How to create your first Swimlane Diagram or Cross-Functional Flowchart Diagram by using Microsoft Visio 2010 (Part 2)

- How to create your first Swimlane Diagram or Cross-Functional Flowchart Diagram by using Microsoft Visio 2010 (Part 3)
Popular Tags
Microsoft Access Microsoft Excel Microsoft OneNote Microsoft PowerPoint Microsoft Project Microsoft Visio Microsoft Word Active Directory Biztalk Exchange Server Microsoft LynC Server Microsoft Dynamic Sharepoint Sql Server Windows Server 2008 Windows Server 2012 Windows 7 Windows 8 Adobe Indesign Adobe Flash Professional Dreamweaver Adobe Illustrator Adobe After Effects Adobe Photoshop Adobe Fireworks Adobe Flash Catalyst Corel Painter X CorelDRAW X5 CorelDraw 10 QuarkXPress 8 windows Phone 7 windows Phone 8