SQL Injection : Code-Level Defenses - Canonicalization

10/1/2012 2:04:25 AM


A difficulty with input validation and output encoding is ensuring that the data being evaluated or transformed is in the format that will be interpreted as intended by the end user of that input. A common technique for evading input validation and output encoding controls is to encode the input before it is sent to the application in such a way that it is then decoded and interpreted to suit the attacker's aims. For example, Table 1 lists alternative ways to encode the single-quote character.

Table 1. Example Single-Quote Representations
RepresentationType of encoding
%27URL encoding
%2527Double URL encoding
%%317Nested double URL encoding
%u0027Unicode representation
%u02b9Unicode representation
%ca%b9Unicode representation
'HTML entity
'Decimal HTML entity
'Hexadecimal HTML entity
%26apos;Mixed URL/HTML encoding

In some cases, these are alternative encodings of the character (%27 is the URL-encoded representation of the single quote), and in other cases these are double-encoded on the assumption that the data will be explicitly decoded by the application (%2527 when URL-decoded will be %27 as shown in Table 8.6, as will %%317) or are various Unicode representations, either valid or invalid. Not all of these representations will be interpreted as a single quote normally; in most cases, they will rely on certain conditions being in place (such as decoding at the application, application server, WAF, or Web server level), and therefore it will be very difficult to predict whether your application will interpret them this way.

For these reasons, it is important to consider canonicalization as part of your input validation approach. Canonicalization is the process of reducing input to a standard or simple form. For the single-quote examples in Table 1, this would normally be a single-quote character (‘).

Canonicalization Approaches

So, what alternatives for handling unusual input should you consider? One method, which is often the easiest to implement, is to reject all input that is not already in a canonical format. For example, you can reject all HTML-and URL-encoded input from being accepted by the application. This is one of the most reliable methods in situations where you are not expecting encoded input. This is also the approach that is often adopted by default when you do whitelist input validation, as you may not accept unusual forms of characters when validating for known good input. At the very least, this could involve not accepting the characters used to encode data (such as %, &, and # from the examples in Table 8.6), and therefore not allowing these characters to be input.

If rejecting input that can contain encoded forms is not possible, you need to look at ways to decode or otherwise make safe the input that you receive. This may include several decoding steps, such as URL decoding and HTML decoding, potentially repeated several times. This approach can be error-prone, however, as you will need to perform a check after each decoding step to determine whether the input still contains encoded data. A more realistic approach may be to decode the input once, and then reject the data if it still contains encoded characters. This approach assumes that genuine input will not contain double-encoded values, which should be a valid assumption in most cases.

Working with Unicode

When working with Unicode input such as UTF-8, one approach is normalization of the input. This converts the Unicode input into its simplest form, following a defined set of rules. Unicode normalization differs from canonicalization in that there may be multiple normal forms of a Unicode character according to which set of rules is followed. The recommended form of normalization for input validation purposes is NFKC (Normalization Form KC – Compatibility Decomposition followed by Canonical Composition). You can find more information on normalization forms at

The normalization process will decompose the Unicode character into its representative components, and then reassemble the character in its simplest form. In most cases, it will transform double-width and other Unicode encodings into their ASCII equivalents, where they exist.

You can normalize input in Java with the Normalizer class (since Java 6) as follows:

normalized = Normalizer.normalize(input, Normalizer.Form.NFKC);

You can normalize input in C# with the Normalize method of the String class as follows:

normalized = input.Normalize(NormalizationForm.FormKC);

You can normalize input in PHP with the PEAR::I18N_UnicodeNormalizer package from the PEAR repository, as follows:

$normalized = I18N_UnicodeNormalizer::toNFKC($input, 'UTF-8');

Another approach is to first check that the Unicode is valid (and is not an invalid representation), and then to convert the data into a predictable format—for example, a Western European character set such as ISO-8859-1. The input would then be used in that format within the application from that point on. This is a deliberately lossy approach, as Unicode characters that cannot be represented in the character set converted to will normally be lost. However, for the purposes of making input validation decisions, it can be useful in situations where the application is not localized into languages outside Western Europe.

You can check for Unicode validity for UTF-8 encoded Unicode by applying the set of regular expressions shown in Table 2. If the input matches any of these conditions it should be a valid UTF-8 encoding. If it doesn't match, the input is not a valid UTF-8 encoding and should be rejected. For other types of Unicode, you should consult the documentation for the framework you are using to determine whether functionality is available for testing the validity of input.

Table 2. UTF-8 Parsing Regular Expressions
Regular expressionDescription
[\xC2-\xDF][\x80-\xBF]Two-byte representation
\xE0[\xA0-\xBF][\x80-\xBF]Two-byte representation
[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}Three-byte representation
\xED[\x80-\x9F][\x80-\xBF]Three-byte representation
\xF0[\x90-\xBF][\x80-\xBF]{2}Planes 1 through 3
[\xF1-\xF3][\x80-\xBF]{3}Planes 4 through 15
\xF4[\x80-\x8F][\x80-\xBF]{2}Plane 16

Now that you have checked that the input is validly formed, you can convert it to a predictable format—for example, converting a Unicode UTF-8 string to another character set such as ISO-8859-1 (Latin 1).

In Java, you can use the CharsetEncoder class, or the simpler string method getBytes( ) (Java 6 and later) as follows:

string ascii = utf8.getBytes(“ISO-8859-1”);

In C#, you can use the Encoding.Convert class as follows:

ASCIIEncoding ascii = new ASCIIEncoding();
UTF8Encoding utf8 = new UTF8Encoding();
byte[] asciiBytes = Encoding.Convert(utf8, ascii, utf8Bytes);

In PHP, you can do this with utf8_decode as follows:

$ascii = utf8_decode($utf8string);
  •  SQL Injection : Code-Level Defenses - Encoding Output
  •  Upgrading and Converting to Access 2010 : TO CONVERT OR TO ENABLE
  •  Oracle Database 11g : Database Fundamentals - Learn the Basic Oracle Database 11g Data Types, Work with Tables, Work with Stored Programmed Objects
  •  Oracle Database 11g : Database Fundamentals - Define a Database, Learn the Oracle Database 11g Architecture
  •  SQL Server 2005 : Advanced OLAP - Calculations (part 2) - Named Sets, More on Script View
  •  SQL Server 2005 : Advanced OLAP - Calculations (part 1) - Calculated Members
  •   Exploiting SQL Injection : Automating SQL Injection Exploitation
  •   Exploiting SQL Injection : Out-of-Band Communication
  •  SQL Server 2008 R2 : Dropping Indexes, Online Indexing Operations, Indexes on Views
    PS4 game trailer XBox One game trailer
    WiiU game trailer 3ds game trailer
    Top 10 Video Game
    -   Minecraft Mods - MAD PACK #10 'NETHER DOOM!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #9 'KING SLIME!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #2 'LAVA LOBBERS!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #3 'OBSIDIAN LONGSWORD!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Total War: Warhammer [PC] Demigryph Trailer
    -   Minecraft | MINIONS MOVIE MOD! (Despicable Me, Minions Movie)
    -   Minecraft | Crazy Craft 3.0 - Ep 3! "TITANS ATTACK"
    -   Minecraft | Crazy Craft 3.0 - Ep 2! "THIEVING FROM THE CRAZIES"
    -   Minecraft | MORPH HIDE AND SEEK - Minions Despicable Me Mod
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 92 "IS JOE DEAD?!"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 93 "JEDI STRIKE BACK"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 94 "TATOOINE PLANET DESTRUCTION"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 95 "TATOOINE CAPTIVES"
    -   Hitman [PS4/XOne/PC] Alpha Gameplay Trailer
    -   Satellite Reign [PC] Release Date Trailer
    Game of War | Kate Upton Commercial