Server-Side Browser Detection and Content Delivery : Mobile Detection (part 3) - Transcoders

1/29/2011 6:47:11 PM

3. Transcoders

With WAP 1.1, operator gateways were required to precompile WML and make it lighter and more easily parsable by devices with limited memory/CPU resources. Gateways were also relied upon to manage cookies on behalf of the devices (which did not have enough memory) and for integration with the operator’s backend (for example, the gateway was able to inject the users’ phone numbers into HTTP headers, so that authorized content providers could recognize them and bill them for services).

With WAP 2 precompilation was no longer needed, but other aspects of gateways were and still are today, to some extent. In a lot of different contexts, the presence of a WAP gateway is beneficial to developers and content providers.

Around 2002, some companies started selling tools to “mobilize” web content. That is, users could type a URL into a field, and the transcoder/content reformatter would chop it up into pages that could be viewed by mobile devices.

Around 2006, some kind of genetic mutation happened: transcoder vendors realized that transcoders could be deployed in proxy mode and the whole Web could be transcoded behind the backs of the users and content providers, regardless of the presence of a mobile-optimized experience for any given site. This dangerous move posed a serious threat to the mobile ecosystem

To help us to understand this problem, I contacted Luca Passani, CEO of WURFL-Pro and creator of the WURFL libraries. He kindly answered my questions to keep us updated about this topic. The rest of this section was entirely written by Luca. For more information on this topic, see

3.1. What is a transcoder?

Defining transcoders requires a bit of care. Not only has the term “transcoders” been used in other industries, but there are also other terms in the mobile world that identify mobile-web transcoders and what they do: notably, content transformation and content adaptation. Content transformation is basically a synonym for “transcoding,” but content adaptation can also refer to development techniques that, while they bear some similarities to transcoding, are more traditional (and legitimate) approaches to the creation of mobile websites.

When you say “transcoders” to mobile web developers, they’ll most likely think of the proxies (from companies such as Novarra, InfoGin, Openwave, and ByteMobile) that network operators (Vodafone, Verizon, Sprint, etc.) install in their networks to intercept desktop web pages on their way to mobile devices and “reformat” them. By “reformatting,” I mean one or more of the following:

  • Splitting a full page into several smaller pages (which are supposedly more manageable for mobile devices)

  • Replacing the original graphics with images of lower size (and lower quality)

  • Adding an operator navigation bar on every page

  • Injecting advertising from unknown sources

  • Disabling mechanisms to detect successful download of applications, Java ME MIDlets, and other downloadable content

A less obvious effect of transcoders is that they often intercept and modify HTTP requests, removing the UAProf and/or the original user-agent string (which is key to recognizing devices and their capabilities).

I sometimes refer to transcoding as content transformation, but I never use “content adaptation” in this context because it would be very likely to generate confusion.


It is important to observe that WAP gateways are also proxies, but typically they do not interfere with the content that goes through them.

3.2. Why are transcoders a problem?

If you are a content provider or you build mobile websites, transcoders will make you furious. As soon as an operator deploys a transcoder, you will notice that you are unable to recognize devices using that operator’s network and its transcoder. If you have invested time and money in installing a framework to recognize mobile devices, all of your work will be lost: all you will see are web browser requests hitting your server. But that’s not all. Your branding is very likely to be disrupted, and so is your business model if, for example, you are no longer able to bill your customers directly, or if the ad banners you were injecting into your pages get discarded. These are things that have happened in practice and driven people mad. The icing on the cake was when operators started injecting their own advertising into other people’s content, trying to monetize other people’s efforts. It is no surprise that this has caused huge uproar amongst transcoder “victims.”

3.3. Parties involved in the transcoding problem

There are two main culprits behind the mess caused by transcoders: operators and transcoder vendors. Transcoder vendors have been guilty of telling operators, “Look, we have this fantastic technology that will bring the entire Web to all of your users. Loads of fantastic content you don’t need to pay for. Isn’t this great? And who cares if people who have invested millions in creating mobile sites complain…We are the future, after all!” This was deeply irresponsible, but of course part of the blame goes to the operators who actually believed this story, decided to deploy transcoders, and were flooded by a wave of complaints and thrown into disrepute. Of course, I had a role in making mobile developers focus on the problem in detail, but there were thousands of irritated developers who backed me instantly when I blogged about the issue.

3.4. What was the response from carriers after receiving complaints?

Officially, the carriers were silent. After all, these are huge organizations, and getting anyone to stand up and talk on behalf of their employees is simply not realistic. Behind the scenes, I got loads of support from people working for operators (particularly Vodafone Global, Germany, and UK). They recognized that I was right and that the decisions taken at the top floors were wrong.

3.5. Practical tips

There are some practical tips to be given, such as using XHTML MP as the markup or adding the Cache-Control: no-transform header to all your HTTP responses. They are all contained in the “Manifesto for Responsible Reformatting” at Yet, I don’t think that these alone are enough.

I firmly believe (and so do virtually all developers I have talked to) that developers have a right to a “clean” HTTP. They have the right to see HTTP headers as sent by the devices, and they have the right to insist that transcoders keep their dirty hands off other people’s content. Therefore, the main practical tip I want to give is not actually a very technical one: make noise. Complain. Blog and tell the world that operator X or Y is doing something wrong. Get a lawyer to send a letter to the operator and complain about copyright infringement. Using a metaphor, mugging someone to steal his wallet is a more serious offence than simple pickpocketing, yet pickpocketing is still a crime that should be prosecuted, and advice such as “stay home,” “hire a bodyguard,” or “don’t carry your wallet in your pocket” just doesn’t cut it! We have the right to live and work in a better ecosystem; transcoders are the law of the jungle.

3.6. Operator whitelists

Some operators offer whitelists where you can explicitly opt your site out of transcoding. Whitelists are not the way to go, though. Accepting whitelists means accepting the law of the jungle. If we were to agree to whitelisting our sites with each and every operator around the planet, we would effectively be giving operators a right they do not have (and must not have). The network and HTTP must be the same for everyone, or this would represent a giant step back for the whole mobile industry. Just think of what the Web would be today if website creators had to “register” their websites with different ISPs around the globe….

3.7. Making content transformation a standard

Transcoding is stealing. How do you standardize stealing? There is only one answer: you don’t. Stealing is illegal and must remain that way. Of course, transcoding vendors (mainly Novarra Inc., now part of Nokia) are trying to convince the W3C to create “quasi-standards” that allow transcoding in some instances, which in real deployments will naturally translate into “whenever the heck we want.” The problem here is that the corporations who sit at the W3C table call the shots in their own interests.


Nokia acquired Novarra Inc. in 2010, and at the time of this writing there is no information about how the Novarra transcoding politics will evolve.

For those who are curious, Novarra’s attempt at getting the W3C to ratify its way of doing transcoding (its Guidelines for Web Content Transformation) is available online at

3.8. Transcoder detection

There is an ongoing battle between transcoder vendors and the rest of the mobile ecosystem. There are ways to recognize a transcoder, by checking certain headers (either their values, or simply checking for their presence). Again, the Manifesto has more information. The problem is that the business model of transcoder vendors (transcode as much as you can) is in direct conflict with the business model of content owners (protect your content as well as you can). For this reason, transcoders are progressively making it harder for others to detect them, by removing those hints from HTTP requests. A day may come when developers will need to maintain a list of IP ranges to identify transcoders and treat requests coming from them specially (again, don’t forget to complain loudly if you spot one).

3.9. What to do after detection

Assuming you have detected a transcoder, adopting the Cache-Control: no-transform header and using mobile-specific MIME types and DTDs (XHTML Mobile Profile, for example) is your best bet to prevent your content from being touched. Serving your content from a hostname with a pattern such as m.*, wap.*, or *.mobi will usually also help.

I say “help” because this is not a guarantee. Your content is still at the mercy of the operators’ transcoder policies, and nobody will go after them on your behalf if they decide not to respect your directives.


Novarra and InfoGin are the two largest companies in the transcoding field. Other names include Openwave, ByteMobile, and Volantis.
  •  Server-Side Browser Detection and Content Delivery : Mobile Detection (part 2) - Detecting the Context
  •  Server-Side Browser Detection and Content Delivery : Mobile Detection (part 1) - HTTP
  •  Using Windows Phone 7 Technologies : Retrieving Accelerometer Data (part 2)
  •  Using Windows Phone 7 Technologies : Retrieving Accelerometer Data (part 1)
  •  Using Windows Phone 7 Technologies : Understanding Orientation and Movement
  •  Programming the Mobile Web : HTML 5 (part 4) - Client Storage
  •  Programming the Mobile Web : HTML 5 (part 3) - Offline Operation
  •  Programming the Mobile Web : HTML 5 (part 2) - The canvas Element
  •  Programming the Mobile Web : HTML 5 (part 1)
  •  Windows Phone 7 : Submitting Your First Windows Phone Application to the Windows Phone Marketplace
  •  Windows Phone 7 : Packaging, Publishing, and Managing Applications
  •  Mobile Application Security : Windows Mobile Security - Development and Security Testing (part 3)
  •  Mobile Application Security : Windows Mobile Security - Development and Security Testing (part 2)
  •  Mobile Application Security : Windows Mobile Security - Development and Security Testing (part 1)
  •  Programming the Mobile Web : Mobile Rich Internet Applications (part 2) - JavaScript Mobile UI Patterns
  •  Programming the Mobile Web : Mobile Rich Internet Applications (part 1) - JavaScript UI Libraries
  •  Windows Mobile Security - Kernel Architecture
  •  Windows Mobile Security - Introduction to the Platform
  •  iPhone Programming : Table-View-Based Applications - Building a Model
  •  Mobile Application Security : The Apple iPhone - Push Notifications, Copy/Paste, and Other IPC
    Top 10
    Windows Phone 7 Development : Building a Trial Application (part 2) - Connecting to a Web Service & Adding Page-to-Page Navigation
    Windows Defender
    Consuming a WCF Service with Silverlight
    Full-Text Indexing in SQL Server 2008
    Windows Phone 7 Development : Adding a WebBrowser Control
    Windows 7 : Navigating the Computer Security Maze
    Implementing Client Access and Hub Transport Servers : Understanding the Client Access Server (part 1)
    SQL Server 2005 : Extending Your Database System with Data Mining - Data Mining Applied (part 2)
    Programming with DirectX : Shading and Surfaces - Types of Textures
    Designing a Windows Server 2008 R2 Active Directory : Understanding the Single Domain Model
    Most View
    Biggest tips guide ever! (Part 4) - Broadband: Wireless Security
    Optimizing for Vertical Search : Optimizing for Local Search (part 1) - Check Your Local Listings
    Creating a Web Application with VB 2010 with Navigation and Data-Binding
    SharePoint 2010 : Physically Securing SharePoint Servers
    Reporting Services with SQL Azure : Deploying the Report & Creating a Subreport
    Windows Server 2008 : DHCP/WINS/Domain Controllers - Exploring Advanced DHCP Concepts
    Algorithms for Compiler Design: PEEPHOLE OPTIMIZATION
    Sharepoint 2010 : Maintaining SQL Server in a SharePoint Environment
    A First Silverlight Example: Creating a Web Site
    Saying Hello to Web Services
    Security Changes in IIS 7.0 : Reducing Attack Surface Area (part 2)
    Xen Virtualization : Installing Xen from Source
    Windows 7 : Detecting and Resolving Computer Problems (part 2) - Tracking Errors in the Event Logs
    Enterprise Patterns with WCF RIA Services
    Cloud Application Architectures : Machine Image Design
    Designing and Configuring Unified Messaging in Exchange Server 2010 : Unified Messaging Features
    The Language of Apple Platforms : Memory Management
    Mindgenius 4
    The New iPad
    Microsoft XNA Game Studio 3.0 : Displaying Images - Using Resources in a Game (part 3) - Sprite Drawing with SpriteBatch