3. Transcoders
With WAP 1.1, operator gateways were required to
precompile WML and make it lighter and more easily parsable by devices
with limited memory/CPU resources. Gateways were also relied upon to
manage cookies on behalf of the devices (which did not have enough
memory) and for integration with the operator’s backend (for example,
the gateway was able to inject the users’ phone numbers into HTTP
headers, so that authorized content providers could recognize them and
bill them for services).
With WAP 2 precompilation was no longer needed, but other aspects
of gateways were and still are today, to some extent. In a lot of
different contexts, the presence of a WAP gateway is beneficial to
developers and content providers.
Around 2002, some companies started selling tools to “mobilize”
web content. That is, users could type a URL into a field, and the
transcoder/content reformatter would chop it up into pages that could be
viewed by mobile devices.
Around 2006, some kind of genetic mutation happened: transcoder
vendors realized that transcoders could be deployed in proxy mode and
the whole Web could be transcoded behind the backs of the users and
content providers, regardless of the presence of a mobile-optimized
experience for any given site. This dangerous move posed a serious
threat to the mobile ecosystem
To help us to understand this problem, I contacted Luca Passani, CEO of WURFL-Pro and creator of the
WURFL libraries. He
kindly answered my questions to keep us updated about this topic. The
rest of this section was entirely written by Luca. For more information
on this topic, see http://wurfl.sourceforge.net/manifesto.
3.1. What is a transcoder?
Defining transcoders requires a bit of care. Not only has the
term “transcoders” been used in other industries, but there are also
other terms in the mobile world that identify mobile-web transcoders
and what they do: notably, content transformation and
content adaptation. Content transformation is
basically a synonym for “transcoding,” but content adaptation can also
refer to development techniques that, while they bear some
similarities to transcoding, are more traditional (and legitimate)
approaches to the creation of mobile websites.
When you say “transcoders” to mobile web developers, they’ll
most likely think of the proxies (from companies such as Novarra,
InfoGin, Openwave, and ByteMobile) that network operators (Vodafone,
Verizon, Sprint, etc.) install in their networks to intercept desktop
web pages on their way to mobile devices and “reformat” them. By
“reformatting,” I mean one or more of the following:
Splitting a full page into several smaller pages (which are
supposedly more manageable for mobile devices)
Replacing the original graphics with images of lower size
(and lower quality)
Adding an operator navigation bar on every page
Injecting advertising from unknown sources
Disabling mechanisms to detect successful download of
applications, Java ME MIDlets, and other downloadable
content
A less obvious effect of transcoders is that they often
intercept and modify HTTP requests, removing the UAProf and/or the
original user-agent string (which is key to recognizing devices and
their capabilities).
I sometimes refer to transcoding as content transformation, but
I never use “content adaptation” in this context because it would be
very likely to generate confusion.
Note:
It is important to observe that WAP gateways are also proxies,
but typically they do not interfere with the content that goes
through them.
3.2. Why are transcoders a problem?
If you are a content provider or you build mobile websites,
transcoders will make you furious. As soon as an operator deploys a
transcoder, you will notice that you are unable to recognize devices
using that operator’s network and its transcoder. If you have invested
time and money in installing a framework to recognize mobile devices,
all of your work will be lost: all you will see are web browser
requests hitting your server. But that’s not all. Your branding is
very likely to be disrupted, and so is your business model if, for
example, you are no longer able to bill your customers directly, or if
the ad banners you were injecting into your pages get discarded. These
are things that have happened in practice and driven people mad. The
icing on the cake was when operators started injecting their own
advertising into other people’s content, trying to monetize other
people’s efforts. It is no surprise that this has caused huge uproar
amongst transcoder “victims.”
3.3. Parties involved in the transcoding problem
There are two main culprits behind the mess caused by
transcoders: operators and transcoder vendors. Transcoder vendors have
been guilty of telling operators, “Look, we have this fantastic
technology that will bring the entire Web to all of your users. Loads
of fantastic content you don’t need to pay for. Isn’t this great? And
who cares if people who have invested millions in creating mobile
sites complain…We are the future, after all!” This was deeply
irresponsible, but of course part of the blame goes to the operators
who actually believed this story, decided to deploy transcoders, and
were flooded by a wave of complaints and thrown into disrepute. Of
course, I had a role in making mobile developers focus on the problem
in detail, but there were thousands of irritated developers who backed
me instantly when I blogged about the issue.
3.4. What was the response from carriers after receiving
complaints?
Officially, the carriers were silent. After all, these are huge
organizations, and getting anyone to stand up and talk on behalf of
their employees is simply not realistic. Behind the scenes, I got
loads of support from people working for operators (particularly
Vodafone Global, Germany, and UK). They recognized that I was right
and that the decisions taken at the top floors were wrong.
3.5. Practical tips
There are some practical tips to be given, such as using XHTML
MP as the markup or adding the Cache-Control:
no-transform header to all your HTTP responses. They are all
contained in the “Manifesto for Responsible Reformatting” at http://wurfl.sourceforge.net/manifesto. Yet, I don’t
think that these alone are enough.
I firmly believe (and so do virtually all developers I have
talked to) that developers have a right to a “clean” HTTP. They have
the right to see HTTP headers as sent by the devices,
and they have the right to insist that
transcoders keep their dirty hands off other people’s content.
Therefore, the main practical tip I want to give is not actually a
very technical one: make noise. Complain. Blog and tell the world that
operator X or Y is doing something wrong. Get a lawyer to send a
letter to the operator and complain about copyright infringement.
Using a metaphor, mugging someone to steal his wallet is a more
serious offence than simple pickpocketing, yet pickpocketing is still
a crime that should be prosecuted, and advice such as “stay home,”
“hire a bodyguard,” or “don’t carry your wallet in your pocket” just
doesn’t cut it! We have the right to live and work in a better
ecosystem; transcoders are the law of the jungle.
3.6. Operator whitelists
Some operators offer whitelists where you can explicitly
opt your site out of transcoding. Whitelists are not the way to go,
though. Accepting whitelists means accepting the law of the jungle. If
we were to agree to whitelisting our sites with each and every
operator around the planet, we would effectively be giving operators a
right they do not have (and must not have). The
network and HTTP must be the same for everyone, or this would
represent a giant step back for the whole mobile industry. Just think
of what the Web would be today if website creators had to “register”
their websites with different ISPs around the globe….
3.7. Making content transformation a standard
Transcoding is stealing. How do you standardize stealing? There
is only one answer: you don’t. Stealing is illegal and must remain
that way. Of course, transcoding vendors (mainly Novarra Inc., now part of Nokia) are trying to convince
the W3C to create “quasi-standards” that allow transcoding in some
instances, which in real deployments will naturally translate into
“whenever the heck we want.” The problem here is that the corporations
who sit at the W3C table call the shots in their own interests.
Note:
Nokia acquired Novarra Inc. in 2010, and at the time of this
writing there is no information about how the Novarra transcoding
politics will evolve.
For those who are curious, Novarra’s attempt at getting the W3C
to ratify its way of doing transcoding (its Guidelines for Web Content Transformation) is available
online at http://www.w3.org/TR/ct-guidelines.
3.8. Transcoder detection
There is an ongoing battle between transcoder vendors and the
rest of the mobile ecosystem. There are ways to recognize a
transcoder, by checking certain headers (either their values, or
simply checking for their presence). Again, the Manifesto has more
information. The problem is that the business model of transcoder
vendors (transcode as much as you can) is in direct conflict with the
business model of content owners (protect your content as well as you
can). For this reason, transcoders are progressively making it harder
for others to detect them, by removing those hints from HTTP requests.
A day may come when developers will need to maintain a list of IP
ranges to identify transcoders and treat requests coming from them
specially (again, don’t forget to complain loudly if you spot
one).
3.9. What to do after detection
Assuming you have detected a transcoder, adopting the Cache-Control: no-transform header and using
mobile-specific MIME types and DTDs (XHTML Mobile Profile, for
example) is your best bet to prevent your content from being touched.
Serving your content from a hostname with a pattern such as
m.*, wap.*, or
*.mobi will usually also help.
I say “help” because this is not a guarantee. Your content is
still at the mercy of the operators’ transcoder policies, and nobody
will go after them on your behalf if they decide not to respect your
directives.
Note:
Novarra and InfoGin are the two largest companies in the
transcoding field. Other names include Openwave, ByteMobile, and
Volantis.
