The localization industry has long suffered at the hands of the character set problem. Any localization specialist who has strayed away from Western European languages on a single computing platform has been bitten by the complexities of this hairy subject, and the related subject of fonts.

The problem is not new. It finds its sources, dating back to the dawn of the computer age, in the need and desire to get computers to process text, in the existence of numerous languages and in the creativity of computer scientists and engineers. The result has been, as is well known, a plethora of text encoding schemes: the Babel tower of languages multiplied by the variety of computing platforms, compounded by the rapid evolution of the latter. And the result of that, in turn, is a lot of problems in data exchange, data durability, software porting and, of course, localization.

A little history

Problems call for solutions. In the case at hand, people started really suffering enough and devising a good solution in the 80’s. This is when the Unicode effort for a Universal Character Set started. 1993 saw the publication of Unicode 1.0 [1]. In 1993, the Unicode Consortium and the ISO sub-committee on character sets merged their efforts, resulting in the publication of Unicode 1.1 and ISO/IEC 10646:1993. Since then, new versions have been periodically published (3.1 is the latest [2], aligned with ISO/IEC 10646:2000 [3]), each one adding new characters so as to get closer to the goal of a character set that encompasses all of the world’s written languages.

Ten years after version 1.0, Unicode is well on its way to conquering the world. Knowingly or not, most computer users have been exposed to Unicode, use or generate data in Unicode, and, even when legacy encodings* are involved, often interact with the data through Unicode-based software.

Case in point: Microsoft has been an early member of the Unicode Consortium as well as an early adopter. Windows NT, initially released in 1993, is entirely Unicode-based (as are of course its descendants Windows 2000 and the forthcoming Windows XP). The Office suite has been slower to adopt Unicode, but many of its components (notably Word) have been Unicode-based since the 97 release and the others have been or will soon be made Unicode-based. Many other major Microsoft applications (Internet Explorer, SQL Server, etc.) have also adopted the standard. Therefore, many users today are literally bathed in Unicode, often without realizing it. Microsoft has stopped designing code pages for newly supported languages; these languages are supported only through Unicode. This alone indicates clearly that those in the localization industry that have not learned to deal with Unicode will need to do so shortly!

One of the design principles of Unicode was Han Unification. This principle says that a given Han character (Chinese, Japanese, Korean or Vietnamese) should have only one code point in Unicode, even though two or more languages use it, possibly with some variation in shape details. If the shape differences are so large that readers are unlikely to recognize the character, however, it gets a separate code point for each distinct variant. This is very much analogous to how we have never had separate code points for roman, italic, serif or sans-serif A (regardless of language), but runs counter to the long standing habit of having separate character sets—and therefore code points—for each Asian language. This policy has generated a significant amount of opposition to Unicode in Asia, especially in Japan. Much of that pushback, however, tends to dissolve when Japanese opponents learn that their beloved JustSystem Ichitaro word processor, a best seller in Japan, has been Unicode-based for about five years. Despite such evidence, the belief that Unicode is inadequate for Asian languages is still very much alive; see for instance the recent diatribe at http://www.hastingsresearch.com/net/04-unicode-limitations.shtml (but don’t miss the response from one of Unicode’s technical directors at http://slashdot.org/article.pl?sid=01/06/06/0132203).

Of course, JustSystem and Microsoft are far from alone in having adopted Unicode. Every major database supports it in one form or another; Web browsers have supported it for some time and the major ones are now based on it; a serious Linux internationalization effort is under way (see http://www.li18nux.org/), in good part based on Unicode. In fact, almost all the household names in the software industry have something to do with Unicode today. For an unfortunately incomplete list, see http://www.unicode.org/unicode/onlinedat/products.html.

Reference Processing Model

But just how does one go about applying Unicode today, in this period of transition when data and programs constitute a mixed bag of Unicode and non-Unicode? The answer lies in good part in a notion called the Reference Processing Model, which in turn has its roots in the effort of internationalization of the World Wide Web of the mid-nineties. Back then, the Web entirely revolved around html, but HTML was defined to be based on the ISO/IEC 8859-1 (a.k.a. ISO Latin-1) character set. Alas, this in principle denied Web access to any language not supported by this very limited encoding, i.e. to most languages. Things had to move, formally or not, and they did. Problems and ambiguities cropped up when people used other encodings in HTML. For instance, HTML defines the Numeric Character Reference (NCR) ‘é’ as an alternate representation of character number 233. But is it number 233 in the encoding used by the page, or in the official ISO Latin-1? Opinions diverged, interoperability suffered, etc.

The Reference Processing Model, first embodied in RFC 20704, was formalized to deal with these issues. It goes roughly as follows:

1. Define HTML in terms of Unicode characters, not bytes, glyphs, or characters in some other, non-universal encoding.
3. Allow the whole Unicode range of characters, from 0 to 0x10ffff inclusive, to be usable in HTML.
4. Allow use of any character encoding which can be transcoded† to Unicode (Unicode-compatible encodings). Independent of the actual encoding of a page, the behavior of an application (e.g. a browser) is prescribed to be the same as if processing happened as follows:
   1. Determine the encoding of any text received by the application and then interpret the text as a sequence of Unicode characters. This is equivalent to transcoding the entity to some Unicode encoding form and then reading it in that encoding form;
   2. All processing takes place on this sequence of Unicode characters;
   3. If the application outputs text, it can also transcoded it to any Unicode-compatible encoding.

It is noteworthy that Unicode’s coverage today is wide enough that Unicode-compatible encoding means just about any encoding. In fact, almost all important legacy encodings served as sources for the Unicode standard.

As mentioned above, this model was first formalized for HTML in RFC 2070; it was later built into the W3C’s HTML [4][5], built into the very first (and still current) version of XML [6] and is a crucial part of the W3C’s Character Model for the World Wide Web [7].

Formally, this model applies mostly to Web specifications. Nevertheless, a growing number of applications use it, at least informally; these applications implement a Unicode text model and transcode anything that they read or write in legacy encodings. Unfortunately, some other applications that claim to support Unicode encodings do so in the reverse way: when presented with Unicode input, they transcode it to their legacy code page (thereby losing any character not in that code page), do the processing in that code page and transcode back the output to Unicode. This strategy offers the advantage of a very quick Unicode enablement, but the limitations are obvious. Buyers beware! ‘Unicode-enabled’ may not mean all that it seems to imply!

Unicode and Fonts

Localization specialists are very much aware of the issues with fonts, some of which they encounter every day when localizing to languages using foreign scripts. Does Unicode change the picture? The short answer is that it has already done so, and a longer answer is that more change is happening now or coming soon.

First the changes already under our belts: the TrueType specification has been based on Unicode from day one. Since TrueType is used by MacOS, Windows and, iNCReasingly, by various Unix systems, this is an area where Unicode is already prevalent. Technically, this reliance on Unicode means that TrueType fonts normally contain a table mapping the glyphs (character images) in the font to Unicode characters. One short-term problem is that toolmakers and font designers are not all aware of that aspect, do not all understand it correctly and, consequently, we end up with fonts that lie about their contents. Many fonts claim (through their Unicode mapping table) to contain characters in the 0-255 range (Latin-1), but they actually contain other stuff. Until applications have been migrated to Unicode, there is actually an incentive for fonts to lie in this way, when the font is meant to be used with an application that supports only Latin-1. This is the old font-mapping trick to access foreign characters. For reasons not to go that route, see considered harmful at http://babel.alis.com/web_ml/html/fontface.html.