1. Introduction

The globalization of Internet is leading to a number of new requirements. These include the need for multi-media communication containing textual components across nationalities and cultures, rapid translation of electronic messages, and access to multi-lingual databases and help facilities representing the textual compounds of multi-media servers. Images and sounds in the multi- media services help and provide user friendliness but the textual compound in such services still represents the biggest core of such services. Human beings are used to communicate in the language of their origin either orally or as a written text. Ideally, each individual must be able to work or play as efficiently, effectively and as easily with remote users via international telematic services, as he can with local users from his native domain.

One of the most important compound of multi-media services and most frequently used by the users of telematic services, is reading the displayed text, no matter what is its origin. Clearly many applications of all kinds, from public information sources such as Directories, Library and Distance learning applications and all diverse services including shopping and trading provided through World Wide Web service, require a sort of displayed text. The recent and ongoing explosion of the development and use of multi-media services is a striking example of culture dependent software based communication.

For an application to support any language of choice, a user needs to be able to input and output text in that language, including all the characters, accents and diacritical marks as well as all special sings necessary to support particular cultural environment. In addition, a user needs to be able to receive and possibly convey information that may be in a convention or notation different to his native one, and peculiar to a particular remote nationality or culture (including business or scientific sub- cultures).

In summary, users of telematic multi- media based service of the global information infrastructure need to be able to:

• use the service in a language and script of their choice, both for input and output
• work in their own national or cultural environment
• switch between different languages and scripts.

In one word the telematic based services should be capable to provide Transparent Language Processing.

2. Transparent Language Processing - User Requirements

The current way of data transmission and presentation in the telematic services is the English language and the basic encoding system used for English - '-bit ASCII. There are some exception like in e-mail service (i.e. MIME and X.400) where 8-bit encoding is possible but this is seldom used due to the fact that some e-mail gateways on the global network still are hostile to 8-bit encoding and the text prepared with 8-bit encoding scheme is distorted. Furthermore, 8-bit encoding schemes are built around some region and do not cover many languages e.g. Europe needs more than five 8-bit coded character sets tables: Latin 1, Latin 2, Cyrillic, Greek and Latin 6 (Baltic rim). The WWW according to the current standards [1] currently support only the coding and repertoire provided in Latin 1-ISO 8859-1 [2]. Due to historical facts other encoding schemes are used within network services, this is true especially when existing documents or data are provided through WWW. Usually these are local types of encoding and are not readable from sites outside the region. Servers with X.500 application use the local language (i.e. Poland and Norway) of the country and for that reason the retrieval of the information for a user outside the country is very difficult.

In order to easily classify the various needs of users accessing telematic services a framework [3] was proposed for identification of the various levels of language support and processing within a telematic service. The underlying concept is that the language and encoding of data should be transparent to the end user accessing the data.

The Levels of Transparent Language Processing (TLP) range from Level 0, which is considered the bare minimum for non-English use of telematic services to Level 3, which delves into the realm of artificial intelligence and automatic translation of text. The levels can be represented as follows:

Level 0 TLP

The first level, Level 0 is considered to be the minimum level of functionality needed to be able to access and use of data written with language which is not English. The primary purpose is to convert human- readable text into an encoding format that the user's operating environment can understand.

The level 0 of TLP has already been implemented in some areas, but all existing solutions have not been standardized. The solutions are implemented either partially and in an inconsistent manner, between vendors, service providers and developers.

Level 1 TLP

The next level of TLP enables the user to read text that was originally in a different writing system or script (transliteration), and to have culturally expected formatting of elements such as date, time, and numbers (internationalization - i18n) [4].

To be able to access and read data on the WWW that may be encoded in a character set or script that is not supported in the user's operating environment, the data needs to be transliterated to a readable form. Transliteration has not be widely implemented, and has not been standardized in a manner that leads itself well to cross - language use. Once data is received, it may contain strings that, when viewed, would be expected to take on a certain cultural format to be unambiguously identifiable or understandable to the reader.

Level 2 TLP

In telematic processing, a message may originate from any node in the data path from source data to target user. If the message originator has no knowledge of the language the end user is using, the text may appear to the user in a language he does not know. Another problem is when queries are made to remote services that may have their data stored in another language. This may necessitate re-sending the query in the service's predominant default language to be able to successfully acquire any needed information that may reside on it. For that reason Level 2 is expected to provide a consistent translation memory mechanism by which telematic applications can look up message given a language pair, string of text, and a context identifier. Translation memory will not be effective unless it is mass-produced for needed language-pairs and categories, and made available either by downloading from cyberspace, or shrink-wrapping add- on lexicons.

Level 3 TLP

The ultimate in multilingual, global "Web- surfing", e-mail, directory access, and other uses of telematic services, is to be able to access textual information, and have the ability to have it in your native tongue so that you can read the content even if it is written in a language you do not understand. That is the goal of level 3 TLP.

Let's see at what stage of development are the current network services in the context of user requirements regarding transparent language processing. The next section will examine the two most frequently used services: WWW and e-mail. E-mail is the oldest and most widely used service providing one to one communication and the WWW is the largest service incorporating collection of data, servers and browsers. Each component of the WWW is really nothing more than an object in the most dynamic, and fastest growing, distributed application on the Internet today. By looking at the WWW as a single application, it is quite obvious that the needs of WWW is the representation of the multilingual data available today on the service to the reader in a sensible manner.

3. The TLP and network applications:
what is currently on the scene

3.1 Technical specifications

Most of the standards and documents specifying the generic network services have US origin and for that reason no specific attention was paid in the past to the needs and user requirements concerned with multi-linguality and TLP. Some attempts to address multi-linguality on the Internet have used the same partial solution which have traditionally characterized the software industry: create a solution in English and then produce separate versions in French, Spanish, German etc. The common denominator of the generic services is Level 0 of TLP or lower.

In the last year the Internet community due to the big emerging Asian markets and the demands of the users with different needs for language processing started to re-work on the some of service features for provision of support of Level 0 TLP.

The languages and the multi-linguality of the WWW service are addressed in the last IETF draft on HTML internationalization [5]. The basic scope of the document is to remove the restriction to the ISO 8859-1 coded character set as a document character set encoding which cater for a subset of the Western European languages only and to allow the use of the ISO 10 646-1/ UNICODE [6] - encoding which allows much larger set of characters and as a such representation of more than 75 languages. Together with the HTML 2.0 RFC specification, it defines a new version of HTML to be known as "HTML 2.1". It also defines HTML as an Internet Media Type (RFC 1590) [8] and MIME Content Type (RFC 1521) [9] called "text/html", or "text/html; version=2.1".

To ensure interoperability and proper support for at least ISO 8859-1 in an environment where character encoding schemes other than ISO 8859-1 are present, user agents must correctly interpret the character set parameter accompanying an HTML document received from the service. Further more, conforming user-agents are required to at least parse correctly numeric character references within the range of the Basic Multi-lingual Plane (BMP) of ISO 10 646- 1. For that reason the conformance of user agents with i18n HTML is defined as follows:

• it parses the characters of an HTML document into data characters and mark up according to SGML
• it supports at least the ISO 8859-1 character encoding scheme and process each character in the ISO Latin Alphabet No.1 and for interoperability purposes in an environment where character encoding schemes other than ISO 8859-1 are present, user agents must correctly inerpret the character parameter accompanying an HTML document received from the network.
• Furthermore, conforming user-agents are required to at least parse correctly numeric character references within the range of the Basic Multilingual Plane (BMP) of ISO 10 646-1. This is code- by-code identical with the Unicode standard.
• It allows the user to express all form field values specified in an HTML document and to (attempt to) submit the values as requests to information services.

The TLP in MIME mailers [8] is lower than Level 0. The term "charset" in MIME is used to designate a character encoding, rather than a coded character set as the term may suggest. A character encoding is a mapping (possibly many-to-one) of a sequence of octets to a sequence of characters taken from one or more character repertoires. A coded character set is a mapping between individual bit patterns and individual characters from a single character repertoire.

3.2. The processing model

The processing model for the TLP features used in the IETF draft for HTML 2.1, is the SGML concept of a document character set. Because there are various widely differing encoding of text, SGML does not directly address the question of how characters are encoded e.g. in a file. SGML views the characters as a single set (called a "character repertoire"), and a "code set" that assigns an integer number (known as "character number") to each character in the repertoire. The document character set declaration defines what each of the character numbers represents [10].

The document character set, in the SGML sense, of HTML 2.1 is the Basic Multilingual Plane of ISO 10646:1993, also known as UCS-2. Making UCS-2 the document character set does not create non conformance of any expression, construct or document that is conforming to HTML 2.0. It does make conforming certain constructs that are not admissible in HTML 2.0. One consequence is that data characters outside the repertoire of ISO- 8859-1, but within that of UCS-2 become valid SGML characters. Another is that the upper limit of the range of numeric character references is extended from 255 to 65533.

HTML, as an application of SGML, does not directly address the question of how characters are encoded as octets in external representations such as files. This is deferred to mechanisms external to HTML, such as the HTTP protocol, or MIME for electronic mail.

For the HTTP protocol [19], the way characters are encoded is defined by the "charset" parameter of the "Content-Type" field of the header of an HTTP response. For example, to indicate that the transmitted document is encoded in the "JIS" encoding of Japanese [11], the header will contain the following line:

Content-Type: text/html; charset=ISO-2022-JP

The default charset parameter in case of the HTTP protocol is ISO-8859-1. If HTML documents are transferred by electronic mail, the character encoding is defined by the "charset" parameter of the "Content-Type" MIME header line [8].

Whatever the external character encoding actually is, the reference processing model translates it to a representation of the document character set before processing specific to SGML/HTML.

With the document character set being the full ISO 10646 BMP, the possibility that a character cannot be displayed due to lack of appropriate resources (fonts) cannot be avoided. Because there are many different things that can be done in such a case, the HTML 2.1 document does not recommend any specific behavior. Depending on the implementation, this may also be handled by the underlying display system and not the application itself.

The HTML 2.1 document provides additional parameter: the language tag. Language tags can be used to control rendering of a marked up document in various ways: character disambiguation, in cases where the character encoding is not sufficient to resolve to a specific glyph; quotation marks; hyphenation; ligatures; spacing; voice synthesis, etc. Independently of rendering issues, language markup is useful as content markup for purposes such as classification and searching (sorting, ordering).

The language attribute, LANG, takes as its value a language tag that identifies a natural language spoken, written, or otherwise conveyed by human beings for communication of information to other human beings.

The syntax and registry of HTML language tags is the same as that defined by RFC 1766 [12]. In summary, a language tag is composed of one or more parts: A primary language tag and a possibly empty series of subtags. The namespace of language tags is administered by the IANA. Example tags include:en, en-US, en-cockney, i- cherokee, x-pig-latin

4. Applications and products

4.1 Data representation and manipulation requirements

According to the last i18n html draft document [5] the entity manager, the parser, and the application deal only with characters of the document character set. A display-oriented part of the application or the display machinery itself may again convert characters represented in the document character set to some other representation more suitable for their purpose. In any case, the entity manager, the parser, and the application, as far as character semantics are concerned, are using the HTML 2.1 document character set only.

The rendering of elements is meant to be controlled (in part) by the LANG attribute.

Specific user preferences set within the browser should override the value of the LANG attribute, which in turn overrides the value specified by the LANG attribute of any enclosing element. If none of these are set, a suitable default, perhaps controlled by the user's locale, is proposed to be used to control rendering.

A common misconception is that the existing popular GUI based browser represent the only display technology that need the support of multi-linguality and TLP. In reality , this is very far from the truth. There are TTY based browsers and those supporting people with sight disability. In GUI based browsers the TLP level 0 and 1 can be easily supported by solving the font-handling problems, TTY based browsers and speech synthesis systems, will need support for transcription, transliteration or translation.

In addition to the HTML presentation requirements, there are still data presentation requirements for things like date, time, weights, currency and other culturally dependent items. Presentation and processing of these data is not specified in the draft i18n document [5]. Other yet unsolved problems are related to the input facilities. User would like to input data in form in the manner he is accustomed to, using the language of his choice. He would like then to send the data to a server and will expect the server to accept the data, even if it cannot necessarily process it.

4.2 Some possible solutions

The waste majority of browsers in the WWW today have no support at all for multilingual data presentation. Besides the obvious font handling there are additional problems if levels higher than 0 need to be supported. The WWW browsers available today are uniformly incomplete in their support of input and display methods for date, time, weight, currency etc.

For servers, the problems fall into 2 categories: data received and data sent. The encoding and the character sets used must be specified in standard form in the forms used for sending or receiving documents. Current HTTP servers, however, do not generally include an appropriate charset parameter with the Content-Type header, even when the encoding scheme is different from the default ISO-8859-1. This is not acceptable behavior, and as such must be discouraged, but some preventive measures can be taken to minimize the effects.

In the case where a document is accessed from a hyperlink in an origin HTML document, a CHARSET attribute can be added to the attribute list of elements with link semantics (A and LINK), specifically by adding it to the linkExtraAttributes entity. The value of that attribute could be considered as a hint to the User Agent as to the character encoding scheme used by the resource pointed to by the hyperlink.

4.3 Early birds - some products claiming TLP provision

4.3.1 ALIS Technology multi-lingual browser

Through the partnership with Spyglass, Alis Technologies has integrated some language-handling capabilities with Spyglass Mosaic core technology to create the Alis multilingual browser. The new browser allows users around the world to explore the WWW service and see the documents in the language they are written. (ALIS is a trademark of ALIS Technologies Inc, Quebec, Canada, Accent is a trademark of Accent Software International Ltd. Windows and Windows 95 are trademarks of Microsoft Corporation. All other trademarks are owned by their respective manufacturers.)

The first version of the multilingual browser provides interfaces in French, Italian, German, Spanish, Russian and English. When users change from one language to another, the menus, messages and on-line help appear in the selected language. What's more, the hyperlink buttons in the interface lead to different sites, depending on the language of the interface. The Alis multilingual browser features Unicode-enabled language support, but the current version supports the presentation of the repertoire of following 8-bit based encoding: ISO 8859/1,2,5,7), Ms 1251, 1252, 1253 and KOI8. In general, the ALIS browser supports level 0 of TLP.

Similar facilities are provided in the latest (beta) version of the Netscape browser.

4.3.2 The ACCENT multi-lingual suit

The other new product in the field are the products recently announced by Accent. Accent Software International Ltd. announced a suite of applications introducing multi-linguality to some generic network services. Including six multilingual applications - a viewer, a standalone browser, a browser add-on, an HTML authoring tool, an e-mail add-on and e-mail reader - the Accent suite of Internet products provides some multilingual solution for working within Internet environment.

Accent's viewer is designed to work with other browsers as a help application which enables viewing of multilingual content. The multilingual browser can be used either as a standalone browser solution or as an add-on to an existing browser.

Accent's multilingual add-on to other browsers is designed to be activated when a user clicks on a link to a Web page created in a different language. Accent's browser add-on is fully compatible with leading browsers from companies such as Netscape, Spyglass, Spy and Netcom.

The Accent HTML technology enables the creation of Web pages in Latin, non-Latin and bi-directional alphabets in the same document. Accent offers the ability to author World Wide Web content in over 30 languages, independent of the location or localization of the Windows operating system. An user can create and post on the Internet marketing material in Russian, Arabic or Turkish, as well as in English, Spanish, German, French, etc., without obtaining language-specific operating systems or keyboard(s).

Accent's multilingual e-mail add-on is designed to create, send and receive electronic mail in multiple languages, under any language version of Windows. For example, a Russian speaker living in New York could create a Russian message in the English version of Windows and send it to an associate in Germany using German Windows or Arabic, despite the fact that the direction of writing is different. Accent's multilingual mail add- on is designed to work in tandem with existing e-mail software packages such as cc:Mail, MS Mail through MAPI (Mail Application Programming Interface).

4.3.3 The Character set enabled server

The proper work of browsers and user agents with TLP features depends on the capability of the server to respond with information about the character sets and the language used in the documents. The waste majority of servers in the WWW today have no support at all for multilingual data presentation. One attempt to provide such facilities are the patches added to the Apache server (http://www.apache.org) within the WEB site of the Laboratory for Open Systems and Networks of Jozef Stefan Institute, Slovenia (http://www.e5.ijs.si). Apache server is a file based that deduces file type and its language from file suffix. For example, the file welcome.html.sl is a file written in Slovene and its MIME type is "text/html". By adding some additional data to the file name i.e. welcome.html.lt2.sl (lt2 denotes ISO 8859- 2 character set) the browser displays the document in Slovene without asking the user to specify the character set but just the language of his choice. Such suffixes are configurable with AddCharset command in the configuration files. The command takes two arguments: the first is the character set encoding and the second the file suffix corresponding to that character set encoding.

The future work is focused to provision of content negotiation, i.e. the server should have capability to negotiate the encoding and if the browser can not accept the current encoding of the document then proper character sets conversion is provided by the server if such capability does not exist within the browser. This work will be based on the experiences gathered through incorporation of the C3 conversion system [16] in two mailing systems: Z-mail and EXMH. With C3 capabilities these mailing systems provide the Level 0 and Level 1 of TLP i.e.: character sets conversion and cultural driven transliteration and transcription [17].

5. TLP: a service or an user tool - the IV Framework project MAITS

The project MAITS has identified the needs of the users in the global multi- lingual work environment for several levels of TLP. To meet these requirements MAITS intends to create API to enhance some of the existing services and standards for provision of globalization and internationalization. In particular MAITS will create specifications for Level 0 to Level 3 of an API as an enhancement to the C Programming Language, implement the API for Level 0 and Level 2, enhance some products for provision of e-mail, X.500, WWW and SQL to use Level 0 to Level 2 of TLP and do some feasibility studies and demonstration for Level 3 TLP. The application scenarios where MAITS can be used as a solution are:

Level 0 TLP - Character set Conversion

The classic example here is a user working with a PC under PC-DOS, with a default code page cp850, accessing information which is encoded in Latin 1, ISO 8859. All accented characters would be incorrectly displayed and/or input unless codeset conversion took place. With MAITS, the application (X.400, X.500, MIME, SMTP, WWW and SQL) will query the client environment and the server (or source data) formats and then utilize the correct codeset conversion engine accordingly.

Level 1 TLP - Transliteration and Localized Cultural String Formats

In this scenario, a user using WWW follows a link from their home machine e.g. in Paris to a university of Athens. The user does not know how to read Greek, but he knows what reference works he is looking for. In this case, with Level 1 TLP enabled, the Greek script would be automatically transliterated into Roman characters, in a manner sensitive to the French language. This scenario applies for e-mail, directory services (X.500) and SQL vendor applications.

Level 2 TLP - Translation Memory

Now, the user switches over to use directory services, using X.500 to find expert in Greek philosophy throughout Europe. The problem is, the data in the X.500 directories is entered in the local language at each site, German in Germany, Italian in Italy, etc. Using Level 2 TLP, the X.500 service translate the keywords and attributes of the query into each appropriate target language after first submitting the query in the source language and English to maximize the number of valid search "hits". If the target language is encoded in, e.g. Cyrillic or Greek, and the user has no ability to display the script, the Level 1 transliteration can pop into effect to make the results usable to the end user.

Level 3 TLP - Machine Translation

Once an expert is found, a WWW reference is found in the directory, so the end user attempts to access the information he needs. The user finds that the Web page is written in Russian, so he requests a machine-translated copy to be sent to him via e-mail (or some other means). The machine translated copy is then read to glean enough information out of the text to decide if further research is needed along those lines or not.

6. Conclusion

According to these application scenarios and existing technical specification no one can give a global answer to the question: will the TLP be a user tool or a part of a network service? Some of the scenarios will find better solution in the provision of the required facilities in the user tools such as browsers (GUI or TTY based). Some current applications already provide Level 0 TLP.

Some TLP facilities can be found as a service added to some other services such as the weather broadcasting service on the Web in Canada/Quebec where the weather broadcast is translated by MT from English to French.

The Globalink software recently announced [18] that the next generation of the Machine Translation (MT) solutions will be integrated on the Netscape browser so the user can translate the document he is viewing still on-line and that such software will be a part of every PC as it is today a spell checker. Certainly, such announcement are still too optimistic as much more work will be required for provision of good solutions for all problems related to data presentation, data manipulation, data input and data display requirement.

Solutions are still sought for the cultural elements and related issues like searching, retrieval, ordering, sorting in multi-lingual environment. Transliteration and transcriptions as a part of the cultural elements are not yet provided with few exception e.g. for Greek. Certain facilities will remain to be provided within the network service itself. The most probable candidate is the Memory Translation within X.500 service. By considering the last announcements and the emerging of new products with some support for TLP the global village of the information society looks little bit closer but to make it really close further work is still needed.

References

[1] T. Berners-Lee and D. Connolly, "Hypertext Markup Language - 2.0", RFC 1866, MIT/W3C, November 1995. T. Berners-Lee, R.T. Fieldieng and H. Frystyk Nielsen, "Hypertext Transfer Protocol - HTTP/1.0", Work in progress (draft-ietf-http-v10-spec-00.ps), MIT, UC Irvine, CERN, March 1995.

[2] ISO 8859-1:1987. International Standard -- Information Processing -- 8-bit Single-Byte Coded Graphic Character Sets -- Part 1: Latin Alphabet No. 1.

[3] Multilingual Application Interface for Telematic Services, CEC funded project from IV Framework, Telematic Programme, LE-1347, November 1995

[4] ISO/IEC JTC1 SC22WG20 DTR 1107 Framework for Internationalization, September 1995

[5] F.Yergeau, G.Nicol,G.Adams, M.Duerst, "Internationalization of the Hypertext Markup Language", Work in progress (draft-ietf--html-i18n- 01.txt), September 1995

[6] ISO/IEC 10646-1:1993. International Standard -- Information technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane. The Unicode Standard - Worldwide Character Encoding -- Version 1.0", Addison- Wesley, Volume 1, 1991, Volume 2, 1992

[7] ISO 8879:1986. International Standard -- Information Processing -- Text and Office Systems -- Standard Generalized Markup Language (SGML).

[8] RFC1521: N. Borenstein and N. Freed, "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, Bellcore, Innosoft, September 1993.

[9] J. Postel, "Media Type Registration Procedure", RFC 1590, USC/ISI, March 1994.

[10] C. F. Goldfarb, "The SGML Handbook", Y. Rubinsky, Ed., Oxford University Press, 1990.

[11] J. Murai, M. Crispin and E. van der Poel, "Japanese Character Encoding for Internet Messages", RFC 1468, Keio University, Panda Programming, June 1993.

[12] H. Alverstrand, "Tags for the Identification of Languages", RFC 1766, UNINETT, March 1995.

[13] ISO 639:1988. Codes pour la representation des noms de langue.

[14] "Ethnologue, Languages of the World", 12th Edition, Barbara F. Grimes editor, Summer Institute of Linguistics, Dallas, 1992.

[15] ISO 3166:1993. Codes pour la representation noms de pays.

[16] B.Jerman-Blazic, Tool supporting the internationalization of the generic network services, Computer Networks and ISDN Systems, 27 (1994), 429-435

[17] B.Jerman-Blazic, A.Gogala, Seventh International Unicode Conference, San Jose, September 14-15, 1995, Proceedings, Multilingual Application Interface for Internationalization of the Network Services

[18] Sunday's observer, October 22, 1995

---

Contact: Borka Jerman-Blazic: jerman-blazic@e5.ijs.si
Andrej Gogala: andrej@e5.ijs.si
Dusan Gabrijelcic: dusan@e5.ijs.si
Laboratory for Open Systems and Networks
Jozef Stefan Institute
61000 Ljubljana
Slovenia