In spite of the obvious predominance of English, there are dozens of other languages that have a significant and growing presence in this mass of electronic documentation. And to complicate matters, each language may be coded in different character sets. For example, the same French text may be coded one way under MS-DOS (the cp850 character set), another way under MS-WINDOWS (cp1252), and still another way for Macintosh systems (the macintosh character set).

Any computer system that needs to manipulate documents on the Internet should in principle know what language each document is written in and how that language has been encoded; otherwise, it won't be able to display or print the documents correctly. Moreover, a system that doesn't know what language the documents in a database are written in won't be able to search that database intelligently. For example, when a user submits a search on the French word lime (in the sense of tool), the system should not retrieve documents that contain the English word lime (in the sense of fruit). Similarly, if the system does not know what language the document is written in, it will be difficult for it to invoke the appropriate machine translation process.

In theory, HTML provides for the explicit marking of the language and the character set of a document, but more often than not these are omitted in practice. It is unrealistic, moreover, to expect the user to provide this information, since the great majority of users cannot recognize more than a few languages with confidence. Hence, the best solution is to develop systems that are capable of automatically identifying both the language and the character set of electronic documents.

This is exactly what the SILC system does. When the user submits a document to the system, SILC attempts to determine what language the document is written in and the character set in which it is encoded. To do so, it possesses, for each of the language and coding pairs it handles, a model that assigns a certain probability to the text in question. The system also incorporates formal criteria that allow it to select the model that obtains the best score.

On texts that are sufficiently long (at least 50 characters), SILC's performance is practically flawless. Version 2.2 of the system can identify no less than 25 different languages, each of which may be encoded in two or three different character sets.

A demonstration of our language identification system is available.

Contact: Guy Lapalme.