Creation of dictionary entries from GCIDE_XML files

Context

The GNU version of The Collaborative International Dictionary of English is a well-formed XML version of the CIDE. GCIDE is a freely available set of ASCII files containing the marked-up text of a substantial English dictionary (more than 130 000 headwords).

From the introduction to each file:

               This dictionary was derived from the
             Webster's Revised Unabridged Dictionary
                     Version published 1913
                   by the  C. & G. Merriam Co.
                       Springfield, Mass.
                     Under the direction of
                    Noah Porter, D.D., LL.D.

                            and from
               WordNet, a semantic network created by
                  the Cognitive Science Department
                     of Princeton University
                      under the direction of
                       Prof. George Miller

                 and is being updated and supplemented by
             an open coalition of volunteer collaborators from
                           around the world.

         This electronic dictionary is the starting point for an 
    ongoing project to develop a modern on-line comprehensive encyclopedic
    dictionary, by the efforts of all individuals willing to help build a
    large and freely available knowledge base.  Contributions of data,
    time, and effort are requested from any person willing to assist creation
    of a comprehensive and organized knowledge base for free access on the
    internet.  Anyone willing to assist in any way in constructing such a
    knowledge base should contact:

         Patrick Cassidy          pc@worldsoul.org
         735 Belvidere Ave.       Office: (908)668-5252
         Plainfield, NJ 07062
         (908) 561-3416

       Last edit January 17, 2002.

In 2002, Patrick Cassidy converted the original SGML-like ASCII files into a well-formed XML document (about 56 MB). The set of tags used in the dictionary is given in the GCIDE's tagset.web file. The files that describe the structure of the GCIDE are available at http://micra.com/dictionary/. The complete XML document is composed of a main document, which refers to several auxiliary documents, one for each letter of the alphabet. Each one of these auxiliary files contains the information about the words starting with a given letter. The mainXML file is uses System entities that include these auxiliary files to create a global XML document. Only the global XML file is a well-formed XML document.

This XML version is very useful in order to extract some parts of the information but it keeps most of the original tagging that had been developed for printing purposes. In particular, the whole document is seen as a series of paragraphs. To retrieve all the information for a given head word, one must parse from a given point until the next headword. This cumbersome and error-prone process is not very convenient for Natural Language Processing purposes.

We present here a new XML version of the GCIDE with the following characteristics:

Getting the dictionary

For the impatient interested in the final product, i.e. the XML files of the dictionary:

Conversion process

The conversion has been done using a bash script that starts from a directory containing the old GCIDE_XML files, modified only slightly to correct obvious tagging errors that were inducing validation errors, and creates a new directory with a new structure. As the original file is a valid XML file, we used XSLT 2.0 stylesheets to create the dictionary entries.

The script is given here. Here are the main steps

  1. Create a file containing only XML character entities;
  2. Create a separate file for each letter and sections of the dictionary;
  3. Create the main file including the files for each letter and sections;
  4. Create a single file combining all the files in case the XML processor does not allow XInclude processing.
  5. Create a new directory that contains only the file with entries and zip it

Originally, we intended to create a simple structure from the old one, but given the size of the original file and different types of tagging that had been used over the years, we finally decided to keep most of the original tags and we more or less transposed into a schema the original DTD (that itself had been created automatically from the files). We leave it to other volunteers to redesign the entries along a simpler schema.

We created an Excel file starting from the original DTD to define the mapping between the original tags and attributes and the ones used in the new. As an XSLT stylesheet cannot deal easily with native excel formats, we export a CSV version.

We developed a conversion stylesheet that takes an input the tag mapping and produces a part of the RNC schema and a part of the XSLT stylesheet for the conversion process. These files are then included into the full Relax NG (compact form) schema and the conversion stylesheet.

The conversion stylesheet is a relatively ad-hoc process that does the following transformations on the content of the p tags:

The XML transformation process was developed using the Oxygen XML Editor and appropriate processing instructions have been inserted in the XML files. But external validation using Jing or transformation using Saxon can be used.

Access to the files

To continue this work (or debug it...), you might like to look at the following files and directories:

Exploring the dictionary

You can use this program to explore and extract entries from the huge XML file, which can then be displayed in an HTML page with this XSLT stylesheet.

To get an idea of the content of the dictionary, you can search for all entries matching a given regular expression on our server, which can be

The regular expression is matched case-insensitively on the whole key, i.e. the content of the first hw tag but keeping only letters, spaces, dashes and apostrophes.

Find «.» for any char; «.*» for any sequence, possibly empty, of chars

Given the fact that a large XML has to be read, this operation can take some time on our server.

Please, do not use this form to get hundreds of dictionary entries, download the above files instead.

If the first letter of the regular expression is a letter then only the XML file of the words starting with this letter will be read, otherwise the whole XML dictionary file will be traversed.

The retrieved entries from this query are displayed with this XSLT stylesheet. It does not deal with all tags so some parts of the entries might appear verbatim within red tags, but it gives an idea of the content of the entries.

More info

If you have any problem (or fun!) using these files, contact

Guy Lapalme, DIRO (Computer Science Department), Université de Montréal.
October 2009 - Work done during a stay at the LIF, Université de Marseille