The GNU version of The Collaborative International Dictionary of English is a well-formed XML version of the CIDE. GCIDE is a freely available set of ASCII files containing the marked-up text of a substantial English dictionary (more than 130 000 headwords).
From the introduction to each file:
This dictionary was derived from the
Webster's Revised Unabridged Dictionary
Version published 1913
by the C. & G. Merriam Co.
Springfield, Mass.
Under the direction of
Noah Porter, D.D., LL.D.
and from
WordNet, a semantic network created by
the Cognitive Science Department
of Princeton University
under the direction of
Prof. George Miller
and is being updated and supplemented by
an open coalition of volunteer collaborators from
around the world.
This electronic dictionary is the starting point for an
ongoing project to develop a modern on-line comprehensive encyclopedic
dictionary, by the efforts of all individuals willing to help build a
large and freely available knowledge base. Contributions of data,
time, and effort are requested from any person willing to assist creation
of a comprehensive and organized knowledge base for free access on the
internet. Anyone willing to assist in any way in constructing such a
knowledge base should contact:
Patrick Cassidy pc@worldsoul.org
735 Belvidere Ave. Office: (908)668-5252
Plainfield, NJ 07062
(908) 561-3416
Last edit January 17, 2002.
In 2002, Patrick Cassidy converted the original SGML-like ASCII files into a well-formed XML document (about 56 MB). The set of tags used in the dictionary is given in the GCIDE's tagset.web file. The files that describe the structure of the GCIDE are available at http://micra.com/dictionary/. The complete XML document is composed of a main document, which refers to several auxiliary documents, one for each letter of the alphabet. Each one of these auxiliary files contains the information about the words starting with a given letter. The mainXML file is uses System entities that include these auxiliary files to create a global XML document. Only the global XML file is a well-formed XML document.
This XML version is very useful in order to extract some parts of the information but it keeps most of the original tagging that had been developed for printing purposes. In particular, the whole document is seen as a series of paragraphs. To retrieve all the information for a given head word, one must parse from a given point until the next headword. This cumbersome and error-prone process is not very convenient for Natural Language Processing purposes.
We present here a new XML version of the GCIDE with the following characteristics:
For the impatient interested in the final product, i.e. the XML files of the dictionary:
The conversion has been done using a bash script that starts from a directory containing the old GCIDE_XML files, modified only slightly to correct obvious tagging errors that were inducing validation errors, and creates a new directory with a new structure. As the original file is a valid XML file, we used XSLT 2.0 stylesheets to create the dictionary entries.
The script is given here. Here are the main steps
Originally, we intended to create a simple structure from the old one, but given the size of the original file and different types of tagging that had been used over the years, we finally decided to keep most of the original tags and we more or less transposed into a schema the original DTD (that itself had been created automatically from the files). We leave it to other volunteers to redesign the entries along a simpler schema.
We created an Excel file starting from the original DTD to define the mapping between the original tags and attributes and the ones used in the new. As an XSLT stylesheet cannot deal easily with native excel formats, we export a CSV version.
We developed a conversion stylesheet that takes an input the tag mapping and produces a part of the RNC schema and a part of the XSLT stylesheet for the conversion process. These files are then included into the full Relax NG (compact form) schema and the conversion stylesheet.
The conversion stylesheet is a relatively ad-hoc process that does the following transformations on the content of the p tags:
The XML transformation process was developed using the Oxygen XML Editor and appropriate processing instructions have been inserted in the XML files. But external validation using Jing or transformation using Saxon can be used.
To continue this work (or debug it...), you might like to look at the following files and directories:
You can use this program to explore and extract entries from the huge XML file, which can then be displayed in an HTML page with this XSLT stylesheet.
To get an idea of the content of the dictionary, you can search for all entries matching a given regular expression on our server, which can be
The regular expression is matched case-insensitively on the whole key, i.e. the content of the first hw tag but keeping only letters, spaces, dashes and apostrophes.
Given the fact that a large XML has to be read, this operation can take some time on our server.
Please, do not use this form to get hundreds of dictionary entries, download the above files instead.
If the first letter of the regular expression is a letter then only the XML file of the words starting with this letter will be read, otherwise the whole XML dictionary file will be traversed.
The retrieved entries from this query are displayed with this XSLT stylesheet. It does not deal with all tags so some parts of the entries might appear verbatim within red tags, but it gives an idea of the content of the entries.
Guy Lapalme, DIRO (Computer Science Department), Université de Montréal.
October 2009 - Work done during a stay at the LIF, Université de Marseille