Creation of dictionary entries from GCIDE_XML files
Context
The GNU version of The Collaborative International Dictionary of English is a well-formed XML version of the CIDE. GCIDE is a freely available set of ASCII files containing the marked-up text of a substantial English dictionary (more than 130 000 headwords).
From the introduction to each file:
This dictionary was derived from the
Webster's Revised Unabridged Dictionary
Version published 1913
by the C. & G. Merriam Co.
Springfield, Mass.
Under the direction of
Noah Porter, D.D., LL.D.
and from
WordNet, a semantic network created by
the Cognitive Science Department
of Princeton University
under the direction of
Prof. George Miller
and is being updated and supplemented by
an open coalition of volunteer collaborators from
around the world.
This electronic dictionary is the starting point for an
ongoing project to develop a modern on-line comprehensive encyclopedic
dictionary, by the efforts of all individuals willing to help build a
large and freely available knowledge base. Contributions of data,
time, and effort are requested from any person willing to assist creation
of a comprehensive and organized knowledge base for free access on the
internet. Anyone willing to assist in any way in constructing such a
knowledge base should contact:
Patrick Cassidy pc@worldsoul.org
735 Belvidere Ave. Office: (908)668-5252
Plainfield, NJ 07062
(908) 561-3416
Last edit January 17, 2002.
In 2002, Patrick Cassidy converted the original SGML-like ASCII files into a well-formed XML document (about 56 MB). The set of tags used in the dictionary is given in the GCIDE's tagset.web file. The files that describe t he structure of the GCIDE are available at http://micra.com/dictionary/. The complete XML document is composed of a main document, which refers to several auxiliary documents, one for each letter of the alphabet. Each one of these auxiliary files contains the information about the words starting with a given letter. The mainXML file is uses System entities that include these auxiliary files to create a global XML document. Only the global XML file is a well-formed XML document.
This XML version is very useful in order to extract some parts of the information but it keeps most of the original tagging that had been developed for printing purposes. In particular, the whole document is seen as a series of paragraphs. To retrieve all the information for a given head word, one must parse from a given point until the next headword. This cumbersome and error-prone process is not very convenient for Natural Language Processing purposes.
We present here a new XML version of the GCIDE with the following characteristics:
- The dictionary is structured as a series of entries gathering, within a single XML tag, all the information about a given headword. That entry keeps the original information but further divides the numerous senses and subsenses within a headword;
- most typographical tags have been removed but semantic and functional tags have been kept;
- the dictionary is validated with a RelaxNG schema that acts as documentation of the structure of the entries;
- each file (one for each letter) is also a valid XML file;
- the whole dictionary is built from each letter file using XInclude instructions;
- most of the original tag names have been kept, even though some renaming might have been appropriate;
- CAVEAT: in this process all references to character entities have been replaced by the content.
Getting the dictionary
For the impatient interested in the final product, i.e. the XML files of the dictionary:
- download this Zip file (27 MB), 114 MB once unzipped!
- new-entries/gcide.xml is the root file XIncluding all others
- new-entries/gcide-entries.xml (64 MB) is the whole dictionary in a single file.
- The structure of the dictionary can be seen in the Relax NG schema with its automatically generated included schema
- A sample XSLT 2.0 stylesheet is provided to display the entries in an HTML page
Conversion process
The conversion has been done using a bash script that starts from a directory containing the old GCIDE_XML files, modified only slightly to correct obvious tagging errors that were inducing validation errors, and creates a new directory with a new structure. As the original file is a valid XML file, we used XSLT 2.0 stylesheets to create the dictionary entries.
The script is given here. Here are the main steps
- Create a file containing only XML character entities;
- Create a separate file for each letter and sections of the dictionary;
- Create the main file including the files for each letter and sections;
- Create a single file combining all the files in case the XML processor does not allow XInclude processing.
- Create a new directory that contains only the file with entries and zip it
Originally, we intended to create a simple structure from the old one, but given the size of the original file and different types of tagging that had been used over the years, we finally decided to keep most of the original tags and we more or less transposed into a schema the original DTD (that itself had been created automatically from the files). We leave it to other volunteers to redesign the entries along a simpler schema.
We created an Excel file starting from the original DTD to define the mapping between the original tags and attributes and the ones used in the new. As an XSLT stylesheet cannot deal easily with native excel formats, we export a CSV version.
We developed a conversion stylesheet that takes an input the tag mapping and produces a part of the RNC schema and a part of the XSLT stylesheet for the conversion process. These files are then included into the full Relax NG (compact form) schema and the conversion stylesheet.
The conversion stylesheet is a relatively ad-hoc process that does the following transformations on the content of the p tags:
- remove the source tag and its immediate surroundings (<br/> and brackets) and embed this meta-information in the first tag of the content of the p tag;
- insert the text and tags between “ and ” within <q> tags;
- when an <au> tag follows an element, its content is inserted as the au attribute of the element;
- group all paragraphs under the same headword;
- for each headword, divide in sense numbers and then further into subsenses;
- remove text nodes containing only punctuations or dashes and strip initial and final parentheses, brackets and braces.
The XML transformation process was developed using the Oxygen XML Editor and appropriate processing instructions have been inserted in the XML files. But external validation using Jing or transformation using Saxon can be used.
Access to the files
To continue this work (or debug it...), you might like to look at the following files and directories:
- Excel file giving the mapping between the original tags and the ones used in this version. This file contains the following columns:
- type of tags according to the classification given in the GCIDE's tagset.web file;
- a very short definition of the meaning of the tag. When it is blank, we considered that the name of the tag was self-explanatory;
- tag in the old gcide;
- attributes of the original tag
- content of the tag using the DTD notation; the content of this column was first taken from the original DTD but then adjusted in order to be able to validate the XML file
- new tag with the following convention
- =: the tag is kept the same in the new
- #: the tag is removed in the new
- *: the tab is defined explicitly in gcide.rnc
- name: the tag is changed for this new name (this is was done to limit the number of different tags that seemed synonymous)
- attributes of the new tag
If you change this file, then save it also in csv format and run the stylesheet to regenerate the corresponding part of RelaxNG schema and the conversion stylesheet.
- the conversion script in Bash;
- the XSLT 2.0 stylesheet to create entries which can be executed using Saxon;
- the Relax NG validation with the two schemas (gcide.rnc and gcide-conv.rnc) done using Jing which uses: jing.jar, isorelax.jar, saxon.jar, xercesImpl.jar and xml-apis.jar;
- The whole project directory is available as a single zip file.
Exploring the dictionary
You can use this program to explore and extract entries from the huge XML file, which can then be displayed in an HTML page with this XSLT stylesheet.
Headword Search
To get an idea of the content of the dictionary, you can search for all entries matching a given regular expression on our server, which can be
- a single word such as give or jack
- an expression, using the XSL regular expression syntax, like tr.*d which returns the entries starting with tr and ending with d (there are currently 89 of them)
The regular expression is matched case-insensitively on the whole key, i.e. the content of the first hw tag but keeping only letters, spaces, dashes and apostrophes.
Given the fact that a large XML has to be read, this operation can take some time on our server.
Please, do not use this form to get hundreds of dictionary entries, download the above files instead.
If the first letter of the regular expression is a letter then only the XML file of the words starting with this letter will be read, otherwise the whole XML dictionary file will be traversed.
The retrieved entries from this query are displayed with this XSLT stylesheet. It does not deal with all tags so some parts of the entries might appear verbatim within red tags, but it gives an idea of the content of the entries.
More info
If you have any problem (or fun!) using these files, contact
Guy Lapalme, DIRO (Computer Science Department), Université de Montréal.
October 2009 - Work done during a stay at the LIF, Université de Marseille
