jsRealB Tutorial

Overview

jsRealB is a text realizer for French and English written in Javascript to ease its integration in web applications. It can be used either within a web page such as this one in which examples in this font following an arrow (→) are dynamically realized or as a Node.js module. The realizer is self-contained and does not need an external server or another Javascript framework, once it is loaded with the web page.

The realizer takes care of many language peculiarities such as conjugation, word order, elision, number and person agreements, all small details but that users appreciate or find very annoying or disturbing when they are not properly dealt with. It is also possible to get many variations (e.g. negative, passive or interrogative) of a single sentence structure.

In jsRealB, Javascript expressions create data structures corresponding to the constituents of the sentence to be produced. When there is a need to get the string output (i.e. its toString() function is called), the data structure (a tree) traversed to produce the tokens of the sentence, taking care of capitalization, elision and appropriate spacing around punctuation. It is also possible to wrap portions of text within HTML tags.

The data structure is built by function calls (Javascript constructors) whose names were chosen to be similar to the symbols usually used for constituent syntax trees: for example, N to create a noun structure, NP for a Noun Phrase, V for a Verb, D for a Determiner, S for a Sentence and so on. Features added to the structures using the dot notation can modify the values according to what is intended.

The syntactic representation is patterned after the classical constituent grammar notation. For example,

        S(NP(D("a"),N("woman")).n("p"),
          VP(V("eat")).t("ps")).typ({perf:true})

is the JSrealB specification for Plural is indicated with feature n("p") where n indicates number and p plural. The verb is conjugated to past tense indicated by the feature tense t and value ps; the sentence should also use the perfect form. The agreement between NP and VP is performed automatically.

This document first gives some general information about the way to create a sentence with jsRealB. It then illustrates some of the capabilities of jsRealB with two examples: a random sentence generator and a more complete data-to-text web application that finds a metro route and generates a text (either in French or in English) describing the itinerary which is also displayed in the page.

Creating a simple sentence

We start with a very simple sentence combining a Noun Phrase as subject of a Verb Phrase with the following steps:

Constituents

jsRealB expressions are built using Constituents which can be either Terminals or Phrases shown in the following table. Terminals take a single string as parameter and Phrases can take an indefinite number of parameters, either Terminals or Phrases.

Terminal Phrase
DeterminerD("..")
PronounPro("..")
NounN("..")NP(...)
AdjectiveA(("..")AP(...)
VerbV("..")VP(...)
AdverbAdv("..")AdvP(...)
PrepositionP("..")PP(...)
ConjunctionC("..")CP(...)
SubordinationSP(...)
SentenceS(...)
DateDT("..")
NumberNO("..")
Quoted textQ("..")

The first parameter in a Phrase is usually its head. For example in a Sentence, the first parameter is taken as the subject. The first parameter of a Verb Phrase is its Verb, followed by a direct object unless it is a Prepositional Phrase. These implicit roles are used for setting up the agreement links between constituents. For example, the number of the subject is used as the number of the verb. If these links are not properly set, it is possible to set features explicitly, but most often the default rules are sufficient.

Variations on a simple sentence

From a single sentence structure, it is possible to get many variations (in blue in this table) by simply adding features (written as methods called using the dotted notation) such as the ones in the following table in which the previous sentence is at the future tense:

future S(NP(D("the"),N("cat")).n("p"),
  VP(V("eat").t("f"),
     NP(D("a"),A("grey"),N("mouse"))))
passive S(NP(D("the"),N("cat")),
  VP(V("eat").t("f"),
     NP(D("a"),A("grey"),N("mouse"))))
 .typ({"pas":true})
question about the subject S(NP(D("the"),N("cat")),
  VP(V("eat").t("f"),
     NP(D("a"),A("grey"),N("mouse"))))
 .typ({"int":"why"})
negative S(NP(D("the"),N("cat")),
  VP(V("eat").t("f"),
     NP(D("a"),A("grey"),N("mouse"))))
 .typ({"neg":true})
combination of the above S(NP(D("the"),N("cat")),
  VP(V("eat").t("f"),
     NP(D("a"),A("grey"),N("mouse"))))
 .typ({"int":"why","pas":true,"neg":true})

Formatting

Because language generation is not limited to producing list of words, jsRealB also provides facilities for adding punctuation, wrapping constituents in symbols and HTML formatting. Formatting is indicated with features added to constituents. To add a string before a constituent, use .b("..") after building the constituent, for example N("man").b("*"), similarly .a("..") to add a string after. If the string is an usual punctuation, appropriate spacing before and after the sign is added. Pairs of symbols can be added with .en(".."), for example N("mouse").en("[").

It is also possible to add HTML tag arount any constituent with the .tag(name,attributes). The tag name is a string and the optional attributes are given by an object whose keys are the attribute names with the corresponding value. For exemple, N("mouse").tag("i") displayed as in an HTML page. To create hyperlinks, we can use the a tag, for example
N("mouse").tag("a",{href:"https://en.wikipedia.org/wiki/Mouse"})
rendered as in a web page.

Saving and reusing jsRealB expressions

jsRealB constructions can be saved in variables or data structures like any other Javascript value. So our first sentence () could have been coded like this.

var cats=NP(D("the"),N("cat")).n("p");
var mouse=NP(D("a"),A("grey"),N("mouse"));
S(cats,VP(V("eat"),mouse));

This is very useful for structuring a complex text built incrementally. This also makes possible to share repeating parts in a text.

The constructors corresponding to constituents create new objects, but features (i.e. methods called with the dot) modify them permanently. So some care must be taken when reusing structures saved in variables.

For example, with those declarations:

    var cat1=NP(D("the"),N("cat"));
    var mouse1=NP(D("a"),A("grey"),N("mouse"));

  S(cat1.n("p"),VP(V("eat"),mouse1.n("p").pro()))
in which cat1 is plural and mouse1 replaced by a pronoun.

If we reuse those variables in another context such as
  S(cat1,VP(V("love"),mouse1))
in which mouse1 is again pronominalized which is probably not intended.

This happened because the features modified the original structure. In this case, the modification was applied directly on the value of the variable, but as some properties (e.g. gender, noun, negation, etc.) are propagated between constituents for achieving the appropriate agreements, modifications to shared variables can be more subtle and hard to identify because modifications can happen from a governing or from a sibling constituent. To get around this problem, we can clone any jsRealB structure to get a new copy of a value.

So going back to our original declarations

    var cat2=NP(D("the"),N("cat"));
    var mouse2=NP(D("a"),A("grey"),N("mouse"));
  S(cat2.clone().n("p"),VP(V("eat"),mouse2.clone().n("p").pro()))
without modifying the original as can be seen with
  S(cat2,VP(V("eat"),mouse2))

Generation of random sentences

Computer generated texts often suffer from repetitions which can be quite annoying for humans who prefer some variation in the texts they read. jsRealB makes it easy to create different phrase and sentence structures by creating different constituent tree expressions from which the program can choose.

In many cases, outside constraints, semantic or pragmatic constraints determine the choice, but in some cases it is appropriate to let a random choice between few possibilities. The oneOf(e1,e2 ...) function lets jsRealB choose one of the arguments ei. As Javascript evaluates all arguments of a function before the call, we can prefix an argument with ()=> to transform it into a a zero argument function which will only be evaluated after the selection by oneOf.

The following code defines three functions:

function np(){
    return NP(D("a"),a(),N(oneOf("cat","mouse","dog","rabbit"))).n(oneOf("s","p"));
}
function a(){
    return oneOf(
        ()=>A(oneOf("hungry","grey","nervous")),
        ()=>Q("")
    );
}
function vp(){
    return oneOf(
        ()=>VP(V(oneOf("eat","run","love")).t(oneOf("p","ps","f"))),
        ()=>VP(V(oneOf("eat","love")).t(oneOf("p","ps","f")),
               np())
    );
}

With these definitions, S(np(),vp()) will produce a random sentence (containing a noun phrase followed by verb phrase) at each call.

Click this button to

This random sentence example is a Javascript transcription of the Prolog/DCG version described in: Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 256-257, Ellis Horwood, 1990.

Generation of subway/metro directions

We now present a complete web application for finding a path in the Montréal Métro (subway) network. We will then explain how to generate the itinerary between two stations clicked by user that is generated to the right of the graphic. The application can be used in either French or English with the text generated in the same language, by clicking on the [EN] or [FR] in the top right corner. In this tutorial, only the English version is described, as the French version is similarly organized and differs mainly in the choice of words.

Generation of the itinerary

Text generation is classically divided in two main steps:

Data

The first step in a data-to-text project is to find the appropriate data and to massage it into a usable format. In this case, the data was harvested from a public dataset, although in a proprietary format for which we fortunately found a web site that allowed us to transform it to geojson. We then used a jq script to extract only the information we needed: metro lines with their stations for which we kept their name and coordinates. The jq output was then feed to a small Python script to get a more useful display. We finally hand sorted the station according to their real ordering on each line. As always, this conceptually simple task took more effort than we had initially expected.

Here is a short excerpt of the data for the 68 stations of the metro. Some transfer stations (e.g. Berri-UQAM) appear in more than one line as indicated in their route field.

var routes = [
  {"route":1, "stations":[
    {"id": "18", "station": "Honoré-Beaugrand", "route": "1", "coords": [302040.08698887925, 5050741.44628119]},
    {"id": "19", "station": "Radisson", "route": "1", "coords": [301735.9800496101, 5049947.9922192525]},
    ...
    {"id": "43", "station": "Angrignon", "route": "1", "coords": [296733.6694496933, 5034064.601947192]}
  ]},
  {"route":2, "stations":[
    {"id": "68", "station": "Montmorency", "route": "2", "coords": [287432.767743489, 5046526.781216757]},
    ...
    {"id": "65", "station": "Côte-Vertu", "route": "2", "coords": [290521.09550694964, 5041607.070800175]}
  ]},
  {"route":4, "stations":[
    {"id": "11", "station": "Berri-UQAM", "route": "1,2,4", "coords": [300055.51811076206, 5041690.301771889]},
    ...
    {"id": "44", "station": "Longueuil/Université-de-Sherbrooke", "route": "4", "coords": [303073.79689146805, 5042774.93822535]}
  ]},
  {"route":5, "stations":[
    {"id": "49", "station": "Snowdon", "route": "2,5", "coords": [294829.623185072, 5038412.072364335]},
    ...
    {"id": "64", "station": "Saint-Michel", "route": "5", "coords": [296993.06301460037, 5046632.485787434]}
  ]}
]

Path computation

With this data we build an internal data structure of the underlying network having stations as node and estimated travel time (computed from the distance from the coordinates of the stations) as value on the arcs between two consecutive stations. Transfer stations are duplicated on each line and an internal link with transfer time between each line is added.

In the application, once the user has chosen the two stations between which she wants to travel, the shortest path between them is computed using the Bellman-Ford algorithm which is very simple to implement and efficient enough on such a small network.

The shortest path is a list of nodes that is split into legs between transfer stations. For example, the path to go from Honoré-Beaugrand (id="18") to Université de Montréal (id="57") will be split in three legs corresponding to travels on a line between transfer:

[[["18",0],["19",1.41],...,["11-1",15.91]],
 [["11-2",20.91],["10",22.23],...,["5-2",29.14]],
 [["5-5",34.14],["61",35.17],...,["57",40.75]]]

The trip is thus a list of legs which are themselves lists of pairs also lists, the first element being the node id and the second element being the distance in minutes from the starting node.

Path computation code

Text generation

The generated text is organized as:

This is implemented in this function, which first creates a title wrapped in an h2 tag. That title is added in front of the rest of the text either as a direct trip singleLine or one separated into multiple legs (introduction, body, conclusion).

function generate(trip){
    var duration=last(last(trip))[1];
    var title=VP(V("go").t("b"),P("from"),Q(network[trip[0][0][0]].stationName),
                 P("to"),Q(network[last(last(trip))[0]].stationName)).cap().tag("h2")+"\n";
    if (trip.length==1){
        return title+singleLine(trip[0],duration);
    } else {
        let text=title;
        text+=introduction(trip[0],duration).tag("p")+" ";
        text+="<p>"+body(trip.slice(1,trip.length-1))+"</p>";
        text+=conclusion(last(trip)).tag("p");
        return text;
    }
}

We define a few auxiliary functions to realize:

.
function genLine(leg){
    var from=network[leg[0][0]];
    var to=network[last(leg)[0]];
    var res=NP(D("the"),A(lineColor[from.route]),N("line"));
    if (from.index<to.index){
        if (to!=from.end)
            // do not add direction when the destination is at the last station which will be realized anyway
            res.add(PP(P("towards"),Q(from.end.stationName)));
    } else {
        if (to!=from.start)
            // do not add direction when the destination is at the first station which will be realized anyway
            res.add(NP(N("direction"),Q(from.start.stationName)).en('('));
    }
    return res;
}

function nbStations(leg,ord){
    if (ord===undefined)ord=false;
    var st=N(oneOf("station","stop"));
    if (leg.length==2){
        return NP(Adv("only"),NO(1).dOpt({"nat": true}),st);
    }
    return NP(NO(leg.length-1).dOpt({"ord":ord}),st)
}

function destination(leg){
    return Q(network[last(leg)[0]].stationName);
}
A direct trip

After testing the boundary case that the trip starts and ends at the same station, the realizer creates two subordinate phrases by choosing randomly between a few formulations. These two subordinate phrases are then wrapped into a single sentence.

function singleLine(leg,duration){
    if (duration==0)
        return S(Pro("I").pe(2),V("be"),Adv("already"),PP(P("at"),D("my").pe(2),N("destination"))).a("!");
    var sp1 = oneOf(()=>SP(D("this"),V("be"),CP(C("and"),A("simple"),A("fast")),nbStations(leg),
                           Adv("no"),N("transfer")),
                    ()=>SP(Pro("I").pe(2),
                           VP(V("be"),Adv("only"),nbStations(leg),Adv("away"))),
                    ()=>SP(Pro("this"),VP(V("make").t("f"),nbStations(leg),
                                          PP(P("for"),NP(D("the"),A("whole"),N("trip")))))
                    ).a(",");
    var sp2 = oneOf(()=>SP(V("take").t("ip"),genLine(leg)),
                    ()=>SP(V("follow").t("ip"),genLine(leg),
                           PP(P("for"),NP(NO(Math.round(duration)),N("minute")))));
    return S(sp1,sp2).toString();
}
Trip with transfers

The introduction starts with global information: the estimated time to destination and then, with different formulations, invite the user to start the trip with the first leg.

function introduction(leg,duration){
    var sp1 = S(Pro("I").pe(2),VP(V("be"),NP(NO(Math.round(duration)),N("minute"))),
                 PP(P("from"),NP(D("my").pe(2),N("destination")))).a(",");
    var sp2 = oneOf(
                ()=>SP(V("take").t("ip"),genLine(leg),
                       PP(P("for"),nbStations(leg))),
                ()=>SP(V("board").t("ip"),PP(P("on"),genLine(leg)))
              );
    var pp = oneOf(()=>PP(P("until"),SP(Pro("I").pe(2),
                                        VP(V("be"),PP(P("at"),destination(leg))))),
                   ()=>PP(P("up"),P("to"),destination(leg))
             );
    return S(sp1,sp2.add(pp));
}

The body recursively realizes each leg with different formulations

function body(legs){
    if (legs.length==0)return "";
    // console.log(legs);
    var leg=legs[0]
    var out=oneOf(
        ()=>S(VP(V("take").t("ip"),genLine(leg),PP(P("up"),P("to"),destination(leg)))),
        ()=>S(VP(V("change").t("ip"),PP(P("on"),genLine(leg),P("until"),destination(leg)))),
        ()=>S(VP(V("transfer").t("ip"),PP(P("for"),destination(leg),P("on"),genLine(leg)))),
        ()=>S(VP(V("switch")).t("ip"),
               PP(P("on"),genLine(leg),P("for"),nbStations(leg)))
    );
    return out+" "+body(legs.slice(1));
}

The conclusion ends by indicating that the trip is finished.

function conclusion(leg){
    var out=oneOf(
        ()=>S(Adv("finally"),Pro("I").pe(2),V("take").t("f"),genLine(leg),
               V("arrive").t("b"),PP(P("to"),destination(leg))),
        ()=>S(Q("at last").a(","),Pro("I").pe(2),
              VP(V("find").t("f"),destination(leg).a(","),
                 PP(nbStations(leg,true),P("on"),genLine(leg)))),
        ()=>S(NP(D("my").pe(2),N("destination"),destination(leg)),
              VP(V("be").t("f"),P("after"),nbStations(leg),
                             P("on"),genLine(leg)))
    );
    return out;
}

This Javascript realizer using jsRealB is an adaptation of the Prolog/DCG version shown in Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 258-260, Ellis Horwood, 1990.

Conclusion

This document has illustrated how jsRealB can be integrated in a web page to create dynamic textual content. More information can be found at the jsRealB web site. There is also an interactive development environment in which jsRealB code can be tested with the corresponding constituent structure displayed. It is also possible to check the lexicon and the rule tables of the language and find how to realize a given word in jsRealB.