jsRealB Tutorial

Overview

jsRealB is a text realizer for French and English written in Javascript to ease its integration in web applications. It can be used either within a web page such as this one whose examples in this font are dynamically realized or as a Node.js module. The realizer is self-contained and does not need an external server or another Javascript framework, once it is loaded with the web page.

The realizer takes care of many language peculiarities such as conjugation, word order, elision, number and person agreements, all small details but that users appreciate or find very annoying or disturbing when they are not properly dealt with. It is also possible to get many variations (e.g. negative, passive or interrogative) of a single sentence structure.

In jsRealB, Javascript expressions create data structures corresponding to the constituents of the sentence to be produced. When there is a need to get the string output (i.e. its toString() function is called), the data structure (a tree) traversed to produce the tokens of the sentence, taking care of capitalization, elision and appropriate spacing around punctuation. It is also possible to wrap portions of text in HTML tags.

The data structure is built by function calls (Javascript constructors) whose names were chosen to be similar to the symbols usually used for constituent syntax trees: for example, N to create a noun structure, NP for a Noun Phrase, V for a Verb, D for a Determiner, S for a Sentence and so on. Features added to the structures using the dot notation can modify the values according to what is intended.

The syntactic representation is patterned after the classical constituent grammar notation. For example,

        S(NP(D("a"),N("woman")).n("p"),
          VP(V("eat")).t("ps").perf(true))

is the JSrealB specification for S(NP(D("a"),N("woman")).n("p"),VP(V("eat")).t("ps").perf(true)) Plural is indicated with feature n("p") where n indicates number and p plural. The verb is conjugated to past tense indicated by the feature tense t and value ps; the verb should also use the perfect form. The agreement between NP and VP is performed automatically.

This document illustrates some of the capabilities of jsRealB with two examples: a simple sentence generator and a more complete data-to-text web application that finds a metro route and generates a text describing the itinerary which is also displayed in the page.

Generating sentences

Variations on a simple sentence

We start with a very simple sentence combining a Noun Phrase that is the subject of a Verb Phrase. jsRealB expressions being ordinary Javascript expressions they can saved in variables or data structure like any other Javascript value. In the following

var cat=NP(D("the"),N("cat"));
var mouse=NP(D("a"),A("grey"),N("mouse"));
var sent=S(cat,VP(V("eat"),mouse));

We define two noun phrases: cat for cat and mouse for mouse. The whole sentence sent is created by creating a S constituent having the value of variable cat as subject (the first child in a S is considered as such) and a Verb Phrase containing the verb eat and the value of variable mouse as object complement.

From this single sentence structure, it is possible to get many variations (in blue in this table) by simply adding features:

plural future S(cat,VP(V("eat"),mouse))
 .n("p").t("f")
S(cat,VP(V("eat"),mouse)).n("p").t("f")
passive S(cat,VP(V("eat"),mouse))
 .typ({"pas":true})
S(cat,VP(V("eat"),mouse)).typ({"pas":true})
question on the subject (what direct) S(cat,VP(V("eat"),mouse))
 .typ({"int":"wad"})
S(cat,VP(V("eat"),mouse)).typ({"int":"wad"})
negative S(cat,VP(V("eat"),mouse))
 .typ({"neg":true})
S(cat,VP(V("eat"),mouse)).typ({"neg":true})
combination of the above S(cat,VP(V("eat"),mouse))
 .typ({"int":"wad","pas":true,"neg":true})
S(cat,VP(V("eat"),mouse)).typ({"int":"wad","pas":true,"neg":true})

Caution with modifications

The constructors corresponding to constituents create objects and features modify them permanently (this is probably a dubious implementation choice that should be revised, but for the moment we have to live with it!). So some care must be taken when reusing structures saved in variables. For example, with those declarations:

    var cat1=NP(D("the"),N("cat"));
    var mouse1=NP(D("a"),A("grey"),N("mouse"));

S(cat1.n("p"),VP(V("eat"),mouse1.n("p").pro())) is realized as S(cat1.n("p"),VP(V("eat"),mouse1.n("p").pro())) in which cat is plural and mouse replaced by a pronoun. If we reuse those variables in another context such as S(cat1,VP(V("eat"),mouse1)), the realized sentence is still S(cat1,VP(V("eat"),mouse1)) which is probably not intended, because the features modified the original structure. In this case, the modification was applied directly on the variable, but as some properties (e.g. gender, noun, negation, etc.) are propagated between constituents for achieving the appropriate agreements, modifications to shared variables can be more subtle and hard to identify because modifications can happen from a governing or from a sibling constituent.

To get around this problem, we can clone any jsRealB structure to get a new copy within an expression. So getting back to our original cat and mouse, we can still get S(cat.clone().n("p"),VP(V("eat"),mouse.clone().n("p").pro())) with S(cat.clone().n("p"),VP(V("eat"),mouse.clone().n("p").pro())) without modifying the original, as can be seen with S(cat,VP(V("eat"),mouse)) which is still realized as S(cat,VP(V("eat"),mouse))

Generation of random sentences

Computer generated texts often suffer from repetitions which can be quite annoying for humans who prefer some variation in the texts they read. jsRealB makes it easy to create different phrase and sentence structures by creating different constituent tree expressions from which the program can choose.

In many cases, outside constraints, semantic or pragmatic constraints determine the choice, but in some cases it is appropriate to let a random choice between few possibilities. The oneOf(e1,e2 ...) function lets jsRealB choose one of the arguments ei. As Javascript evaluates all arguments of a function before the call, we can prefix an argument with ()=> to transform it into a a zero argument function which will only be evaluated after the selection by oneOf.

The following code defines three functions:

function np(){
    return NP(D("a"),a(),N(oneOf("cat","mouse","dog","rabbit"))).n(oneOf("s","p"));
}
function a(){
    return oneOf(
        ()=>A(oneOf("fussy","grey","nervous")),
        ()=>Q("")
    );
}
function vp(){
    return oneOf(
        ()=>VP(V(oneOf("eat","run","love"))),
        ()=>VP(V(oneOf("eat","love")),np())
    );
}

With these definitions, S(np(),vp()) will produce a random sentence (containing a noun phrase followed by verb phrase) at each call.

Click this button to

This random sentence example is a Javascript transcription of the Prolog/DCG version described in: Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 256-257, Ellis Horwood, 1990.

Generation of subway/metro directions

We now present a complete web application for finding a path in the Montréal Métro (subway) network. We will then explain how to generate the itinerary between two stations clicked by user that is generated to the right of the graphic. The application can be used in either French or English with the text generated in the same language, by clicking on the [EN] or [FR] in the top right corner. In this tutorial, only the English version is described, as the French version is similarly organized and differs mainly in the choice of words.

Generation of the itinerary

Text generation is classically divided in two main steps:

Data

The first step in a data-to-text project is to find the appropriate data and to massage it into a usable format. In this case, the data was harvested from a public dataset, although in a proprietary format for which we fortunately found a web site that allowed us to transform it to geojson. We then used a jq script to extract only the information we needed: metro lines with their stations for which we kept their name and coordinates. The jq output was then feed to a small Python script to get a more useful display. We finally hand sorted the station according to their real ordering on each line. As always, this conceptually simple task took more effort than we had initially expected.

Here is a short excerpt of the data for the 68 stations of the metro. Some transfer stations (e.g. Berri-UQAM) appear in more than one line as indicated in their route field.

var routes = [
  {"route":1, "stations":[
    {"id": "18", "station": "Honoré-Beaugrand", "route": "1", "coords": [302040.08698887925, 5050741.44628119]},
    {"id": "19", "station": "Radisson", "route": "1", "coords": [301735.9800496101, 5049947.9922192525]},
    ...
    {"id": "43", "station": "Angrignon", "route": "1", "coords": [296733.6694496933, 5034064.601947192]}
  ]},
  {"route":2, "stations":[
    {"id": "68", "station": "Montmorency", "route": "2", "coords": [287432.767743489, 5046526.781216757]},
    ...
    {"id": "65", "station": "Côte-Vertu", "route": "2", "coords": [290521.09550694964, 5041607.070800175]}
  ]},
  {"route":4, "stations":[
    {"id": "11", "station": "Berri-UQAM", "route": "1,2,4", "coords": [300055.51811076206, 5041690.301771889]},
    ...
    {"id": "44", "station": "Longueuil/Université-de-Sherbrooke", "route": "4", "coords": [303073.79689146805, 5042774.93822535]}
  ]},
  {"route":5, "stations":[
    {"id": "49", "station": "Snowdon", "route": "2,5", "coords": [294829.623185072, 5038412.072364335]},
    ...
    {"id": "64", "station": "Saint-Michel", "route": "5", "coords": [296993.06301460037, 5046632.485787434]}
  ]}
]

Path computation

With this data we build an internal data structure of the underlying network having stations as node and estimated travel time (computed from the distance from the coordinates of the stations) as value on the arcs between two consecutive stations. Transfer stations are duplicated on each line and an internal link with transfer time between each line is added.

In the application, once the user has chosen the two stations between which she wants to travel, the shortest path between them is computed using the Bellman-Ford algorithm which is very simple to implement and efficient enough on such a small network.

The shortest path is a list of nodes that is split into legs between transfer stations. For example, the path to go from Honoré-Beaugrand (id="18") to Université de Montréal (id="57") will be split in three legs corresponding to travels on a line between transfer:

[[["18",0],["19",1.41],...,["11-1",15.91]],
 [["11-2",20.91],["10",22.23],...,["5-2",29.14]],
 [["5-5",34.14],["61",35.17],...,["57",40.75]]]

The trip is thus a list of legs which are themselves lists of pairs also lists, the first element being the node id and the second element being the distance in minutes from the starting node.

Path computation code

Text generation

The generated text is organized as:

This is implemented in this function, which first creates a title wrapped in an h2 tag. That title is added in front of the rest of the text either as a direct trip singleLine or one separated into multiple legs (introduction, body, conclusion).

function generate(trip){
    var duration=last(last(trip))[1];
    var title=VP(V("go").t("b"),P("from"),Q(network[trip[0][0][0]].stationName),
                 P("to"),Q(network[last(last(trip))[0]].stationName)).cap().tag("h2")+"\n";
    if (trip.length==1){
        return title+singleLine(trip[0],duration);
    } else {
        let text=title;
        text+=introduction(trip[0],duration).tag("p")+" ";
        text+="<p>"+body(trip.slice(1,trip.length-1))+"</p>";
        text+=conclusion(last(trip)).tag("p");
        return text;
    }
}

We define a few auxiliary functions to realize:

.
function genLine(leg){
    var from=network[leg[0][0]];
    var to=network[last(leg)[0]];
    var res=NP(D("the"),A(lineColor[from.route]),N("line"));
    if (from.index<to.index){
        if (to!=from.end)
            // do not add direction when the destination is at the last station which will be realized anyway
            res.add(PP(P("towards"),Q(from.end.stationName)));
    } else {
        if (to!=from.start)
            // do not add direction when the destination is at the first station which will be realized anyway
            res.add(NP(N("direction"),Q(from.start.stationName)).en('('));
    }
    return res;
}

function nbStations(leg,ord){
    if (ord===undefined)ord=false;
    var st=N(oneOf("station","stop"));
    if (leg.length==2){
        return NP(Adv("only"),NO(1).dOpt({"nat": true}),st);
    }
    return NP(NO(leg.length-1).dOpt({"ord":ord}),st)
}

function destination(leg){
    return Q(network[last(leg)[0]].stationName);
}
A direct trip

After testing the boundary case that the trip starts and ends at the same station, the realizer creates two subordinate phrases by choosing randomly between a few formulations. These two subordinate phrases are then wrapped into a single sentence.

function singleLine(leg,duration){
    if (duration==0)
        return S(Pro("I").pe(2),V("be"),Adv("already"),PP(P("at"),D("my").pe(2),N("destination"))).a("!");
    var sp1 = oneOf(()=>SP(D("this"),V("be"),CP(C("and"),A("simple"),A("fast")),nbStations(leg),
                           Adv("no"),N("transfer")),
                    ()=>SP(Pro("I").pe(2),
                           VP(V("be"),Adv("only"),nbStations(leg),Adv("away"))),
                    ()=>SP(Pro("this"),VP(V("make").t("f"),nbStations(leg),
                                          PP(P("for"),NP(D("the"),A("whole"),N("trip")))))
                    ).a(",");
    var sp2 = oneOf(()=>SP(V("take").t("ip"),genLine(leg)),
                    ()=>SP(V("follow").t("ip"),genLine(leg),
                           PP(P("for"),NP(NO(Math.round(duration)),N("minute")))));
    return S(sp1,sp2).toString();
}
Trip with transfers

The introduction starts with global information: the estimated time to destination and then, with different formulations, invite the user to start the trip with the first leg.

function introduction(leg,duration){
    var sp1 = S(Pro("I").pe(2),VP(V("be"),NP(NO(Math.round(duration)),N("minute"))),
                 PP(P("from"),NP(D("my").pe(2),N("destination")))).a(",");
    var sp2 = oneOf(
                ()=>SP(V("take").t("ip"),genLine(leg),
                       PP(P("for"),nbStations(leg))),
                ()=>SP(V("board").t("ip"),PP(P("on"),genLine(leg)))
              );
    var pp = oneOf(()=>PP(P("until"),SP(Pro("I").pe(2),
                                        VP(V("be"),PP(P("at"),destination(leg))))),
                   ()=>PP(P("up"),P("to"),destination(leg))
             );
    return S(sp1,sp2.add(pp));
}

The body recursively realizes each leg with different formulations

function body(legs){
    if (legs.length==0)return "";
    // console.log(legs);
    var leg=legs[0]
    var out=oneOf(
        ()=>S(VP(V("take").t("ip"),genLine(leg),PP(P("up"),P("to"),destination(leg)))),
        ()=>S(VP(V("change").t("ip"),PP(P("on"),genLine(leg),P("until"),destination(leg)))),
        ()=>S(VP(V("transfer").t("ip"),PP(P("for"),destination(leg),P("on"),genLine(leg)))),
        ()=>S(VP(V("switch")).t("ip"),
               PP(P("on"),genLine(leg),P("for"),nbStations(leg)))
    );
    return out+" "+body(legs.slice(1));
}

The conclusion ends by indicating that the trip is finished.

function conclusion(leg){
    var out=oneOf(
        ()=>S(Adv("finally"),Pro("I").pe(2),V("take").t("f"),genLine(leg),
               V("arrive").t("b"),PP(P("to"),destination(leg))),
        ()=>S(Q("at last").a(","),Pro("I").pe(2),
              VP(V("find").t("f"),destination(leg).a(","),
                 PP(nbStations(leg,true),P("on"),genLine(leg)))),
        ()=>S(NP(D("my").pe(2),N("destination"),destination(leg)),
              VP(V("be").t("f"),P("after"),nbStations(leg),
                             P("on"),genLine(leg)))
    );
    return out;
}

This Javascript realizer using jsRealB is an adaptation of the Prolog/DCG version shown in Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 258-260, Ellis Horwood, 1990.

Conclusion

This document has illustrated how jsRealB can be integrated in a web page to create dynamic textual content. More information can be found at the jsRealB web site. There is also an interactive development environment in which jsRealB code can be tested with the corresponding constituent structure displayed. It is also possible to check the lexicon and the rule tables of the language and find how to realize a given word in jsRealB.