jsRealB is a text realizer for French and English written in JavaScript to ease its integration in web applications. It can be used either within a web page such as this one in which examples in this font following an arrow (→) are dynamically realized or as a Node.js module. The realizer is self-contained and does not need an external server or another JavaScript framework, once it is loaded with the web page.
The realizer takes care of many language peculiarities such as conjugation, word order, elision, number and person agreements, all small details but that users appreciate or find very annoying or disturbing when they are not properly dealt with. It is also possible to get many variations (e.g. negative, passive or interrogative) of a single sentence structure.
In jsRealB, JavaScript expressions create data structures corresponding to the constituents of the sentence to be produced. When there is a need to get the string output (i.e. its toString() function is called), the data structure (a tree) traversed to produce the tokens of the sentence, taking care of capitalization, elision and appropriate spacing around punctuation. It is also possible to wrap portions of text within HTML tags.
The data structure is built by function calls (JavaScript constructors) whose names were chosen to be similar to the symbols usually used for constituent syntax trees: for example, N to create a noun structure, NP for a Noun Phrase, V for a Verb, D for a Determiner, S for a Sentence and so on. Features added to the structures using the dot notation can modify the values according to what is intended.
The syntactic representation is patterned after the classical constituent grammar notation. For example,
S(NP(D("a"),N("woman").n("p")),
VP(V("eat").t("ps"))).typ({perf:true})
is the JSrealB specification for Plural is indicated with feature n("p") where n indicates number and p plural. The verb is conjugated to past tense indicated by the feature tense t and value ps; the sentence should also use the perfect form. The agreement between the determiner and the noun within the NP and the verb of the VP is performed automatically.
This document first gives some general information on how to create a sentence with jsRealB which also allows to generate variations such as its negative, passive or interrogative form.
It then illustrates some of the capabilities of jsRealB with two examples:
This tutorial only gives a few indications on the many capabilities and features of the realizer, see this document for a complete documentation;
Realize a jsRealB expression is a good way to experiment with jsRealB.
We start with a very simple sentence combining a Noun Phrase as subject of a Verb Phrase with the following steps:
NP(D("the"),N("cat")) → NP(D("a"),A("grey"),N("mouse")) → VP(V("eat"),NP(D("a"),A("grey"),N("mouse"))) →
S(NP(D("the"),N("cat").n("p")),
VP(V("eat"),
NP(D("a"),A("grey"),N("mouse"))))
→
jsRealB expressions are built using Constituents which can be either Terminals or Phrases shown in the following table. Terminals take a single string as parameter and Phrases can take an indefinite number of parameters, either Terminals or Phrases. For building sentences, an alternative dependency grammar notation can also be used, but it is not described in this tutorial.
| Terminal | Phrase | |
|---|---|---|
| Determiner | D("..") | |
| Pronoun | Pro("..") | |
| Noun | N("..") | NP(...) |
| Adjective | A(("..") | AP(...) |
| Verb | V("..") | VP(...) |
| Adverb | Adv("..") | AdvP(...) |
| Preposition | P("..") | PP(...) |
| Conjunction | C("..") | CP(...) |
| Subordination | | SP(...) |
| Sentence | | S(...) |
| Date | DT("..") | |
| Number | NO("..") | |
| Quoted text | Q("..") |
The first parameter in a Phrase is usually its head. For example in a Sentence, the first parameter is taken as the subject. The first parameter of a Verb Phrase is its Verb, followed by a direct object unless it is a Prepositional Phrase. These implicit roles are used for setting up the agreement links between constituents. For example, the number of the subject is used as the number of the verb. If these links are not properly set, it is possible to set features explicitly, but most often the default rules are sufficient.
From a single sentence structure, it is possible to get many variations (in blue in this table) by simply adding features (written as methods called using the dotted notation) such as the ones in the following table in which the previous sentence is at the future tense:
| future | S(NP(D("the"),N("cat")),
VP(V("eat").t("f"),
NP(D("a"),A("grey"),N("mouse")))) |
|
| passive | S(NP(D("the"),N("cat")),
VP(V("eat").t("f"),
NP(D("a"),A("grey"),N("mouse"))))
.typ({"pas":true})
|
|
| question about the subject | S(NP(D("the"),N("cat")),
VP(V("eat").t("f"),
NP(D("a"),A("grey"),N("mouse"))))
.typ({"int":"why"})
|
|
| negative | S(NP(D("the"),N("cat")),
VP(V("eat").t("f"),
NP(D("a"),A("grey"),N("mouse"))))
.typ({"neg":true})
|
|
| combination of the above | S(NP(D("the"),N("cat")),
VP(V("eat").t("f"),
NP(D("a"),A("grey"),N("mouse"))))
.typ({"int":"why","pas":true,"neg":true})
|
|
Because language generation is not limited to producing list of words, jsRealB also
provides facilities for adding punctuation, wrapping constituents in symbols and HTML formatting.
çFormatting is indicated with features added to constituents. To add a string before a constituent,
use .b("..") after building the constituent, for example
N("man").b("*") → ,
similarly .a("..") to add a string after. If the string is an usual punctuation,
appropriate spacing before and after the sign is added. Pairs of symbols can be added with .ba(".."),
for example N("mouse").ba("[") → .
It is also possible to add HTML tag around any constituent with the .tag(name,
attributes). The tag name is a string and the optional attributes are given by
an object whose keys are the attribute names with the corresponding value. For exemple,
N("mouse").tag("i") →
displayed as in an HTML page. To create hyperlinks,
we can use the a tag, for example
N("mouse").tag("a",{href:"https://en.wikipedia.org/wiki/Mouse"}) →
rendered as in a web page.
const cats=NP(D("the"),N("cat").n("p"));
const mouse=NP(D("a"),A("grey"),N("mouse"));
S(cats,VP(V("eat"),mouse));
This is very useful for structuring a complex text built incrementally. This also makes possible to share repeating parts in a text.
The constructors corresponding to constituents create new objects, but features
(i.e. methods called with the dot) modify them permanently.
So some care must be taken when reusing structures saved in variables.
For example, with those declarations:
const cat1=NP(D("the"),N("cat"));
const mouse1=NP(D("a"),A("grey"),N("mouse"));
S(cat1,VP(V("eat"),mouse1.pro())) →
in which mouse1 replaced by a pronoun. If we reuse those variables in another context such as
S(cat1,VP(V("love"),mouse1)) → in which mouse1 is again pronominalized which is probably not intended.
This happened because the features modified the original structure. In this case, the modification was applied directly on the value of the variable, but as some properties (e.g. gender, noun, negation, etc.) are propagated between constituents for achieving the appropriate agreements, modifications to shared variables can be more subtle and hard to identify because modifications can happen from a governing or from a sibling constituent. To get around this problem, we can clone any jsRealB structure to get a new copy of a value. So going back to our original declarations
const cat2=NP(D("the"),N("cat"));
const mouse2=NP(D("a"),A("grey"),N("mouse"));
S(cat2,VP(V("eat"),mouse2.clone().pro())) →
without modifying the original as can be seen with
S(cat2,VP(V("eat"),mouse2)) →
A very convenient way of defining a jsRealB object is using an arrow function that returns a new structure at each call, thus minimizing the risk of inadvertant permament modification. For example
const mouse3 = ()=>NP(D("a"),A("grey"),N("mouse"))
which can be used by applying the function mouse3 as
S(cat2,VP(V("eat"),mouse3().pro())) → S(cat2,VP(V("eat"),mouse3())) →
Computer generated texts often suffer from repetitions which can be quite annoying for humans who prefer some variation in the texts they read. jsRealB makes it easy to create different phrase and sentence structures by creating many constituent tree expressions from which the program can choose.
In many cases semantic or pragmatic constraints determine the choice, but in some cases it is appropriate
to choose randomly.
The oneOf(e1,e2 ...) function lets jsRealB choose one of
the arguments ei. As JavaScript evaluates all arguments of a function before the call, we can prefix an
argument with ()=> to transform it into an arrow function which will delay evaluation
after the selection by oneOf.
We define three functions:
np() to create a noun phrase with different animal nouns, either singular or plural;a() to provide an optional adjectivevp() to create a verb phrase (at present, past or future) possibly with a complement
function np(){
return NP(D("a"),a(),N(oneOf("cat","mouse","dog","rabbit")).n(oneOf("s","p")));
}
function a(){
return oneOf(
()=>A(oneOf("hungry","grey","nervous")),
()=>Q("")
);
}
function vp(){
return oneOf(
()=>VP(V(oneOf("sit","run","laugh")).t(oneOf("p","ps","f"))), // intransitive verb
()=>VP(V(oneOf("eat","love")).t(oneOf("p","ps","f")), // transitive verb
np())
);
}
With these definitions, S(np(),vp()) will produce a grammatically correct random sentence (containing a noun phrase followed by verb phrase) at each call.
Click this button to
This example is a JavaScript transcription of the Prolog/DCG version described in: Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 256-257, Ellis Horwood, 1990.
We now present a complete web application for finding a path in the Montréal Métro (subway) network. We will then explain how to generate the itinerary between two stations clicked by user that is generated to the right of the graphic. The application can be used in either French or English with the text generated in the same language, by clicking on the [EN] or [FR] in the top right corner. In this tutorial, only the English version is described, as the French version is similarly organized and differs mainly in the choice of words.
Text generation is classically divided in two main steps:
The first step in a data-to-text project is to find the appropriate data and to massage it into a usable format. In this case, the data was harvested from a public dataset, although in a proprietary format for which we fortunately found a web site that allowed us to transform it to geojson. We then used a jq script to extract only the information we needed: metro lines with their stations for which we kept their name and coordinates. The jq output was then feed to a small Python script to get a more useful display. We finally hand sorted the station according to their real ordering on each line. As always, this conceptually simple task took more effort than we had initially expected.
Here is a short excerpt of the data for the 68 stations of the metro. Some transfer stations (e.g. Berri-UQAM) appear in more than one line as indicated in their route field.
var routes = [
{"route":1, "stations":[
{"id": "18", "station": "Honoré-Beaugrand", "route": "1", "coords": [302040.08698887925, 5050741.44628119]},
{"id": "19", "station": "Radisson", "route": "1", "coords": [301735.9800496101, 5049947.9922192525]},
...
{"id": "43", "station": "Angrignon", "route": "1", "coords": [296733.6694496933, 5034064.601947192]}
]},
{"route":2, "stations":[
{"id": "68", "station": "Montmorency", "route": "2", "coords": [287432.767743489, 5046526.781216757]},
...
{"id": "65", "station": "Côte-Vertu", "route": "2", "coords": [290521.09550694964, 5041607.070800175]}
]},
{"route":4, "stations":[
{"id": "11", "station": "Berri-UQAM", "route": "1,2,4", "coords": [300055.51811076206, 5041690.301771889]},
...
{"id": "44", "station": "Longueuil/Université-de-Sherbrooke", "route": "4", "coords": [303073.79689146805, 5042774.93822535]}
]},
{"route":5, "stations":[
{"id": "49", "station": "Snowdon", "route": "2,5", "coords": [294829.623185072, 5038412.072364335]},
...
{"id": "64", "station": "Saint-Michel", "route": "5", "coords": [296993.06301460037, 5046632.485787434]}
]}
]
With this data we build an internal data structure of the underlying network having stations as node and estimated travel time (computed from the distance from the coordinates of the stations) as value on the arcs between two consecutive stations. Transfer stations are duplicated on each line and an internal link with transfer time between each line is added.
In the application, once the user has chosen the two stations between which she wants to travel, the shortest path between them is computed using the Bellman-Ford algorithm which is very simple to implement and efficient enough on such a small network.
The shortest path is a list of nodes that is split into legs between transfer stations. For example, the path to go from Honoré-Beaugrand (id="18") to Université de Montréal (id="57") will be split in three legs corresponding to travels on a line between transfer:
[[["18",0],["19",1.41],...,["11-1",15.91]],
[["11-2",20.91],["10",22.23],...,["5-2",29.14]],
[["5-5",34.14],["61",35.17],...,["57",40.75]]]
The trip is thus a list of legs which are themselves lists of pairs also lists, the first element being the node id and the second element being the distance in minutes from the starting node.
The generated text is organized as:
This is implemented in this function, which first creates a title wrapped in an h2 tag. That title is added in front of the rest of the text either as a direct trip singleLine or one separated into multiple legs (introduction, body, conclusion).
function generate(trip){
var duration=last(last(trip))[1];
var title=VP(V("go").t("b"),P("from"),Q(network[trip[0][0][0]].stationName),
P("to"),Q(network[last(last(trip))[0]].stationName)).cap().tag("h2")+"\n";
if (trip.length==1){
return title+singleLine(trip[0],duration);
} else {
let text=title;
text+=introduction(trip[0],duration).tag("p")+" ";
text+="<p>"+body(trip.slice(1,trip.length-1))+"</p>";
text+=conclusion(last(trip)).tag("p");
return text;
}
}
We define a few auxiliary functions to realize:
genLine(...) to realize the "name" of a leg: its color and the direction given as the name of start/end station depending on the direction of the trip as given by the indices of the first and last station of the leg;nbStations(...) to realize the number of stations or stops (chosen arbitrarily): when there is only station, it is written differently.destination(...) to realize the station name of the destinationfunction genLine(leg){
var from=network[leg[0][0]];
var to=network[last(leg)[0]];
var res=NP(D("the"),A(lineColor[from.route]),N("line"));
if (from.index<to.index){
if (to!=from.end)
// do not add direction when the destination is at the last station which will be realized anyway
res.add(PP(P("towards"),Q(from.end.stationName)));
} else {
if (to!=from.start)
// do not add direction when the destination is at the first station which will be realized anyway
res.add(NP(N("direction"),Q(from.start.stationName)).en('('));
}
return res;
}
function nbStations(leg,ord){
if (ord===undefined)ord=false;
var st=N(oneOf("station","stop"));
if (leg.length==2){
return NP(Adv("only"),NO(1).dOpt({"nat": true}),st);
}
return NP(NO(leg.length-1).dOpt({"ord":ord}),st)
}
function destination(leg){
return Q(network[last(leg)[0]].stationName);
}
After testing the boundary case that the trip starts and ends at the same station, the realizer creates two subordinate phrases by choosing randomly between a few formulations. These two subordinate phrases are then wrapped into a single sentence.
function singleLine(leg,duration){
if (duration==0)
return S(Pro("I").pe(2),V("be"),Adv("already"),PP(P("at"),D("my").pe(2),N("destination"))).a("!");
var sp1 = oneOf(()=>SP(D("this"),V("be"),CP(C("and"),A("simple"),A("fast")),nbStations(leg),
Adv("no"),N("transfer")),
()=>SP(Pro("I").pe(2),
VP(V("be"),Adv("only"),nbStations(leg),Adv("away"))),
()=>SP(Pro("this"),VP(V("make").t("f"),nbStations(leg),
PP(P("for"),NP(D("the"),A("whole"),N("trip")))))
).a(",");
var sp2 = oneOf(()=>SP(V("take").t("ip"),genLine(leg)),
()=>SP(V("follow").t("ip"),genLine(leg),
PP(P("for"),NP(NO(Math.round(duration)),N("minute")))));
return S(sp1,sp2).toString();
}
The introduction starts with global information: the estimated time to destination and then, with different formulations, invite the user to start the trip with the first leg.
function introduction(leg,duration){
var sp1 = S(Pro("I").pe(2),VP(V("be"),NP(NO(Math.round(duration)),N("minute"))),
PP(P("from"),NP(D("my").pe(2),N("destination")))).a(",");
var sp2 = oneOf(
()=>SP(V("take").t("ip"),genLine(leg),
PP(P("for"),nbStations(leg))),
()=>SP(V("board").t("ip"),PP(P("on"),genLine(leg)))
);
var pp = oneOf(()=>PP(P("until"),SP(Pro("I").pe(2),
VP(V("be"),PP(P("at"),destination(leg))))),
()=>PP(P("up"),P("to"),destination(leg))
);
return S(sp1,sp2.add(pp));
}
The body recursively realizes each leg with different formulations
function body(legs){
if (legs.length==0)return "";
// console.log(legs);
var leg=legs[0]
var out=oneOf(
()=>S(VP(V("take").t("ip"),genLine(leg),PP(P("up"),P("to"),destination(leg)))),
()=>S(VP(V("change").t("ip"),PP(P("on"),genLine(leg),P("until"),destination(leg)))),
()=>S(VP(V("transfer").t("ip"),PP(P("for"),destination(leg),P("on"),genLine(leg)))),
()=>S(VP(V("switch")).t("ip"),
PP(P("on"),genLine(leg),P("for"),nbStations(leg)))
);
return out+" "+body(legs.slice(1));
}
The conclusion ends by indicating that the trip is finished.
function conclusion(leg){
var out=oneOf(
()=>S(Adv("finally"),Pro("I").pe(2),V("take").t("f"),genLine(leg),
V("arrive").t("b"),PP(P("to"),destination(leg))),
()=>S(Q("at last").a(","),Pro("I").pe(2),
VP(V("find").t("f"),destination(leg).a(","),
PP(nbStations(leg,true),P("on"),genLine(leg)))),
()=>S(NP(D("my").pe(2),N("destination"),destination(leg)),
VP(V("be").t("f"),P("after"),nbStations(leg),
P("on"),genLine(leg)))
);
return out;
}
This JavaScript realizer using jsRealB is an adaptation of the Prolog/DCG version shown in Michel Boyer and Guy Lapalme, Text Generation, Chapter 12 of Logic and Logic Grammar for Language Processing, pp. 258-260, Ellis Horwood, 1990.
This tutorial has illustrated how jsRealB can be integrated in a web page to create dynamic textual content.