Description of the jsRealB generator for the E2E Challenge

Guy Lapalme, RALI-DIRO, Université de Montréal

Note: this work was initially performed in October 2018, so these results were not part of the E2E Challenge official competition. In October 2023, the system was reengineered using JavaScript classes.

Context

The E2E Challenge dataset was designed to provide data for developing algorithms to generate text from a dialogue act-based meaning representation (MR). The underlying assumption was that this problem should be dealt with a machine-learning (ML) approach. The data, described in this paper, was used in a NLG competition .

Although we have great respect for ML, we thought it might be interesting to compare the results of these systems with a much simpler symbolic system, using jsRealB, a bilingual English/French realizer written in Javascript that we have developed over the last few years. We describe here only the English realization, but we also developed a French version having the similar structure for which we mostly translated the words in French and adapted some formulations. Because of the independent random variation between the English and French realization, these sentences are not literal translations of each other, although they convey the same meaning being generated from the same MR. When used in the demonstration page, the realization is performed by the browser, but for the batch tests shown below corresponding to the shared task, we use the same code as a node.js module.

After examining a few sentences of the development set (we never looked at the test set during the development), we designed a template-based text structure, filled with the values of the attributes of MR. In jsRealB, we first build a constituency structure for the sentence with constructors corresponding to the conventional constituent symbols (e.g. S for a sentence, NP for a Noun Phrase, N for a noun, etc...). When the string value structure is needed, most of the details for the final realization (capitalization, gender and number agreement, etc.) are dealt by jsRealB. For more details, see the tutorial and the documentation.

Our system realizes English and French sentences using methods in JavaScript classes that return jsRealB structures, but we show here only the functions of the methods of the class for English sentences as the ones for the French are similar.

Text organization

Restaurant description
A sentence that starts with the name of the place (given by the attribute name), the verb be, the nearby place (attribute near), the region (attribute area), the type of food served (attribute food) and the range of prices (attribute priceRange). Undefined values are generated for missing attributes and then ignored by jsRealB. This is written as follows in Javascript:
advice(fields) {
    // "name":[  "Alimentum", ... ],
    const name="name" in fields?Q(fields["name"]):NP(D("the"),N(oneOf("restaurant","establishment"))); 
    // "eatType":[ "coffee shop", "pub", "restaurant" ]
    const eatType=NP(D("a"),N("eatType" in fields?fields["eatType"]:oneOf(this.allPlaces)));
    // "near":[ "Yippee Noodle Bar", ... ],
    const near="near" in fields?PP(P("near"),Q(fields["near"])):null;
    // "area":[ "riverside", "city centre" ],
    let area, food, priceRange;
    if ("area" in fields){
        area=fields["area"]=="riverside" 
          ? oneOf(()=>PP(P("on"),NP(D("the"),N("riverside"))),
                  ()=>PP(P("in"),NP(D("the"),N("riverside"),N("area"))),
                 )
          : PP(P("in"),NP(D("the"),N("city"),N("centre")))
    }

    // "food":[ "Chinese", "English", "Fast food", "French", "Indian", "Italian", "Japanese" ],
    if ("food" in fields){
        const fo=fields["food"];
        if (fo=="Fast food")fo="fast";
        food=Q(fo);
        const npFood=NP(food,N("food"));
        const serve=V(oneOf("serve","provide","have","offer"))
        food = oneOf(()=>SP(Pro("that"),serve,npFood),
                     ()=>VP(serve.t("pr"),npFood))
    };
    // "priceRange":[ "cheap", "high", "less than £20", "moderate", "more than £30", "£20-25" ],
    if ("priceRange" in fields){
        const pr=fields["priceRange"];
        priceRange=pr.indexOf("£")>=0?PP(P("with"),N("price").n("p"),Q(pr))
                                    :PP(P("with"),NP(A(pr),N("price").n("p")));
    }
    return S(name,VP(V("be"),eatType,near,area,food,priceRange))
}
The customer rating (attribute customerRating) written in Javascript as follows:
customerRating(fields){
    // "customer rating":[ "5 out of 5", "average", "1 out of 5", "low", "3 out of 5", "high" ]
    if ("customer rating" in fields){
        const cr=fields["customer rating"];
        return S(Pro("I").g("n"),
                         VP(V("have"),
                            oneOf(()=>NP(D("a"),Q(cr),oneOf(N("customer"),Q("")),N("rating")),
                                  ()=>NP(D("a"),oneOf(N("customer"),Q("")),N("rating"),P("of"),Q(cr))))
                         )
    }
}
The family friendliness (attribute familyFriendly) written in Javascript as follows. jsRealB can realize the negation of a sentence by a simple flag.
familyFriendly(fields){
    // "familyFriendly":[ "yes", "no" ],
    if ("familyFriendly" in fields){
        return S(oneOf(()=>Pro("I").g("n"),
                       ()=>NP(D("the"),N(oneOf(this.allPlaces))),
                       ()=>Pro("I").n("p")
                      ),
                         VP(V("be"),
                            NP(oneOf(N("family").lier(),N("kid")),A("friendly").pos("post"))))
                       .typ({"neg":fields["familyFriendly"]=="no"})
    }
}

These three structures are then realized and concatenated.

realize(fields,phrase_type){
	// Simple concatenation of the realization taking care of undefined return
    function noUndef(struct){
        if (struct == undefined) return ""
        return struct.realize()
    }
    return noUndef(this.advice(fields))
           + noUndef(this.customerRating(fields))
           + noUndef(this.familyFriendly(fields))
}

This simple solution was developed in a few hours (50 lines of Javascript), but it gave good enough results with instantaneous response time all in the browser. The javascript code of the realizer for both French and English.

Evaluation

We then ran the evaluation scripts of the E2E competition on the test set and we obtained the following results (we could not manage to get the METEOR scoring to work on our machine):

ScoreValue
BLEU0.4980
NIST7.3038
ROUGE_L0.5878
CIDEr1.8387

These scores are not the best, but they are competitive. Given the fact, that all sentence are well formed, with good syntax and capitalization, they would probably have gotten good human evaluation scores!

Conclusion

We found this exercise very interesting and we thank the organizers to accept to show an alternative to the usual ML approach for the E2E challenge. It would probably be interesting to see if an ML approach between the attribute values and the constituency structure of jsRealB would be feasible and if it would improve the results.