HardEval: Focusing on Challenging Tokens to Assess Robustness of NER

TitleHardEval: Focusing on Challenging Tokens to Assess Robustness of NER
Publication TypeConference Paper
Year of Publication2020
AuthorsBernier-Colborne, G., and P. Langlais
Conference NameProceedings of The 12th Language Resources and Evaluation Conference
PublisherEuropean Language Resources Association
Place PublishedMarseille, France
ISBN Number979-10-95546-34-4
AbstractTo assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity. These subsets provide a system-agnostic basis for evaluating specific sources of NER errors and assessing room for improvement in terms of robustness. We analyze these subsets of challenging tokens in two widely-used NER benchmarks, then exploit them to evaluate NER systems in both in-domain and out-of-domain settings. Results show that these challenging tokens explain the majority of errors made by modern NER systems, although they represent only a small fraction of test tokens. They also indicate that label shift is harder to deal with than unknown words, and that there is much more room for improvement than the standard NER evaluation procedure would suggest. We hope this work will encourage NLP researchers to adopt rigorous and meaningful evaluation methods, and will help them develop more robust models.