Affordable Access

Unsupervised Evaluation of Parser Robustness

Publication Date
  • Computer Science


Unsupervised Evaluation of Parser Robustness Johnny Bigert1, Jonas Sjo¨bergh1, Ola Knutsson1, and Magnus Sahlgren2 1 KTH Nada, 100 44 Stockholm, Sweden, {johnny,knutsson,[email protected] 2 SICS, Box 1263, 164 29 Kista, Sweden, [email protected] Abstract. This article describes an automatic evaluation procedure for NLP sys- tem robustness under the strain of noisy and ill-formed input. The procedure requires no manual work or annotated resources. It is language and annotation scheme independent and produces reliable estimates on the robustness of NLP systems. The only requirement is an estimate on the NLP system accuracy. The procedure was applied to five parsers and one part-of-speech tagger on Swedish text. To establish the reliability of the procedure, a comparative evaluation involv- ing annotated resources was carried out on the tagger and three of the parsers. 1 Introduction Automatic parsing of text is a popular field of research. Many of the applications where parsing is used, such as parsing human input to a computer system, handle text that is not proofread. Depending on the application, the text can be relatively error free (e.g. parsing newspaper articles from the internet) or contain large amounts of errors (e.g. us- ing a parser as a tool for second language learners when writing essays). If the intended use of a parser is domains with many errors, it must be robust enough to produce useful output despite noisy input. It is not sufficient to achieve a good performance on error- free text. Usually, the accuracy of a parser on error-free text is known, but the accuracy on texts containing errors is often unknown. Carroll and others give a comprehensive overview of different parser evaluation methods and discuss some shortcomings [1]. Evaluation of parsers is usually carried out by comparing the parser output to a manually annotated or manually corrected version of a test text. Manual work is expensive, and not necessarily error free. If the NLP system is under development, the

There are no comments yet on this publication. Be the first to share your thoughts.