BarcelonaEvaluation

Discussion: Evaluation

Moderator: Montserrat Marimon; Scribe: Rebecca Dridan

Objective

Share ideas among grammarians about evaluation:

1- Diagnosis evaluation (regressing testing while developing):

Test cases which include all variations of a particular phenomenon and related ungrammatical constructions: linguistically interesting – controlled hand-built test suites.

Test data that present combinations of different phenomena, showing the real world language complexity (not linguistically interesting) – real text.

2- Direct evaluation (grammar performance in terms of recall and precision (robustness)).

Using a Treebank...

- Built by chosen sentences - Built by real text, showing interaction of phenomena : What kind of corpus? Newspaper articles (syntactic errors, misspellings, foreign words...), domain specific (missing lexical entries), mixture

3- Indirect evaluation (grammar usability/adequacy for a specific application).

Notes