a workshop to be
held at Coling 2000,
This workshop will focus on methods, grammars, and data to facilitate empirical assessment and comparison of the efficiency of large-scale parsing systems.
Interest in large-scale, grammar-based parsing has recently seen a large increase, in response to the complexities of language-based application tasks such as speech-to-speech translation, and enabled by the availability of more powerful computational resources and by efforts in large-scale and collaborative grammar engineering.
There are two main paradigms in the evaluation and comparison of the performance of parsing algorithms and implemented systems: (i) the formal, complexity-theoretic analysis of how an algorithm behaves, typically focussing on worst-case time and space complexity bounds; and (ii) the empirical study of how properties of the parser and input (possibly including the grammar used) affect actual, observed run-time efficiency.
It has often been noted that the theoretical study of algorithms alone does not (yet) suffice to provide an accurate prediction about how a specific algorithm will perform in practice, when used in conjunction with a specific grammar (or type of grammar), and when applied to a particular domain and task. Therefore, empirical assessment of practical parser performance has become an established technique and continues to be the primary means of comparison among algorithms. At the same time, system competence (i.e. coverage and overgeneration with respect to a particular grammar and test set) cannot be decoupled from the evaluation of parser performance, because two algorithms can only be compared meaningfully when they really solve the same problem, i.e. either directly use the same grammar, or at least achieve demonstrably similar competence on the same test set.
The focus of the workshop is on large-scale parsing systems and precise, comparable empirical assessment. We envisage discussion at the workshop will centre on methods, reference grammars, and test data that will facilitate improved comparability. The workshop is intended to bring together representatives from sites working on grammar-based parsing (both in academic and corporate environments) to help the field focus and converge on a common, pre-standard practice in empirical assessment of parsing systems.
John Carroll, University
of Sussex, UK;