The official start of the project was on 1 December 1993, and it will run until October 1996. Its consortium includes several academic partners who already have some experience in building and using test suites: the University of Essex (Colchester, UK), DFKI--the Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH.--(Saarbruecken, Germany), and ISSCO (Geneva, Switzerland); and one industrial partner who needs to process large amounts of texts and has also built its own test suites for evaluating NLP applications: Aerospatiale (France).

Obviously, any NLP system, whether commercial or under development, possesses specific features which make it unique, and any user or developer of an NLP system has specific needs and requirements. Moreover, testing or evaluating NLP systems is performed for a variety of purposes. For all these reasons, our approach to TS design is based on the assumption that, to yield informative and interpretable results, any TS used for an actual test or evaluation must be specific, at least to some degree, to the system and the user. On the other hand, it is also guided by the need to provide test material which is easily reusable for a variety of purposes.

In order to achieve these two goals, the traditional notion of a TS as a monolithic list of test items has been abandoned in favour of the notion of a database, where stored test items are associated with various types of annotations. From this database, purpose- and application-specific sets of test items can be extracted by searching on specific annotations or annotation types. Reusability is enhanced by tools to extract and manipulate test items, in particular to perform lexical replacement, as well as by tools to automatically generate additional test items. The TS database can be seen as a "virtual test suite" which allows users to construct their own specialized TSs.

The first stage of the project involved reviewing already existing test suites and guidelines for test suite construction. The second phase of the project consisted in the design of the annotation scheme: how to describe the input data for each test item and how to specify the expected output. Equally important is the fact that the annotation scheme also makes explicit the relationship between the various test items in a TS. Indeed, since a TS must test single phenomena one at a time and must also allow testing for interacting phenomena, it is necessary to create test items in a systematic and progressive way, going from simple to complex items, with negative as well as positive test items, and to make explicit the relationship between the test items.

The project aims to produce a set of guidelines for the construction of TSs for a range of NLP products, concentrating on Grammar Checkers, Parsers and Controlled Language Checkers. Important issues in the design of TSs are currently being identified and will be addressed for each application type.

In addition, the project is producing test data in three languages: English, French and German, thus building a multilingual TS database, and it strives to make the data as parallel as possible by following a common design and trying to cover a common set of linguistic phenomena. The results of the project will become public domain and will, hopefully, provide impetus for a more general development of test suites and other evaluation tools.

Intermediate reports are now available.