Scientific Context

The current lack of standardized practices and definitions in NLP systems hinders the progress of the field. Indeed, there is not always consensus on which evaluation methods are meaningful and fruitful, or which of their implementations are to be used with which parameters (eg. SacreBLEU, Post 2018). In some cases, there is no general agreement on the very definition of a task. This situation calls for work on standardizing NLP practices.

The International Organization for Standardization (ISO) has just created a dedicated working group on NLP (as a joint effort of the AI and Language committees), and 2 standards are already under way. Topics under consideration by the ISO standardization committees include NLP terminology, evaluation metrics, interoperability, annotation guidelines, good practices in NLP development/evaluation/corpora, documentation.

These topics are already heavily discussed in academia, and a number of informal guidelines have already been proposed. We believe that the creation of NLP standards can significantly benefit from the input of both NLP academics and industry NLP practitioners. Reciprocally, NLP researchers would benefit from getting involved in the standardization effort, thus ensuring that academia’s views are listened to, in particular in the context of the AI Act (the European regulation on AI that has been finalized in December), whose enforcement will strongly rely on those standards.

The objective of the STAND workshop is

  • to foster discussion on existing standards, their creation and use
  • to assess the current needs of the community for standardization
  • to share experience on the impact on the research activities when lacking good practices
  • to collect existing good practices (and propose new ones)

We invite contributions from NLP practitioners from both the industry and academia, as well as standardization experts.

See the call for contributions and the program.

Invited Speakers

Organizing committee

  • Lauriane Aufrant, Inria
  • Rania Wazir, leiwand.ai
  • Timothée Bernard, Université Paris Cité, LLF
  • Taras Holoyad, BNetzA
  • Yoann Dupont, Université Sorbonne Nouvelle
  • Maximin Coavoux, CNRS, LIG
  • Arnaud Ferré, Inrae, MAIAGE