Program

All timeslots are CET (UTC+1).

January 29: STAND workshop

• [09:00-09:10] Welcome & logistics
• [09:10-10:10] Introduction to standardization, ongoing activities in NLP standardization, and the AI Act context
• [10:15-11:00] Keynote: Joakim Nivre
  • Towards a Standard for Morphosyntactic Annotation: The Universal Dependencies Experience
• [11:00-11:30] Invited talk: Matt Post
  • Setting parameters for metrics: a few ideas for creating a thriving and useful metric ecosystem
• [11:30-11:40] Poster boosters
• [11:40-13:00] Poster session & lunch
  • Annotated corpora through time: a case study on NER (Yoann Dupont)
  • Assessing the performance of LLMs: the problem of emergent abilities (Céline Budding)
  • Fine-Tuning Llama 2: Evaluating Text Generation with Cosine Similarity and Human Assessment (Alina Lozovskaya)
  • Standardization challenges to a better evaluation in entity normalization (Arnaud Ferré)
• [13:00-13:30] Invited talk: Dirk Hovy
  • Measurement Error: Are we Evaluating the Right Things?
• [13:45-14:00] Invited talk: APIL (French association of NLP industry)
• [14:00-16:30] Moderator-led breakout discussions, 3 slots (14:00-14:45, 14:45-15:30, 15:45-16:30).
  • Room 1:
    • [sharing / drafting] Standardizing good practices for evaluation
    • [sharing / drafting] Standardizing evaluation metrics (definitions, implementation, sharing scripts)
    • [debate] Explainability and ethics in NLP: what needs for standards?
  • Room 2:
    • [sharing / drafting] Standardizing good practices for corpus management (collection, annotation, versioning)
    • [sharing / drafting] Standardizing annotation schemes (formats and guidelines)
    • [debate] Comparing standardization needs with limitations of the state-of-the-art: how to bridge the gap?
• [16:30-17:30] Reports from breakouts, definition of community-level actions & wrap-up. Example outcomes that are envisioned include:
  • Collection and drafting of existing good practices
  • Preparation of a joint submission for a position paper
  • Creation of common repositories for evaluation scripts, corpus documentation

An informal dinner will be organized with participants staying overnight: 19:00 at Le Viaduc, 43 avenue Daumesnil, 75012 Paris, France.

January 30: meeting of the European AI standardization committee about NLP standardization

Workshop participants are invited to stay to contribute any insight. The outputs of that meeting will be used in direct support of the EU AI Act, according to the European Commission’s mandate.

• [09:00-09:10] Welcome & logistics
• [09:10-10:00] Preparing inputs to ISO/IEC AWI 23282
  • Artificial Intelligence — Evaluation methods for accurate natural language processing systems
• [10:00-10:30] New standardization needs: language diversity?
  • A standard on language diversity in NLP? Considering language diversity in NLP standards?
• [10:45-11:30] Towards standardizing robustness criteria & domains in NLP
  • Building a collaborative taxonomy
• [11:30-13:00] Lunch break
• [13:00-14:00] New standardization needs: bias in NLP?
  • Sources, mitigations, metrics?
• [14:00-14:30] Preparing inputs to ISO/IEC TR 23281
  • Artificial intelligence — Overview of AI tasks and functionalities related to natural language processing
• [15:00-16:00] Addressing the AI Act requirements on general-purpose AI models: let’s be creative
• [16:00-18:00] Roadmapping NLP standardization: searching for blind spots
  • Documentation? Annotation process?