Minnesota Natural Language Processing Seminar Series: Dynabench: Rethinking Benchmarking in AI
The Minnesota Natural Language Processing (NLP) Seminar is a venue for faculty, postdocs, students, and anyone else interested in theoretical, computational, and human-centric aspects of natural language processing to exchange ideas and foster collaboration. The talks are every other Friday from 12 p.m. - 1 p.m. during the Spring 2022 semester.
This week's speaker, Douwe Kiela (Huggingface), will be giving a talk titled "Dynabench: Rethinking Benchmarking in AI."
The current benchmarking paradigm in AI has many issues: benchmarks saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts, have unclear or imperfect evaluation metrics, and do not necessarily measure what we really care about. I will talk about our work in trying to rethink the way we do benchmarking in AI, specifically in natural language processing, focusing mostly on the Dynabench platform (dynabench.org).
Douwe Kiela (@douwekiela, https://douwekiela.github.io/) is the Head of Research at Hugging Face. Before, he was a Research Scientist at Facebook AI Research. His current research interests lie in developing better models for (grounded, multi-agent) language understanding and better tools for evaluation and benchmarking.