Evaluation and fairness
AI systems do not perform equally across languages, dialects, accents and recording conditions. Aggregate results can hide failures that matter to particular communities, while evaluation datasets may inadequately represent the people a system is intended to serve.
Read more +Read less
We investigate how speech and language models should be evaluated across their intended populations and operating environments. This includes language-specific benchmarks, performance disparities, dataset limitations, robustness testing and transparent reporting of results.
Our research is guided by the principle that model performance should be assessed in context. Evaluation should inform decisions about a system's intended uses, known limitations and readiness for release.
