#benchmarks #sota

Я не согласен с автором, что надо забить на бенчмарки в сфере интерпретабельности. "Просто" нужны хорошие синтетические бенчмарки.

"The obsession with benchmarks and SOTA runs deep:

Creation of benchmark islands.
People on social media arguing over which ML algorithm is better.
Difficulties in publishing new approaches that don’t beat the state-of-the-art.
LLM evaluation based on benchmarks even when they start memorizing them.

The hope is that the performance on these benchmark tasks and datasets are predictive of performance on new datasets. Ideally, the benchmark datasets are representative of the typical dataset you would work on in the future. But it’s not like we can sample from the distribution of datasets. Benchmarks are guided by what datasets are openly available (huge selection bias already) and which datasets are convenient to use (for example in clean CSV format and not in some wild Excel construct). Benchmarks are not representative samples, they are arbitrary samples."

https://mindfulmodeler.substack.com/p/we-are-obsessed-with-benchmarks