A 90% score does not make an AI an expert

What is a benchmark?

A benchmark is a fixed set of questions that many AI models answer. The score lets people compare them.

Moritz Hardt (a computer scientist at the Max Planck Institute for Intelligent Systems) says the idea is old. In an interview published on 16 October 2026, he says the first one was in 1959. It tested how well machines read digits and letters on punch cards.

Why can’t we trust the score?

Hardt says a small change to the test can move the score a lot. His example: a model might get 90% on one set of questions and 73% on a slightly different set. Which number is the truth?

Another problem: AI makers can practise on the test. The questions are often public. A model can be trained on questions like them, then look better than it is. Hardt uses a slang word for this, “benchmaxxed”.

So what can we trust?

Hardt says the order of the models has held up well. If one model beats another on one test, it usually does on others too. He says no one has a full explanation for why.

That order breaks down for chatbots, because their makers do not share what they trained on.

Hardt’s idea is to give every model the same preparation before the test. He compares it to the 1968 Olympics in Mexico City. A Kenyan runner who trained at high altitude, Kip Keino, won the 1,500 metres. The answer, Hardt says, was to let everyone train at altitude.

What’s the catch?

• This is one expert’s view, given in an interview. His fix is a proposal, not a standard that anyone uses yet.
• The 90% and 73% figures are his example of how scores can move. They are not results for named models.

Try it

This step is our idea, not Hardt’s. Next time a company says its AI scored 90% on a test, ask two questions. What did it beat? Was the test new, or older than the model?

A score tells you more as a comparison than as a grade.

Sources

• He Wants AI Benchmarking To Be a Science, Not a Dark Art, Quanta Magazine, 16 October 2026
• The Emerging Science of Machine Learning Benchmarks, Moritz Hardt, Princeton University Press