“When a measure becomes a target, it ceases to be a good measure” – Goodhart’s law
Current AI research, especially the frontier LLM research, is dominated by benchmarks. It is the first thing we look at when a new model comes out, it is the headline of each release, and they dominate the discourse when it comes to measuring progress. There are more benchmarks than you or I could count and new ones come out every few weeks. I am looking at some of them for many years and I want to give an introduction into some of them, but more importantly talk about some of the problems with certain benchmarks, and the bench-marking methodology in general.
What is a benchmark
Benchmarks are tests designed to measure the performance of a system in certain categories or metrics. For LLMs, specifically this means we have benchmarks that measure, capabilities in math, reasoning, long context understanding, etc, but also alignment, bias, safety, generalization, and so on.
I generally sort benchmarks into four categories. The first is benchmarks that define a question or task and have a fixed correct answer key that the model has to output. The second one are benchmarks that are more open ended where the final output of the model needs to be judged in some way, often using a separate AI model. The third category are Elo-based benchmarks where models get a relative score based on how they perform against other models. A good example is ChessBench where models play chess against each other. Or more famously LM arena which also fits in the second category of judged results. The last category are meta benchmarks that try to measure the overall performance of a model by averaging and weighting other benchmark scores to get a overall number representing the model. A popular example is the Artificial analysis score.
Problems
Never trust a 100%
Most benchmarks contains all kinds of errors: questions without an objective correct answer, bad judges, a mislabeled answer key, an unsolvable task, or just a typo. You will often see that even benchmarks that are considered “solved” and easy only get high scores around 95%-98%. If you look at the remaining wrong answers you often find a mistake in the benchmark rather than a genuine mistake by the model. This is why a perfect score on a benchmark can be an indicator that the model did not solve the benchmark but just learned all the “correct” answers. This can happen in many different ways and we will look at it now.
Goodhart’s law
Benchmarks have become so popular and relevant for the success, and especially the economic success of companies and their models that there is a huge incentive to do whatever it takes to perform well on these benchmarks. It is scientific standard to test AI Models on different data than they where trained on since we are generally interested in how they perform on new problems and not how well they remember their training data. This means that labs should spend some time to ensure that the model does not see the questions and answers of benchmarks in their training data. For nearly all labs their training data is not known, but I assume that most of them at least try to filter for evaluation data and benchmarks. Given the size of these datasets this can become quite complex. Simply searching for matching strings is not enough since a benchmark question could be written in another language for example. One benchmark that shows this contamination quite well is GSM8K. In 2024 GSM1k was created in a similar style but with new questions and scores deviated greatly suggesting contamination of the training data. There are also entire classes of benchmarks that are not just a set of questions and answers. Lets take ChessBench as a elo-based benchmark again; Since the score just depends how well the model plays chess against other models the performance can depend a lot on the amount of chess data in the training set. Obviously the Benchmark is not a reason for labs to filter out all chess data in their collection. You can even argue good chess performance is desirably. So how do we judge the scores of the models on this benchmark? The gpt-3.5-turbo-instruct model was far better than other LLMs at the time and even outperformed models that are quite a bit newer. Does this mean the model is better? The simple answer is that you should pay attention to the scope of a benchmark. ChessBench tells you how well models play chess (, against each other) and nothing more. A lab that wants to perform well on this benchmark has another option besides training on more chess data.
On the older benchmark the two models are 0.3 points apart — inside the run-to-run noise of most benchmarks. On the harder version the gap is 32.7 points. A clear sign of a model overfitting to an older version.
Show the numbers
| Model | Terminal-Bench 2.1 | Terminal-Bench 4.0 |
|---|---|---|
| Gemini 3.8 Flash | 89.4 | 19.1 |
| Claude Opus 5 | 89.1 | 51.8 |
Picking and Choosing
While training on the test data is considered bad practice there is a grey zone that labs tend to do quite a lot: Choosing based on scores. As a short explanation how ML tended to solve this in the past: You would train different models on the training data, see their scores on a validation set, and than choose the model based on the validation score and test it on a third test set. The separation between validation and test set has the purpose of avoiding choosing a model based on the final score that you want to maximize, precisely because of Goodhart's law. In the modern era of LLMs this separation is somewhat dropped. A good example is another elo-based benchmark that I mentioned earlier: LM Arena. Especially 1-2 years ago LM Arena was one of the most influential benchmarks and labs started testing their models before the release. But they did not just test one model that they wanted to release, they tested multiple versions. This became a controversy when Meta released a different version than the one getting the LM arena score they reported. But looking at it from the Goodhart lens the inverse is even worse. The labs using the scores to choose the version to release basically used the test data as the validation data and this lead to a indirect optimization towards models that get high scores at LM arena. Just as Goodhart's law says, when the measure of how people rank the output of the model compared to others, became the target it stopped working as a measure. The models started to produce very long very well formatted answers since these tended to win in side by side comparisons. Selection is a strong driver as we well know from evolution. So choosing a model based on the score on this or other benchmarks is another common mistake that both the labs and you should be aware of. This is getting even worse when we not only use the score of different checkpoints or versions to choose, but directly use the benchmark result as a training target. This is openly done as we were recently able to see live as Xiaomi shared a live view of their newest post-training run where they are tracking benchmark scores over the training as a direct target to hit.
Awareness
A combination of the growth in size and capabilities of the models, and the introduction of content about LLMs and their training and evaluation into the training data, is leading to a awareness of the models about their own life cycle. They are increasingly aware of the fact they are being tested. This leads to some interesting behavior which makes certain types of evaluation quite difficult. Imagine testing a models morals or ethical orientation while the model is aware of the fact and also knows that the score might impact its deployment. The same goes for other safety related tests like dangerous capabilities where models can sandbag and perform worse on purpose to appear less dangerous. These things are already happening and as Anthropic reports the models also hide this sometimes. Awareness also impacts how models cheat. In the Huggingface incident, the agents found a shortcut to the answer within hours, but they assumed any good benchmark would include a check for cheating. They then spent days trying to fake legitimate solutions, even though the grader only checked whether the final code was correct. This can further escalate in a related problem: Sandboxing.
Sandboxing
Many modern benchmarks do not just test on text-based tasks but also benchmark agent capabilities which include tool use. This in turn requires more complex environments that the agents can interact with and work in, including a virtual machine, terminal access, internet access, or other sets of tools. The solution for this are sandboxes that provide the needed content to the model while keeping it isolated from other resources that it is not supposed to access. As the recent Huggingface incident shows, this can fail because of the increase in hacking capabilities and the awareness I mentioned before. So not only can models potentially cheat by breaking the environment in some way, but running the benchmark itself becomes a possible risk for the external infrastructure if the model breaks out. Since this gets conflated sometimes I want to briefly explain reward hacking and how it differs from what is happening in this scenario. Reward hacking is a classic problem in reinforcement learning, where the agent that is trained to maximize a reward, does so in a way unintended by the developer who designed the reward function.
This seems similar to a language model breaking out of a sandbox to get a good score on a benchmark but it is in fact not a case of reward hacking since the model does not get a gradient update based on the result. There are however cases that look very similar during RL training where the models perform similar tasks in a sandbox where they get a reward which can lead to this behavior. So in the Huggingface incident we had a first phase where the models were trained and they performed reward hacking to find a way to communicate, and later the same models applied the learned behavior from the reward hacking in the evaluation setting.
How hard can it be?
A problem that the models can run into is finding hard enough problems. While some domains are easy to benchmark, others can be quite expensive or hard to build. Imagine having to test wet lab performance of a model. But quite recently there are some domains where the problem is no longer constructing the benchmark but coming up with problems at all. We now see this in math where new benchmarks are not able to find problems that are difficult enough to not be saturated from the start. This happened in the past for specialized models like digit recognition and other simple tasks. Usually we hit a ceiling and consider the problem solved, but for math specifically this is hard to argue. So instead of testing on problems that we know the solution and just check if the model gets it, recent benchmarks started asking open questions without known answers that are able to be verified. This works but runs into problems in domains where a formal verification is not possible. This is also a potential problem for training but this is not in the scope of this post.
Biased Judges
As mentioned, some benchmarks deploy judges in the form of other LLMs or humans to compare or rate results. This method is subject to an entire set of biases that are well studied and impact the results. Some common ones are position bias where the order of shown results can change how they are perceived, authorship bias where both humans and AI models rate certain models higher (especially AIs rating their own output), and verbosity bias where longer outputs are generally preferred. Some of these can be counteracted, but overall relying on judges always introduces a weak point into your methodology.
Examples
So now lets take a look at some popular examples that cover a range of problems and categories. We will look at their advantages but also at the potential problems and how well they reflect the actual model quality.
ARC AGI
ARC AGI is a series of benchmarks that started in 2019 and it consists of grid based tasks that give a few example pairs and requires the model to complete a final pair. The benchmark is designed based on the principle "Easy for Humans, Hard for AI". There is a large set of public examples that are designed for training, a public eval set and 200 private tasks to prevent training on the test set. As expected the public eval set is considered to be not representative since it leaked into many public training sets for LLMs. The second version of the benchmark follows the same pattern but is harder, but the scores follow a similar pattern even though the cost per task curves are shifted and it took a bit longer. The third version switches the format and uses small grid based games that require learning the controls and mechanics over multiple levels and judges performance based on the number of needed moves. This version was only recently "solved" but at a significant cost per task leaving room for improvement in efficiency. The fourth version is supposed to come out soon. The stated goal is to continue releasing versions until they can no longer find tasks that are easy for humans but hard for AI. I personally find the benchmark quite interesting since it tests a kind of fluid intelligence and often matches my impression of the model performance. I think the biggest criticism I have is the way scores are done for LLMs. In the original design of the benchmark users had to submit a model that were tested on a predefined hardware. This was changed since frontier LLMs are not open source and sharing the model was out of the question and the needed compute also exceeded the one provided by the maintainers. So the Initial idea of efficiency got lost and we can see this now with costs in the tens of thousands. The efficiency is still better than brute force but a limit on compute costs would make the scores more meaningful. And last but not least we cannot forget that the "private" set for the LLM leaderboard is just semi-privat since it needs to be send to the AI labs over their API. This, in theory, would allow the labs to get access to the data and train on them, even though I do not believe this is done.
LM Arena
Despite the problems of this benchmark that I mentioned earlier it has a few advantages over other benchmarks. The core principle of having humans choose between two different model outputs is quite close to how the models actually perform in the real world. The leaderboard also offers additional features such as confidence intervals, and separate leaderboards based on the type of question. Besides the problems with models learning to make output more appealing to humans, there is also the problem that this format limits interactions to one or two messages since the two parallel conversations drift apart and you can only send a message to both. LM Arena tried to address the formatting preference by introducing style control, but the general problem of bias in the judge persists.
Epoch Capabilities Index (ECI)
The ECI is a Meta benchmark by Epoch AI. While less known than the Artificial analysis score, this meta benchmark has the advantage that it addresses another major concern: Saturation and difficulty. Each meta benchmark has to decide how it weights different benchmark scores based on their difficulty, relevance, and how flawed they are. Especially at the extremes benchmark scores have little value. If the benchmark is too hard, scores are very low and a single lucky point can make the difference between models appear quite large, while a simple benchmark that is essentially solved does not allow better models to distinguish themself either. In general the performance on a new benchmark follows an S-curve where the time where the scores are distributed in the middle is the most informative.
ECI uses this by rating benchmarks based on difficulty and the range of difficulty. This makes the final score, while still dependent on the quality of the benchmarks, quite a good representation of the overall performance. I really like that the website offers the option to choose your own combination of benchmarks.
Humanity's last Exam (HLE)
HLE has quite the eye catching name and was designed in 2025 as a set of very hard questions in different areas that came from a wide variety of subject experts around the world. As one of the few pure question answer benchmarks which are not yet saturated it became quite popular and most model releases contain a HLE score. The format is a mix of fixed answer and judge based where the model provides an answer and a judge model compares it against the reference answer. Epoch AI that did the previous benchmark took a closer look at the benchmark and found mistakes in a significant amount of the questions. Given this context, looking at the minor score differences between frontier models who are currently getting scores around 50% seems almost meaningless.
Leave a Reply