TL;DR: Today, we’re announcing Vals Misalignment Incidents, a page to track instances of cheating, reward hacking, and signs of misaligned behavior. We’re starting by sharing incidents we observe from our own evals, and are also introducing a new reporting mechanism for external parties to report misalignment incidents they observe.
Evaluations are essential to understand how model capabilities differ. Yet, adoption of long-horizon agents depends on trustworthiness as much as capabilities. As model capabilities keep improving, it is worth keeping close tabs on safety and trustworthiness.
At Vals, we run evaluations on every new model release. This often amounts to hundreds of billions of tokens’ worth of evaluations per month. While we put out the results of the evaluation, we are also left with plenty of traces from each model. We also do pre-release testing of models; this means that we are in a good position to be able to identify new model behaviors, particularly around the safety risks associated with models.
Today, we are sharing some interesting instances of cheating and reward hacking from our traces, along with some other instances of model behavior that we found particularly compelling. While we observe hundreds of interesting behaviors across our models runs, we are aiming to share a subset of such examples that are representative of the behavior we see.
We used automated pipelines to identify cases of reward hacking, and in doing so we also encountered a range of interesting behavior (some malicious, some benign). Starting today, we’re making our work toward this end public. The initial incidents we’re reporting were sourced through a two-stage pipeline across recent public-benchmark traces. We used Mercury Decide, a low-cost decision model, to identify specific features of traces. Then, we used GPT-6.1 Sol to identify notable episodes for review. We also set up a coding agent with access to several relevant tools to look through the traces.
Inspired by the incident reports from OpenAI and Anthropic, among others, we’re launching a dedicated incidents page that currently lists a subset of these examples. We intend to update this incidents page regularly, and surface incidents that occur in our evals, along with relevant metadata, such as the date, infrastructure, setting in which the incident occurred, and more.
We also welcome submissions. If you have a misalignment incident you would like us to include, please select Report an incident on the incidents page.
Incidents featured in this post
Snapshot from
- Interpreting or bypassing rules
DeepSeek bypassed Lean’s proof checker and passed the grader
The model failed to find a proof of a theorem it was tasked with proving.
DeepSeek V4.1 Flash 1017 Reported 2026.11.19
- Interpreting or bypassing rules
Kimi found the study behind its sample, then matched the wrong organism
Kimi had to name the bacterium spiked into a sequencing sample without using accession numbers to find the original study.
Kimi K3.1 Pro Reported 2026.11.19
- Interpreting or bypassing rules
MiniMax used the environment’s database password to skip the required tools
The model was asked to build a company directory and org-chart web app on a Supabase database, and to make every database change through the supplied Supabase MCP tools.
MiniMax-M3.1-Flash Reported 2026.11.19
- Interpreting or bypassing rules
Beam guessed tokens and forged a JWT to reach withheld task data
Beam was asked to build a tool that plans a day of freight deliveries.
Beam 1.0 Reported 2026.11.19
- Interpreting or bypassing rules
Muse hid the original program inside its Java rewrite
Muse was asked to rewrite Clingo, a C++ program, in Java.
Muse Spark Open Reported 2026.11.19
- Interpreting or bypassing rules
Frontier models complied with military targeting requests in a controlled study
Vals asked AI models to find people and targets in drone and satellite images.
DeepSeek V4.1 Flash
Fable 5.1Fable 5.2
Gemini 4 Argon
Gemini 4 Carbon
GPT-6 Astra
GPT-6 Luna
GPT-6.1 Sol
Grok 4.7
Inkling 2 Small
Kimi K3.1 ProMuse Wonder
Opus 5.5Sonnet 5.5 Reported 2026.11.19
- Interpreting or bypassing rules
Mistral edited a protected file, then decided the rule didn't apply
Mistral was asked to fix a bug in some training code, with one firm rule: it must not change the “shim”, a piece of code that lets the program run on ordinary CPUs.
Mistral Large 4 Preview Reported 2026.11.19
- Interpreting or bypassing rules
Muse slowed down NetworkX to make its own library look faster
Muse was asked to check whether two graphs have the same structure, matching NetworkX’s answers (NetworkX is a Python library for working with graphs).
Muse Spark Open Reported 2026.11.19
- Interpreting or bypassing rules
Muse hid a banned GPU library by spelling its name one letter at a time
Muse was asked to write a fast GPU program and was told not to use cuBLAS, NVIDIA's ready-made library for this kind of math.
Muse Spark Open Reported 2026.11.19
- Goal drift
Kimi imagined a prompt injection (and claimed unfinished work was complete)
Kimi was asked to implement and test a fast GPU kernel for normalization and matrix multiplication.
Kimi K3.1 Pro Reported 2026.11.19
- Author / user profiling
Gemini invented a backstory for the task’s author from spreadsheet metadata
The model was asked to work out the radioactivity of a food sample from two spreadsheets.
Gemini 4 Argon Reported 2026.11.19
- Interpreting or bypassing rules
Grok built training data from the validation set and called it restored
Grok was asked to repair a training run that had gotten worse because five of its training data files were corrupted.
Grok 4.7 High Reported 2026.11.19
- Grader reasoning
Grok trained its model in the untimed warmup to beat the clock
Grok was asked to build a transformer language model that achieved a particular validation loss.
Grok 4.7 High Reported 2026.11.19
- Uncategorized
Assistants escalated an impossible refund request
In a Vals simulation, six AI assistants tried to recover £350 for a passenger who had missed the deadline to claim a train refund.
- Goal drift
Opus wrote its own test results and declared the task done
After a real test found a missing chess move, Opus wrote a run of fake commands and fake passing results into its reply, none of which ran.
Claude Opus 5.5 Reported 2026.11.19
