The MysteryMechanism Evaluation tests whether agents are capable of discovering hidden mathematical relationships via experimentation. This task is designed to simulate the context of scientific discovery, where noisy, expensive experiments are used to derive mechanical relationships.
Evaluation
MysteryMechanism tests whether an agent can rediscover an unknown mathematical
relationship through active experimentation. The task provides anonymous input
variables, physical bounds, two passive observations, a persistent shell, and a
budget of 2d+1 experiments for a d-dimensional mechanism. The agent must
submit one executable law. It receives no domain context, internet access, or
information about the particular mechanism.
The benchmark contains 222 mechanisms spanning biological, physical, chemical, engineering, ecological, and abstract dynamical relationships.
Public examples
These three frozen examples expose the exact task shape without revealing any validation or test mechanism. Each tab pairs the agent-visible question with an illustrative successful path constructed from that public task. The paths are condensed walkthroughs—not verbatim transcripts from a particular model.
Recover an executable mathematical expression mapping the anonymous inputs to one measured output.
Two noisy passive observations · 5 additional experiments · no domain context
- x1 ∈ [0.05, 0.70]
- x2 ∈ [0.05, 30.0]
- (0.5944, 1.9188) → 12.2245
- (0.5044, 0.2068) → 10.3732
- 1 Design
Probe the four corners, then the interior. The corner ratios test whether the two inputs act approximately as separable factors.
- 2 Run experiments(0.05, 0.05) → 4.3715 (0.70, 0.05) → 30.6686 (0.05, 30.0) → 2.3912 (0.70, 30.0) → 16.3743 (0.375, 1.225) → 5.7241
- 3 Infer and fit
The x1 response steepens near its upper bound, while the x2 effect decays with a square-root shape. Fit the compact multiplicative family.
- 4 SubmitAccepted
Nonlinear two-factor response
mu*(1+a*x1/(1-x1)**b)*(1+c/(1+sqrt(x2)))mu=1.7707 · a=0.9128 · b=1.9124 · c=1.6447
structural NMSE 0.000270 ≤ threshold 0.000697
Recover an executable mathematical expression mapping the anonymous inputs to one measured output.
Two noisy passive observations · 5 additional experiments · no domain context
- x1 ∈ [1, 300,000]
- x2 ∈ [0.2, 10.0]
- (63.5657, 1.2799) → 3.3766
- (4,615.6906, 5.3388) → 76.4846
- 1 Design
Use log-scale corners and a geometric midpoint. This exposes the leading square-root scaling and the high-x1 correction despite noisy low-signal readings.
- 2 Run experiments(1, 0.2) → 0.1004 (300,000, 0.2) → 277.5832 (1, 10.0) → −1.1800 (300,000, 10.0) → 1,263.4885 (547.7226, 1.4142) → 13.5045
- 3 Infer and fit
The response follows a fixed dimensionless correlation: a square-root x1 term, a fractional-power x2 correction, and a second high-x1 factor.
- 4 SubmitAccepted
Corrected fractional-power correlation
scale*(0.3+0.62*x1**0.5*x2**(1/3)/(1+(0.4/x2)**(2/3))**0.25*(1+(x1/282000)**(5/8))**(4/5))scale=1.0000
structural NMSE 0 ≤ threshold 0.00000001
Recover an executable mathematical expression mapping the anonymous inputs to one measured output.
Two noisy passive observations · 5 additional experiments · no domain context
- x1 ∈ [0.1, 10,000]
- x2 ∈ [0.001, 0.5]
- (25.5646, 0.1859) → 0.0012396
- (0.1108, 0.0187) → −0.0000064
- 1 Design
Choose a high-signal rectangle on logarithmic scales. Paired ratios isolate one exponent at a time without relying on the near-zero passive sample.
- 2 Run experiments(100, 0.05) → 0.0008841 (10,000, 0.05) → 0.0082336 (100, 0.5) → 0.0049850 (10,000, 0.5) → 0.0454408 (1,000, 0.2) → 0.0078693
- 3 Infer and fit
The paired log slopes are stable, supporting a two-input power law. A log-linear fit estimates both exponents and the scale.
- 4 SubmitAccepted
Two-input power law
k*x1**m*x2**nk=0.0009114 · m=0.4822 · n=0.7478
structural NMSE 0.000123 ≤ threshold 0.000373
Scoring
The primary metric is accuracy of functional recovery. The submitted expression is executed on private, continuously sampled structural probes and receives binary credit when normalized error is below the task’s noise-aware threshold. This deliberately rewards functional equivalence rather than matching the hidden mechanism’s name or symbolic spelling.
Provider or agent failures count as zero.
How models are successful
- Mechanism rediscovery: recognize a compact family such as a saturating response, choose interventions that isolate its terms, fit its parameters, and submit the recovered law.
- Mathematical rediscovery without domain recognition: test invariances and transformations, compare low-complexity candidate families, and recover a functionally equivalent law without naming the source domain.
In a separate blinded, post-hoc audit, we recorded whether each successful trace explicitly named the correct source domain. “Domain not identified” means the trace did not name the correct domain; it does not prove that prior scientific knowledge played no role in the model’s reasoning.
| Model | Accuracy | Successful traces | Domain not identified | Identified domain |
|---|---|---|---|---|
| GPT-6 Astra | 53.15% | 118 | 89.8% | 10.2% |
| Claude Fable 5.1 | 47.75% | 106 | 74.5% | 25.5% |
| Claude Opus 5 | 37.39% | 83 | 79.5% | 20.5% |
| Gemini 3.8 Flash | 36.49% | 81 | 66.7% | 33.3% |
| Muse Spark 1.3 Max | 36.04% | 80 | 92.5% | 7.5% |
| GPT-5.6 Sol | 33.33% | 74 | 89.2% | 10.8% |
| Grok 4.6 | 30.63% | 68 | 82.4% | 17.6% |
| DeepSeek Flash 4.1 | 21.17% | 47 | 97.9% | 2.1% |
| GPT-5.6 Luna | 14.41% | 32 | 100.0% | 0.0% |