Vals RSI Index

Fable 5.1 organized its campaigns around progressively stronger qualification. Compression candidates were screened cheaply, tested on larger inputs, checked for exact reconstruction, and finally run on the full corpus. The selected compressor completed 16 recorded round-trip measurements and produced both its best enwik8 artifact, at 1.3560 BPB, and a successful full-enwik9 result. In LM Training, 47 development trials produced a submission with a 0.8536 median hidden BPB, compared with the published 0.85 BPB reference.

The same process produced 81.30% agreement in Harness Engineering. That result did not come from experiment volume alone: Fable repeatedly increased the strength of the test a candidate had to pass before promotion.

Post-Training provides the clearest illustration. Fable trained two LoRA candidates but rejected both when improvements in tool-use behavior failed to produce better answers. Its final artifact retained the parent weights and paired them with finance-specific prompt guidance, improving the score from the 35.38% parent baseline to 39.99%. Its distinctive strength was not a new training algorithm, but refusing to equate successful training with a better model.

Opus 5 tended to identify one central bottleneck and reorganize the campaign around it. In Post-Training, it traced the parent model's failures to a concrete tool-use error: approximately 84% of its early retrieve_information calls omitted a required argument. It generated targeted training examples for that behavior, reducing the error rate from 84% to 35% in an initial probe and raising the submission rate from 71% to 100%. Its final checkpoint achieved 42.20%.

The same willingness to commit appeared elsewhere. In LM Training, it moved promising candidates into longer runs and replicated its finalists six times before submission.

This concentration can also become a weakness. Opus spent much of Compression developing an increasingly elaborate family of context models, even after marginal gains had become small. Its own retrospective argued that it should have switched earlier to an online neural approach. Opus is strongest when its diagnosis is correct; when it is wrong, it can spend most of the budget refining the wrong research program.

Astra's strongest work combined several complementary techniques into a better final system. Its Compression and LM Training submissions both improved on GPT-5.6 Sol, with the submitted LM recipe reaching 0.8607 BPB.

Its experimentation was less consistent when choosing among late candidates. In LM Training, the selected configuration received only one full-length development run, leaving limited evidence of its repeatability before submission. In Harness Engineering, Astra explored browser-based and source-based judges before selecting a source-code approach. That candidate looked stronger in development but transferred poorly to new tasks, finishing below GPT-5.6 Sol.

Post-Training followed a more elaborate training process without a comparable payoff. Astra fine-tuned checkpoints for financial reasoning and tool use, compared multiple candidates, blended selected weight updates, and revised its prompting and generation settings. Narrow arithmetic and modeling exercises improved, but the final result barely moved: 35.46% against the parent model's 35.38%. Its distinctive strength was combining useful techniques; its recurring limitation was choosing development tests that reliably predicted improvement on the complete task.

GPT-5.6 invested more heavily than the other systems in instrumentation and controlled comparison. In Harness Engineering, it combined source inspection, browser execution, screenshots, DOM evidence, persistent state, and deterministic rules for failures it found the judge model handled unreliably. It changed mechanisms separately, discovered errors in its own evaluation code, and abandoned source-only, browser-only, and additional-judge designs when broader tests contradicted their apparent gains. The resulting harness reached 77.89% agreement.

Its LM Training campaign contained 114 recorded trials, but its result trailed most models. The final LM candidate also received less full-duration confirmation than the campaign's overall experimental rigor would suggest.

GPT-5.6's limitation was therefore not a lack of experimentation, but the conversion of evidence into commitment. It was particularly good at constructing measurement systems, isolating variables, and detecting false gains, yet its many local comparisons did not always produce a correspondingly strong global change to the final artifact.

Qwen's most distinctive behavior appeared when it stopped assuming that evaluation results were trustworthy. In Harness Engineering, apparently improved harnesses sometimes produced inconsistent scores. Qwen responded by interleaving candidate evaluations with byte-identical control submissions, allowing it to estimate how much variation came from the evaluator rather than the harness. Across 45 evaluator runs, it used these controls and broader application panels to reject several changes that initially appeared beneficial. Its final artifact scored 73.17%, 74.63%, and 70.73% across three complete held-out evaluations, for a mean of 72.85%.

This reflects Qwen's broader incremental style. It generally preserved a working system, changed a small number of components, and allowed controlled comparisons to determine the next step.

The tradeoff is that local discipline does not guarantee a strong final direction. Its Post-Training artifact regressed to 32.07%, and some final candidates received less replication than its Harness methodology would imply. Qwen was unusually willing to challenge the meaning of its own measurements, but applied that skepticism more consistently in some campaigns than others.

Kimi K3 repeatedly treated the best confirmed artifact as something that should not be displaced by an uncertain gain. In LM Training, it selected a reliable candidate that could finish within the deadline rather than the configuration with the best isolated result. In Harness Engineering, identical artifacts produced unexpectedly different scores, leading it to replace peak-score selection with repeated means and minimum-score thresholds. In Post-Training, every trained child failed its comparison against the parent, so Kimi restored the unchanged parent rather than submit a model it believed had regressed.

This caution did not prevent rapid experimentation. Kimi moved through concrete architectural and training changes quickly, usually retaining a strong incumbent for comparison.

Kimi was therefore conservative about what it shipped, not about what it tried. Its weakness was premature closure: it sometimes considered a campaign finished before exhausting useful ideas, then found improvements when it resumed experimentation.

Gemini 3.8 Flash concentrated on making working implementations and refining them through repeated experiments. Its Compression campaign refined a context-mixing design to a final result of 1.7261 BPB on enwik8, an improvement that stayed behind the stronger submissions.

LM Training explored broader architectural changes and repeatedly tested finalists, producing a submission that reached 0.9468 BPB. It selected its final candidate based on the strongest individual development result. Harness Engineering showed practical debugging ability: Flash built browser-based judges that used page structure and screenshots, repaired action handling, and maintained memory across steps. It repeatedly revised and tested these systems rather than stopping at the first working implementation.

Post-Training exposed the largest weakness in its experimental process. Flash expanded 242 scripted examples into 2,500 training trajectories through repetition, then selected a checkpoint using simulated tool interactions and successful completion. Those checks provided little evidence that the model could answer real financial questions accurately, and the submitted model fell to 3.54% from the parent's 35.38%. Its recurring limitation was treating a functioning pipeline or a promising individual result as sufficient evidence to select a candidate.