RSI Index

Fable 5.1 organized its campaigns around progressively stronger qualification. Compression candidates were screened cheaply, tested on larger inputs, checked for exact reconstruction, and finally run on the full corpus. The selected compressor completed 16 recorded round-trip measurements and produced both the best enwik8 artifact, at 1.3560 BPB, and a successful full-enwik9 result. In LM Training, 47 development trials produced a submission with a 0.8536 median hidden BPB, compared with the published 0.85 BPB reference.

The same process produced a second-place Parameter Golf result and the leading Harness Engineering result. Its Parameter Golf submission averaged 1.1382 BPB across three clean-room retrainings, beating 14 of 47 published human entries, while its Harness submission reached 81.30% agreement. These results did not come from experiment volume alone: Fable repeatedly increased the strength of the test a candidate had to pass before promotion.

Post-Training provides the clearest illustration. Fable trained two LoRA candidates but rejected both when improvements in tool-use behavior failed to produce better answers. Its final artifact retained the parent weights and paired them with finance-specific prompt guidance, improving the official score from the 35.38% parent baseline to 39.99%. Its distinctive strength was not a new training algorithm, but refusing to equate successful training with a better model.

Opus 5 tended to identify one central bottleneck and reorganize the campaign around it. In Post-Training, it traced the parent model's failures to a concrete tool-use error: approximately 84% of its early retrieve_information calls omitted a required argument. It generated targeted training examples for that behavior, reducing the error rate from 84% to 35% in an initial probe and raising the submission rate from 71% to 100%. Its final checkpoint achieved 42.20%, the strongest Post-Training result.

The same willingness to commit appeared elsewhere. Its Parameter Golf submission used a comparatively ambitious 30-million-parameter model designed around aggressive low-bit compression, rather than staying close to the starter architecture. In LM Training, it moved promising candidates into longer runs and replicated its finalists six times before submission.

This concentration can also become a weakness. Opus spent much of Compression developing an increasingly elaborate family of context models, even after marginal gains had become small. Its own retrospective argued that it should have switched earlier to an online neural approach. Opus is strongest when its diagnosis is correct; when it is wrong, it can spend most of the budget refining the wrong research program.

Astra's strongest work combined several complementary techniques into a better final system. In Parameter Golf, it brought together low-bit training, adaptation during prediction and statistical count models, screening components cheaply before testing complete configurations. Its submission reached 1.1176 BPB, ahead of Sol's 1.1883 and the strongest result among RSI agents. Compression and LM Training also improved on Sol, with the submitted LM recipe reaching 0.8607 BPB.

Its experimentation was less consistent when choosing among late candidates. In LM Training, the selected configuration received only one full-length development run, leaving limited evidence of its repeatability before submission. In Harness Engineering, Astra explored browser-based and source-based judges before selecting a source-code approach. That candidate looked stronger in development but transferred poorly to new tasks, finishing below Sol.

Post-Training followed a more elaborate training process without a comparable payoff. Astra fine-tuned checkpoints for financial reasoning and tool use, compared multiple candidates, blended selected weight updates, and revised its prompting and generation settings. Narrow arithmetic and modeling exercises improved, but the final result barely moved: 35.46% against the parent model's 35.38%. Its distinctive strength was combining useful techniques; its recurring limitation was choosing development tests that reliably predicted improvement on the complete task.

GPT-5.6 invested more heavily than the other systems in instrumentation and controlled comparison. In Harness Engineering, it combined source inspection, browser execution, screenshots, DOM evidence, persistent state, and deterministic rules for failures it found the judge model handled unreliably. It changed mechanisms separately, discovered errors in its own evaluation code, and abandoned source-only, browser-only, and additional-judge designs when broader tests contradicted their apparent gains. The resulting harness reached 77.89% agreement.

Its LM Training campaign contained 114 recorded trials, but finished seventh at 0.9475 BPB. Parameter Golf showed a similar pattern: 84 recorded experiments and decisions made the campaign unusually auditable, but many were narrow analyses of existing checkpoints, and the final 1.1883 BPB submission remained relatively close to the starter design. The final LM candidate also received less full-duration confirmation than the campaign's overall experimental rigor would suggest.

GPT-5.6's limitation was therefore not a lack of experimentation, but the conversion of evidence into commitment. It was particularly good at constructing measurement systems, isolating variables, and detecting false gains, yet its many local comparisons did not always produce a correspondingly strong global change to the final artifact.

Kimi K3 repeatedly treated the best confirmed artifact as something that should not be displaced by an uncertain gain. In LM Training, it selected a reliable candidate that could finish within the deadline rather than the configuration with the best isolated result. In Harness Engineering, identical artifacts produced unexpectedly different scores, leading it to replace peak-score selection with repeated means and minimum-score thresholds. In Post-Training, every trained child failed its comparison against the parent, so Kimi restored the unchanged parent rather than submit a model it believed had regressed.

This caution did not prevent rapid experimentation. Kimi moved through concrete architectural and training changes quickly, usually retaining a strong incumbent for comparison. Parameter Golf shows why persistent execution mattered: after an earlier session had effectively completed, a later session reconsidered the evaluation procedure and found an overlooked evaluation-length change. That discovery improved its development result across repeated runs from approximately 1.1908 to 1.1803 BPB; the frozen submission subsequently achieved 1.1754 BPB under official evaluation.

Kimi was therefore conservative about what it shipped, not about what it tried. Its weakness was premature closure: it sometimes considered a campaign finished before exhausting useful ideas, then found improvements when it resumed experimentation.

Qwen's most distinctive behavior appeared when it stopped assuming that evaluation results were trustworthy. In Harness Engineering, apparently improved harnesses sometimes produced inconsistent scores. Qwen responded by interleaving candidate evaluations with byte-identical control submissions, allowing it to estimate how much variation came from the evaluator rather than the harness. Across 45 evaluator runs, it used these controls and broader application panels to reject several changes that initially appeared beneficial. Its final artifact scored 73.17%, 74.63%, and 70.73% across three complete held-out evaluations, for a mean of 72.85%.

This reflects Qwen's broader incremental style. It generally preserved a working system, changed a small number of components, and allowed controlled comparisons to determine the next step. Its 19-experiment Parameter Golf campaign combined a faster attention backend, larger training batches, value residuals, a wider int6 MLP, and causal score-first evaluation adaptation into a compliant 1.2070 BPB submission.

The tradeoff is that local discipline does not guarantee a strong final direction. Its Post-Training artifact regressed to 32.07%, and some final candidates received less replication than its Harness methodology would imply. Qwen was unusually willing to challenge the meaning of its own measurements, but applied that skepticism more consistently in some campaigns than others.

Gemini 3.8 Flash concentrated on making working implementations and refining them through repeated experiments. In Parameter Golf, it retained the starter architecture and tuned optimizer settings, training schedules and adaptation during prediction, reaching 1.2198 BPB from a 1.2288 baseline. Its Compression campaign refined a context-mixing design to a final result of 1.7261 BPB on enwik8. Both efforts produced improvements, but stayed behind the stronger submissions.

LM Training explored broader architectural changes and repeatedly tested finalists, producing a submission that reached 0.9468 BPB. It selected its final candidate based on the strongest individual development result. Harness Engineering showed practical debugging ability: Flash built browser-based judges that used page structure and screenshots, repaired action handling, and maintained memory across steps. It repeatedly revised and tested these systems rather than stopping at the first working implementation.

Post-Training exposed the largest weakness in its experimental process. Flash expanded 242 scripted examples into 2,500 training trajectories through repetition, then selected a checkpoint using simulated tool interactions and successful completion. Those checks provided little evidence that the model could answer real financial questions accurately, and the submitted model fell to 3.54% from the parent's 35.38%. Its recurring limitation was treating a functioning pipeline or a promising individual result as sufficient evidence to select a candidate.