suitedaces/dorabot
Languages
macOS app for 24/7 AI agents in an IDE with memory, scheduled tasks, browser use + access to Whatsapp, Telegram, Slack.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 20 / 30 | $0.52 | $1.25/$4.25 | 3m43s | ||
2 | Mini-SWE-agent | 19 / 30 | $0.03 | $0.44/$1.32 | 3m34s | ||
3 | Mini-SWE-agent | 19 / 30 | $0.47 | $1.4/$4.4 | 7m50s | ||
4 | Mini-SWE-agent | 19 / 30 | $2.37 | $5/$25 | 8m22s |
Key Takeaways
- Deepseek V4 Flash 0731 with Mini-SWE-agent matches the 19-task results at the lowest cost per test, $0.03, and lowest latency, 214.01 seconds.
- Muse Spark 1.2 with Mini-SWE-agent scores 66.67%, compared with 63.33% for each of the three 19-task results.
Model Comparison
Accuracy
66.67%
Muse Spark 1.2
63.33%
DeepSeek V4 Flash 0731
Task outcomes
30 tasks
Cost / test
$0.52
Muse Spark 1.2
$0.03
DeepSeek V4 Flash 0731
Cost distribution
Latency
3m 43s
Muse Spark 1.2
3m 34s
DeepSeek V4 Flash 0731
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Muse Spark 1.2 leads at 66.67% for $0.52 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 63.33% for $0.03 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Muse Spark 1.2 leads at 66.67%, while DeepSeek V4 Flash 0731 is fastest at 3m 34s with 63.33% accuracy.
Tasks with failures
| Models | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Spark 1.2 | ||||||||||||
| DeepSeek V4 Flash 0731 | ||||||||||||
| GLM 5.2 | ||||||||||||
| Claude Opus 4.8 |
Task detail
6ce6383Issue statement
Update the desktop Codex model catalog for the GPT-5.6 family. GPT-5.6 Terra should be the default Codex model and the catalog should offer Sol, Terra, and Luna with their user-facing names, default reasoning effort, and supported effort levels. Sol defaults to low and supports low, medium, high, xhigh, max, and ultra; Terra defaults to medium and supports the same levels; Luna defaults to medium and supports those levels except ultra. Mark Terra as the catalog default. The reasoning-effort option helper must preserve max and ultra when a model advertises them. Also mark gpt-5.3-codex as deprecated because it is no longer bundled.
View Hidden Tests
diff --git a/valsmith_tests/model_catalog.test.mjs b/valsmith_tests/model_catalog.test.mjsnew file mode 100644index 0000000..ff4de9a--- /dev/null+++ b/valsmith_tests/model_catalog.test.mjs@@ -0,0 +1,45 @@+import assert from 'node:assert/strict';+import {+ CODEX_MODELS,+ DEFAULT_CODEX_MODEL,+ codexReasoningEffortOptions,+} from '../desktop/src/lib/modelCatalog.ts';++export function testCodex56Catalog() {+ assert.equal(DEFAULT_CODEX_MODEL, 'gpt-5.6-terra');++ const byId = new Map(CODEX_MODELS.map((model) => [model.value, model]));+ const expected = {+ 'gpt-5.6-sol': {+ label: 'GPT-5.6 Sol',+ defaultReasoningEffort: 'low',+ supportedReasoningEfforts: ['low', 'medium', 'high', 'xhigh', 'max', 'ultra'],+ },+ 'gpt-5.6-terra': {+ label: 'GPT-5.6 Terra',+ defaultReasoningEffort: 'medium',+ supportedReasoningEfforts: ['low', 'medium', 'high', 'xhigh', 'max', 'ultra'],+ isDefault: true,+ },+ 'gpt-5.6-luna': {+ label: 'GPT-5.6 Luna',+ defaultReasoningEffort: 'medium',+ supportedReasoningEfforts: ['low', 'medium', 'high', 'xhigh', 'max'],+ },+ };++ for (const [id, contract] of Object.entries(expected)) {+ assert.ok(byId.has(id), `${id} should be offered in the Codex model catalog`);+ const model = byId.get(id);+ for (const [field, value] of Object.entries(contract)) {+ assert.deepEqual(model[field], value, `${id}.${field}`);+ }+ assert.deepEqual(+ codexReasoningEffortOptions(model).map((option) => option.value),+ contract.supportedReasoningEfforts,+ `${id} should expose all of its supported reasoning efforts`,+ );+ }++ assert.equal(byId.get('gpt-5.3-codex')?.deprecated, true);+}diff --git a/valsmith_tests/run.mjs b/valsmith_tests/run.mjsnew file mode 100644index 0000000..19f1706--- /dev/null+++ b/valsmith_tests/run.mjs@@ -0,0 +1,20 @@+import { writeFileSync } from 'node:fs';+import { testCodex56Catalog } from './model_catalog.test.mjs';++const tests = [['Codex 5.6 models expose current defaults and reasoning efforts', testCodex56Catalog]];+const escapeXml = (value) => String(value)+ .replaceAll('&', '&').replaceAll('<', '<').replaceAll('>', '>')+ .replaceAll('\"', '"').replaceAll("'", ''');+let failures = 0;+const cases = [];+for (const [name, test] of tests) {+ try {+ test();+ cases.push(`<testcase classname="modelCatalog" name="${escapeXml(name)}"/>`);+ } catch (error) {+ failures += 1;+ cases.push(`<testcase classname="modelCatalog" name="${escapeXml(name)}"><failure message="${escapeXml(error.message)}">${escapeXml(error.stack)}</failure></testcase>`);+ }+}+writeFileSync('valsmith-junit.xml', `<?xml version="1.0" encoding="UTF-8"?><testsuite name="modelCatalog" tests="${tests.length}" failures="${failures}">${cases.join('')}</testsuite>`);+process.exitCode = failures ? 1 : 0;