arandomhooman/hoomans-morphe-patches
Languages
Patches for morphe
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 6 / 30 | $2.09 | $5/$30 | 11m38s | ||
2 | Mini-SWE-agent | 5 / 30 | $0.03 | $0.44/$1.32 | 4m55s | ||
3 | Mini-SWE-agent | 5 / 30 | $3.19 | $5/$25 | 12m32s | ||
4 | Mini-SWE-agent | 5 / 30 | $3.64 | $5/$25 | 15m15s |
Key Takeaways
- This is a directional result because the 30-task benchmark has multiple supplied runs but no repeated-run uncertainty metrics.
- Deepseek V4 Flash 0731 with Mini-SWE-agent matches Claude Opus 4.8 and Claude Opus 5 at 5 of 30 tasks while reporting the lowest cost per test, $0.03.
Model Comparison
Accuracy
20.00%
GPT-5.6 Sol
16.67%
DeepSeek V4 Flash 0731
Task outcomes
30 tasks
Cost / test
$2.09
GPT-5.6 Sol
$0.03
DeepSeek V4 Flash 0731
Cost distribution
Latency
11m 38s
GPT-5.6 Sol
4m 55s
DeepSeek V4 Flash 0731
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 20.00% for $2.09 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 16.67% for $0.03 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 20.00%, while DeepSeek V4 Flash 0731 is fastest at 4m 55s with 16.67% accuracy.
Tasks with failures
| Models | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | |||||||||||||||||||||||||
| DeepSeek V4 Flash 0731 | |||||||||||||||||||||||||
| Claude Opus 5 | |||||||||||||||||||||||||
| Claude Opus 4.8 |
Task detail
5350a27Issue statement
Add a bytecode patch named "Fix login" for Twitch package tv.twitch.android.app, compatible with version 30.7.2. Patched/re-signed builds currently fail login because Twitch's background Play Integrity flow sends a failed attestation to the server. The patch must suppress that attestation flow while leaving the credential login flow unchanged. Identify the dedicated attestation class and its setup method using the unique play_integrity_setup_begin analytics marker, require each lookup to be unambiguous, and make the setup method complete immediately with a null object result so it does not initialize Play Integrity, request a token, or post attestation data.
View Hidden Tests
diff --git a/valsmith_tests/test_fix_login_patch.py b/valsmith_tests/test_fix_login_patch.pynew file mode 100644index 0000000..5f5467e--- /dev/null+++ b/valsmith_tests/test_fix_login_patch.py@@ -0,0 +1,52 @@+import re+from pathlib import Path+++PATCH_FILE = Path(+ "patches/src/main/kotlin/hooman/morphe/patches/twitch/login/FixLoginPatch.kt"+)+++def _source_without_comments() -> str:+ source = PATCH_FILE.read_text(encoding="utf-8")+ source = re.sub(r"/\*.*?\*/", "", source, flags=re.S)+ return re.sub(r"//.*", "", source)+++def test_twitch_login_patch_suppresses_play_integrity_attestation():+ assert PATCH_FILE.is_file(), "the Twitch login fix patch is missing"+ source = _source_without_comments()++ assert re.search(r'bytecodePatch\s*\([^)]*name\s*=\s*"Fix login"', source, re.S)+ assert re.search(+ r'Compatibility\s*\([^)]*name\s*=\s*"Twitch"[^)]*'+ r'packageName\s*=\s*"tv\.twitch\.android\.app"[^)]*'+ r'AppTarget\s*\(\s*"30\.7\.2"\s*\)',+ source,+ re.S,+ )++ assert re.search(+ r'classDefByStrings\s*\(\s*"play_integrity_setup_begin"\s*\)'+ r'\s*\.\s*singleOrNull\s*\(\s*\)',+ source,+ re.S,+ ), "the attestation class must be selected uniquely by its setup event"+ assert re.search(+ r'methods\s*\.\s*singleOrNull\s*\{.*?'+ r'"play_integrity_setup_begin".*?\}',+ source,+ re.S,+ ), "the setup method must be selected uniquely by the same event"++ injection = re.search(+ r'\.addInstructions\s*\(\s*0\s*,\s*"""(.*?)"""\s*,?\s*\)',+ source,+ re.S,+ )+ assert injection, "the attestation setup method must be overridden at entry"+ instructions = [line.strip() for line in injection.group(1).splitlines() if line.strip()]+ assert len(instructions) >= 2+ assert re.fullmatch(r"const/4\s+v(\d+)\s*,\s*0x0", instructions[0])+ register = re.search(r"v(\d+)", instructions[0]).group(0)+ assert instructions[1] == f"return-object {register}"