Public

apache/spark

Updated: 8/15/2026

Languages

Scala68.5%Python16.2%Java6.7%Jupyter Notebook4.3%Other2.4%HiveQL1.9%
14 Models58 Tasks

Apache Spark - A unified analytics engine for large-scale data processing

Harness

1

Mini-SWE-agent
49 / 58

$2.46

9m48s

2

Mini-SWE-agent
48 / 58

$7.44

16m48s

3

Mini-SWE-agent
48 / 58

$4.55

17m18s

4

Mini-SWE-agent
47 / 58

$0.71

3m47s

5

Mini-SWE-agent
47 / 58

$0.88

15m39s

6

Mini-SWE-agent
47 / 58

$9.82

16m32s

7

Mini-SWE-agent
45 / 58

$4.96

21m11s

8

Mini-SWE-agent
44 / 58

$4.92

13m26s

9

Mini-SWE-agent
41 / 58

$0.12

5m21s

10

Mini-SWE-agent
38 / 58

$0.48

3m30s

11

Mini-SWE-agent
38 / 58

$1.26

5m14s

12

Mini-SWE-agent
38 / 58

$2.02

6m19s

13

Mini-SWE-agent
33 / 58

$0.70

3m23s

14

Mini-SWE-agent
33 / 58

$0.41

4m36s

Key Takeaways

  • Claude Fable 5 and Claude Opus 4.8 with Mini-SWE-agent each resolve 48 of 58 tasks (82.76%), while GLM 5.2 (Fireworks), Kimi K3, and Grok 4.5 each resolve 47.
  • Grok 4.5 with Mini-SWE-agent resolves 47 of 58 tasks for $0.71 per test in 226.98 seconds, compared with GPT-5.6 Sol at $2.46 and 588.17 seconds.

Model Comparison

Accuracy

84.48%

GPT-5.6 Sol

82.76%

Claude Opus 4.8

Task outcomes

58 tasks

Both
GPT-5.6 Sol only
Claude Opus 4.8 only
Neither
Not attempted

Cost / test

$2.46

GPT-5.6 Sol

$4.55

Claude Opus 4.8

Cost distribution

$0.00$8.75$17.50

Latency

9m 48s

GPT-5.6 Sol

17m 18s

Claude Opus 4.8

Latency distribution

0s28m 41s57m 22s

Cost Analysis

Cost / Test vs. Accuracy
ACCURACYCOST

Average Token Use / Test

Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
11.2M
Claude Opus 4.7
6.7M
Claude Opus 4.8
4.8M
GLM 5.2
4.0M
Claude Fable 5
3.8M
Kimi K3
3.1M
Gemini 3.5 Flash
2.8M
GPT-5.6 Luna
2.8M
Claude Haiku 4.5 (Nonthinking)
2.3M
GPT-5.6 Sol
2.2M
GPT 5.5
2.0M
Grok 4.5
999K
Gemini 3.1 Pro Preview (02/26)
913K
GPT-5.6 Terra
805K

Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 84.48% for $2.46 per test. Grok 4.5 is the lower-cost option at 81.03% for $0.71 per test.

Latency Analysis

Latency vs. Accuracy
ACCURACYLATENCY

Average Response Time / Test

Response Time
Claude Sonnet 5
21m 11s
Claude Opus 4.8
17m 18s
Claude Fable 5
16m 48s
Kimi K3
16m 32s
GLM 5.2
15m 39s
Claude Opus 4.7
13m 26s
GPT-5.6 Sol
9m 48s
GPT 5.5
6m 19s
GPT-5.6 Luna
5m 21s
Gemini 3.5 Flash
5m 14s
Claude Haiku 4.5 (Nonthinking)
4m 36s
Grok 4.5
3m 47s
GPT-5.6 Terra
3m 30s
Gemini 3.1 Pro Preview (02/26)
3m 23s

Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 84.48%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 23s with 56.90% accuracy.

Tasks with failures

Models
GPT-5.6 Sol
Claude Opus 4.8
Claude Fable 5
Grok 4.5
GLM 5.2
Kimi K3
Claude Sonnet 5
Claude Opus 4.7
GPT-5.6 Luna
GPT-5.6 Terra
Gemini 3.5 Flash
GPT 5.5
Claude Haiku 4.5 (Nonthinking)
Gemini 3.1 Pro Preview (02/26)

Task detail

fe0e502

Issue statement

Correlated scalar and lateral subqueries reference columns from the enclosing query through outer references. When such an outer reference appears as a bare, top-level output column of the subquery's Project or Aggregate (for example SELECT (SELECT a) FROM t1, or a lateral subquery whose GROUP BY produces the grouping column directly), the analyzer currently leaves it as a bare OuterReference in the operator's output list.

Because OuterReference.toAttribute strips the outer wrapper, the attribute exposed through Project.output / Aggregate.output reuses the outer attribute's ExprId. That outer expression id then leaks into the subquery's scope, so downstream operators that reference the subquery's output end up sharing an expression id with an attribute from the outer plan. This breaks the invariant that each operator produces attributes with fresh, self-contained expression ids and can lead to incorrect expression-id tracking.

Expected behavior: when the resolved output of a subquery's Project or Aggregate is a bare top-level OuterReference (i.e. a NamedExpression that is exactly an OuterReference, not already wrapped in an Alias), the analyzer should wrap it in an Alias so that the output attribute gets a fresh ExprId while still carrying the original column name. Outputs that are already an Alias (such as those produced by struct star expansion or explicit user aliases) must be left unchanged. The alias name should match the referenced column's name, and the wrapped OuterReference must keep pointing at the original outer attribute (same ExprId on the inner OuterReference) while the surrounding Alias carries a new ExprId.

View Hidden Tests
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scalaindex 30ef9532..48286359 100644--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala@@ -254,7 +254,8 @@ class ResolveSubquerySuite extends AnalysisTest {     // SELECT (SELECT a) FROM t1     checkAnalysis(       Project(ScalarSubquery(t0.select($"a")).as("sub") :: Nil, t1),-      Project(ScalarSubquery(Project(OuterReference(a) :: Nil, t0), Seq(a)).as("sub") :: Nil, t1)+      Project(ScalarSubquery(+        Project(OuterReference(a).as(a.name) :: Nil, t0), Seq(a)).as("sub") :: Nil, t1)     )     // SELECT (SELECT a + b + c AS r FROM t2) FROM t1     checkAnalysis(@@ -274,6 +275,27 @@ class ResolveSubquerySuite extends AnalysisTest {     )   } +  test("SPARK-58171: alias a bare outer reference that is an Aggregate output") {+    // SELECT * FROM t1, LATERAL (SELECT a, count(b) AS cnt FROM t2 GROUP BY a)+    // The bare outer reference `a` in the Aggregate output must be aliased so it gets a fresh+    // ExprId, instead of leaking the outer attribute's ExprId through the Aggregate output.+    // The query is only resolved (not passed through CheckAnalysis) because a correlated+    // aggregate output is otherwise rejected as an unsupported correlated reference.+    val plan = lateralJoin(t1, t2.groupBy($"a")($"a", Count($"b").as("cnt")))+    val resolved = getAnalyzer.execute(plan)+    val aggregate = resolved.collectWithSubqueries { case a: Aggregate => a }.head+    // The first output is the aliased outer reference; the outer attribute's ExprId is not+    // leaked through the alias (the alias carries a fresh ExprId).+    aggregate.aggregateExpressions.head match {+      case alias @ Alias(o: OuterReference, name) =>+        assert(name == a.name)+        assert(o.exprId == a.exprId)+        assert(alias.exprId != a.exprId)+      case other =>+        fail(s"Expected the outer reference to be wrapped in an Alias, but got: $other")+    }+  }+   test("SPARK-47509: Incorrect results for subquery expressions in LambdaFunctions") {     val data = LocalRelation(Seq(       $"key".int,