Public

mirumee/ariadne

Updated: 8/13/2026

Languages

Python99%HTML0.8%Just0.2%

2 Models30 Tasks

Python library for implementing GraphQL servers using schema-first approach. Created with :heart: by Mirumee

		Harness			Input / Output Cost
1	GLM 5.2	Mini-SWE-agent	25 / 30	$0.37	$1.4/$4.4	7m02s
2	DeepSeek V4 Flash 0731	Mini-SWE-agent	23 / 30	$0.05	$0.44/$1.32	6m26s

Key Takeaways

GLM 5.2 (Fireworks) with Mini-SWE-agent scores 83.33% versus 76.67% for DeepSeek V4 Flash with Mini-SWE-agent.
DeepSeek V4 Flash with Mini-SWE-agent costs $0.05 per test and runs in 385.59 seconds, compared with $0.37 and 421.95 seconds for GLM 5.2 (Fireworks) with Mini-SWE-agent.

Model Comparison

Accuracy

83.33%

GLM 5.2

76.67%

DeepSeek V4 Flash 0731

Task outcomes

30 tasks

Both: 23 tasks

GLM 5.2 only: 2 tasks

Neither: 5 tasks

Both

GLM 5.2 only

DeepSeek V4 Flash 0731 only

Neither

Not attempted

Cost / test

$0.37

GLM 5.2

$0.05

DeepSeek V4 Flash 0731

Cost distribution

Latency

7m 2s

GLM 5.2

6m 26s

DeepSeek V4 Flash 0731

Latency distribution

Cost Analysis

Cost / Test vs. Accuracy

ACCURACYCOST

Average Token Use / Test

Token Usage

InputOutputReasoningCache readCache write

GLM 5.2

1.5M

DeepSeek V4 Flash 0731

1.4M

Cost is the clearest tradeoff in this comparison. GLM 5.2 leads at 83.33% for $0.37 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 76.67% for $0.05 per test.

Latency Analysis

Latency vs. Accuracy

ACCURACYLATENCY

Average Response Time / Test

Response Time

GLM 5.2

7m 2s

DeepSeek V4 Flash 0731

6m 26s

Latency separates several models with similarly strong scores. GLM 5.2 leads at 83.33%, while DeepSeek V4 Flash 0731 is fastest at 6m 26s with 76.67% accuracy.

Tasks with failures

Models
GLM 5.2
DeepSeek V4 Flash 0731

Task detail

6a4f4dc

Issue statement

Fix SQLAlchemy automatic eager loading so query depth is measured as the total relationship nesting depth, starting with the root model at depth 1. The max_depth argument passed to auto_eager_load must be enforced. A model's SQLAlchemyObjectType.max_depth may tighten an inherited limit for that model and all deeper selections, and depth errors should identify both the effective limit and the model that established it. Also ensure fields selected through inline fragments or named fragment spreads participate in scalar and relationship eager-loading exactly like directly selected fields, including recursively nested fragments.

View Hidden Tests

diff --git a/tests/sqlalchemy/test_valsmith_eager_load.py b/tests/sqlalchemy/test_valsmith_eager_load.pynew file mode 100644index 0000000..1829813--- /dev/null+++ b/tests/sqlalchemy/test_valsmith_eager_load.py@@ -0,0 +1,74 @@+from types import SimpleNamespace+from unittest.mock import Mock++import pytest+from graphql import GraphQLError, parse+from graphql.language import FragmentDefinitionNode, OperationDefinitionNode++from ariadne.contrib.sqlalchemy import SQLAlchemyObjectType+from ariadne.contrib.sqlalchemy.utils import auto_eager_load+++def _info(query: str, root_field: str) -> SimpleNamespace:+    document = parse(query)+    operation = next(+        definition+        for definition in document.definitions+        if isinstance(definition, OperationDefinitionNode)+    )+    field_nodes = [+        node+        for node in operation.selection_set.selections+        if node.name.value == root_field+    ]+    fragments = {+        definition.name.value: definition+        for definition in document.definitions+        if isinstance(definition, FragmentDefinitionNode)+    }+    return SimpleNamespace(field_nodes=field_nodes, fragments=fragments)+++def test_explicit_max_depth_limits_total_relationship_depth(models):+    query = Mock(name="query")+    info = _info("query { posts { tags { id } } }", "posts")++    with pytest.raises(GraphQLError, match=r"max_depth=1"):+        auto_eager_load(query, info, models["Post"], max_depth=1)+++def test_child_depth_limit_applies_to_subsequent_relationship_levels(models):+    query = Mock(name="query")+    info = _info("query { posts { tags { posts { id } } } }", "posts")+    post_type = SQLAlchemyObjectType("Post", models["Post"], max_depth=5)+    tag_type = SQLAlchemyObjectType("Tag", models["Tag"], max_depth=1)++    with pytest.raises(GraphQLError, match=r"max_depth=1.*Tag"):+        auto_eager_load(+            query,+            info,+            models["Post"],+            type_registry={models["Post"]: post_type, models["Tag"]: tag_type},+        )+++def test_inline_and_named_fragments_contribute_eager_load_options(models):+    query = Mock(name="query")+    info = _info(+        """+        query {+          posts {+            ... on Post { title }+            ...PostRelations+          }+        }+        fragment PostRelations on Post { tags { id } }+        """,+        "posts",+    )++    auto_eager_load(query, info, models["Post"])++    query.options.assert_called_once()+    paths = " | ".join(str(option.path) for option in query.options.call_args.args)+    assert "Post.tags" in paths