Public

ucbepic/docetl

Updated: 8/17/2026

Languages

Python68.6%TypeScript31.1%Makefile0.1%CSS<0.1%Dockerfile<0.1%JavaScript<0.1%
1 Models30 Tasks

A system for agentic LLM-powered data processing and ETL

Add models

Harness

1

zAI
GLM 5.2
Mini-SWE-agent
9 / 30

$0.52

10m42s

Key Takeaways

  • This 30-task result provides a directional benchmark of GLM 5.2 (Fireworks) with Mini-SWE-agent, with a 30% score.

Cost Analysis

Cost / Test vs. Accuracy
ACCURACYCOST

Average Token Use / Test

Token Usage
InputOutputReasoningCache readCache write

No token usage data available.

Cost is the clearest tradeoff in this comparison. GLM 5.2 leads at 30.00% for $0.52 per test. No other model in this comparison is cheaper.

Latency Analysis

Latency vs. Accuracy
ACCURACYLATENCY

Average Response Time / Test

Response Time
GLM 5.2
10m 42s

GLM 5.2 is both the most accurate and fastest model in this comparison at 30.00% and 10m 42s.

Tasks with failures

Models
GLM 5.2

Task detail

ae7f10f

Issue statement

Map operations with calibration enabled must not mutate their configuration while executing or retain an augmented prompt afterward. The same configured operation should be safely reusable: each execution should calibrate from the original prompt, use the generated calibration context only for that execution, and leave the caller-provided configuration unchanged throughout, including while nested calibration work is running.

View Hidden Tests
diff --git a/tests/test_map_calibration_config.py b/tests/test_map_calibration_config.pynew file mode 100644index 0000000..d394a29--- /dev/null+++ b/tests/test_map_calibration_config.py@@ -0,0 +1,65 @@+import copy+from types import SimpleNamespace++from docetl.operations.map import MapOperation+++def test_map_calibration_does_not_mutate_config():+    config = {+        "name": "calibrated-map",+        "type": "map",+        "prompt": "Classify {{ input.text }}",+        "output": {"schema": {"label": "string"}},+        "calibrate": True,+        "num_calibration_docs": 1,+    }+    original_config = copy.deepcopy(config)++    class CaptureAPI:+        def __init__(self):+            self.calls = []++        def call_llm(self, model, op_type, messages, schema, **kwargs):+            assert config == original_config+            self.calls.append(+                {+                    "op_type": op_type,+                    "prompt": messages[0]["content"],+                    "op_config": copy.deepcopy(kwargs["op_config"]),+                }+            )+            if op_type == "calibration":+                response = {"calibration_context": "Use the stable anchor."}+            else:+                response = {"label": "stable"}+            return SimpleNamespace(response=response, validated=True, total_cost=0.0)++        def parse_llm_response(self, response, **kwargs):+            return [response]++        def validate_output(self, config, output, console):+            return True++    api = CaptureAPI()+    runner = SimpleNamespace(config={}, api=api, is_cancelled=False)+    operation = MapOperation(runner, config, "gpt-4o-mini", max_threads=1)++    for _ in range(2):+        output, cost = operation.execute([{"text": "this"}])+        assert output == [{"text": "this", "label": "stable"}]+        assert cost == 0.0+        assert config == original_config++    map_calls = [call for call in api.calls if call["op_type"] == "map"]+    assert [call["prompt"] for call in map_calls] == [+        "Classify this",+        "Classify this\n\nUse the stable anchor.",+        "Classify this",+        "Classify this\n\nUse the stable anchor.",+    ]+    assert [call["op_config"]["prompt"] for call in map_calls] == [+        original_config["prompt"],+        f"{original_config['prompt']}\n\nUse the stable anchor.",+        original_config["prompt"],+        f"{original_config['prompt']}\n\nUse the stable anchor.",+    ]