/ jax-ml / jax
Updated 7/20/2026
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
Key Takeaways
- GPT-5.6 Sol leads with 43 of 54 tasks resolved, while six models tie one task behind.
- Grok 4.5 is the fastest model in the 42-of-54 tier at 395 seconds; Muse Spark 1.1 reaches the same score for $0.68 per test.
- GPT-5.6 Sol and Claude Haiku 4.5 (Thinking) resolve 45 of 54 together, showing that the lowest-ranked model still contributes complementary successes.
Cost / Test vs. Accuracy
Muse Spark 1.1 resolves 42 of 54 tasks for $0.68 per test, one task behind GPT-5.6 Sol at $2.27.
Latency vs. Accuracy
GPT-5.6 Terra resolves 41 of 54 tasks in 187 seconds, while Grok 4.5 reaches 42 in 395 seconds and the leader reaches 43 in 611 seconds.
Average token use / test
Tasks with failures
| Model | 547cb7e | da78585 | 1946a44 | dd9fc0c | d5bc1dd | f7337b2 | 79f101c | bbd2b24 | 3d946c6 | 9a42858 | e6299e2 | 475db74 | 8c1616e | 140a1aa | 7d5c17a | c07445d | 3e62889 | 0b9e416 | a71c456 | 9120bf8 | 3ca48f5 | a3ba54d | 66d1c11 | b05e7a6 | fe88048 | bf8f885 | c36aaca |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | |||||||||||||||||||||||||||
| Muse Spark 1.1 | |||||||||||||||||||||||||||
| Grok 4.5 | |||||||||||||||||||||||||||
| GPT 5.5 | |||||||||||||||||||||||||||
| Claude Sonnet 5 | |||||||||||||||||||||||||||
| Claude Fable 5 | |||||||||||||||||||||||||||
| Kimi K3 | |||||||||||||||||||||||||||
| GPT-5.6 Terra | |||||||||||||||||||||||||||
| GPT-5.6 Luna | |||||||||||||||||||||||||||
| Gemini 3.5 Flash | |||||||||||||||||||||||||||
| Claude Opus 4.8 | |||||||||||||||||||||||||||
| Claude Opus 4.7 | |||||||||||||||||||||||||||
| GLM 5.2 | |||||||||||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | |||||||||||||||||||||||||||
| Claude Haiku 4.5 (Thinking) |
Task detail
547cb7ea1712Issue statement
The MLIR verifier for Mosaic's tpu.enqueue_indirect_dma operation does not correctly validate the rank of the offsets operand.
An indirect DMA (used for gather/scatter) requires the offsets memref shape to be either 1D ([o]) or a leading-unit 2D shape ([1, o]). Any other rank is unsupported. When an offsets shape has an unsupported rank, the verifier should reject the op with a clear diagnostic of the form:
Offsets shape must be 1D or (1, N), got (<dims>)
where <dims> is the comma-separated list of the offsets dimensions (for a rank-0 shape this renders as an empty list, i.e. got ()).
Today this contract is only partially enforced. In particular, a rank-0 (zero-dimensional) offsets shape is not caught by the offsets rank check: because the rank check is performed after other shape-derived quantities are computed, a zero-rank offsets operand slips past it and instead trips an unrelated error about the source/operand rank, rather than the intended "Offsets shape must be 1D or (1, N)" message. More generally, the offsets rank should be validated up front, before any shape arithmetic that assumes a 1D or (1, N) offsets shape.
Update the gather and scatter verification paths of enqueue_indirect_dma so that offsets whose rank is neither 1 nor a leading-unit 2D shape are rejected with the Offsets shape must be 1D or (1, N), got (...) error, including the rank-0 case.
Hidden tests
diff --git a/jaxlib/mosaic/BUILD b/jaxlib/mosaic/BUILDindex 91244d4..74b9060 100644--- a/jaxlib/mosaic/BUILD+++ b/jaxlib/mosaic/BUILD@@ -272,6 +272,7 @@ cc_test( srcs = ["dialect/tpu/tpu_ops_verification_test.cc"], deps = [ ":tpu_dialect",+ "@xla//xla/tsl/platform:status_matchers", "@com_google_absl//absl/status", "@com_google_googletest//:gtest_main", "@llvm-project//mlir:ArithDialect",diff --git a/jaxlib/mosaic/dialect/tpu/tpu_ops_verification_test.cc b/jaxlib/mosaic/dialect/tpu/tpu_ops_verification_test.ccindex 1f422e5..2681ada 100644--- a/jaxlib/mosaic/dialect/tpu/tpu_ops_verification_test.cc+++ b/jaxlib/mosaic/dialect/tpu/tpu_ops_verification_test.cc@@ -37,13 +37,14 @@ limitations under the License. #include "mlir/Support/LLVM.h" #include "jaxlib/mosaic/dialect/tpu/tpu_dialect.h" #include "xla/mlir/utils/error_util.h"+#include "xla/tsl/platform/status_matchers.h" namespace mlir::tpu { namespace { using ::testing::_; using ::testing::HasSubstr;-using ::testing::status::StatusIs;+using ::tsl::testing::StatusIs; class TpuOpsVerificationTest : public ::testing::Test { protected:@@ -1075,6 +1076,37 @@ TEST_F(TpuOpsVectorSubcoreVerificationTest, IndirectDma1DSemaphore) { StatusIs(_, HasSubstr("Semaphore must be rank 0"))); } +TEST_F(TpuOpsVectorSubcoreVerificationTest,+ IndirectDmaGatherZeroRankOffsetsInvalid) {+ auto dma = Create<EnqueueIndirectDMAOp>(+ /*source=*/AllocaI32({1024, 128}, MemorySpace::kHbm),+ /*target=*/AllocaI32({16, 128}, MemorySpace::kVmem),+ /*offsets=*/AllocaI32({}, MemorySpace::kVmem),+ /*semaphore=*/AllocaSemaphore(),+ /*offset_filter=*/nullptr,+ /*add=*/false);++ ASSERT_THAT(+ VerifyOp(dma),+ StatusIs(_, HasSubstr("Offsets shape must be 1D or (1, N), got ()")));+}++TEST_F(TpuOpsVectorSubcoreVerificationTest,+ IndirectDmaGatherHighRankOffsetsInvalid) {+ auto dma = Create<EnqueueIndirectDMAOp>(+ /*source=*/AllocaI32({1024, 128}, MemorySpace::kHbm),+ /*target=*/AllocaI32({1, 64, 32, 128}, MemorySpace::kVmem),+ /*offsets=*/AllocaI32({1, 64, 32}, MemorySpace::kVmem),+ /*semaphore=*/AllocaSemaphore(),+ /*offset_filter=*/nullptr,+ /*add=*/false);++ ASSERT_THAT(+ VerifyOp(dma),+ StatusIs(_, HasSubstr(+ "Offsets shape must be 1D or (1, N), got (1, 64, 32)")));+}+ TEST_F(TpuOpsVectorSubcoreVerificationTest, IndirectDmaGatherOffsetsShapeInvalid) { auto dma = Create<EnqueueIndirectDMAOp>(
Head-to-head
Accuracy
79.63%Δ 1.85%
77.78%
Cost / test
$2.27
$0.68Δ $1.58
Latency
611sΔ 883s
1494s
Cost distribution
Latency distribution
Input token distribution
Output token distribution
Task outcomes
54 tasks