goharbor/harbor
Languages
An open source trusted cloud native registry project that stores, signs, and scans content.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 18 / 30 | $3.04 | $5/$25 | 10m14s | ||
2 | Mini-SWE-agent | 17 / 30 | $3.59 | $3/$15 | 14m40s | ||
3 | Mini-SWE-agent | 16 / 30 | $0.41 | $2/$6 | 4m04s | ||
4 | Mini-SWE-agent | 14 / 30 | $0.33 | $1/$5 | 3m11s |
Key Takeaways
- Claude Sonnet 5 with Mini-SWE-agent resolves 17 tasks, followed by Grok 4.5 with Mini-SWE-agent at 16 and Claude Haiku 4.5 (Nonthinking) with Mini-SWE-agent at 14.
- Grok 4.5 with Mini-SWE-agent has the lowest cost per test at $0.41 and lowest latency at 244.04 seconds among the supplied runs.
Model Comparison
Accuracy
60.00%
Claude Opus 5
56.67%
Claude Sonnet 5
Task outcomes
30 tasks
Cost / test
$3.04
Claude Opus 5
$3.59
Claude Sonnet 5
Cost distribution
Latency
10m 14s
Claude Opus 5
14m 40s
Claude Sonnet 5
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 60.00% for $3.04 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 60.00%, while Claude Haiku 4.5 (Nonthinking) is fastest at 3m 11s with 46.67% accuracy.
Tasks with failures
| Models | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | |||||||||||||||||
| Claude Sonnet 5 | |||||||||||||||||
| Grok 4.5 | |||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
2f7addaIssue statement
Harbor does not expose that an OCI image manifest is covered by a signature attached to a parent OCI index. Add an opt-in artifact retrieval capability that discovers parent indexes and returns their Cosign or Notation signature accessories separately from the artifact's own accessories. Do not inherit non-signature accessories, and do not add inherited signatures when the child already has a direct signature. The default retrieval path must remain unchanged so callers that do not opt in incur no parent-reference lookup. Expose the separate inherited-accessory collection through the v2 API model and allow API callers to request it.
View Hidden Tests
diff --git a/src/controller/artifact/inherited_accessories_hidden_test.go b/src/controller/artifact/inherited_accessories_hidden_test.gonew file mode 100644index 000000000..111111111--- /dev/null+++ b/src/controller/artifact/inherited_accessories_hidden_test.go@@ -0,0 +1,54 @@+package artifact++import (+ "testing"++ "github.com/stretchr/testify/mock"+ "github.com/stretchr/testify/require"++ "github.com/goharbor/harbor/src/lib/q"+ accessorymodel "github.com/goharbor/harbor/src/pkg/accessory/model"+ basemodel "github.com/goharbor/harbor/src/pkg/accessory/model/base"+ pkgartifact "github.com/goharbor/harbor/src/pkg/artifact"+)++func hiddenAccessory(id, subjectID int64, digest, kind string) accessorymodel.Accessory {+ return &basemodel.Default{Data: accessorymodel.AccessoryData{+ ID: id, ArtifactID: subjectID, SubArtifactDigest: digest, Type: kind,+ }}+}++func hiddenSubjectID(id int64) any {+ return mock.MatchedBy(func(query *q.Query) bool {+ return query != nil && query.Keywords["SubjectArtifactID"] == id+ })+}++func TestHiddenInheritedParentSignaturesAreSeparatedAndFiltered(t *testing.T) {+ suite := &controllerTestSuite{}+ suite.SetT(t)+ suite.SetupTest()++ suite.artMgr.On("ListReferences", mock.Anything, mock.Anything).Return([]*pkgartifact.Reference{+ {ParentID: 10, ChildID: 1},+ {ParentID: 20, ChildID: 1},+ }, nil)+ cosign := hiddenAccessory(1, 10, "sha256:parent-one", accessorymodel.TypeCosignSignature)+ notation := hiddenAccessory(2, 20, "sha256:parent-two", accessorymodel.TypeNotationSignature)+ sbom := hiddenAccessory(3, 10, "sha256:parent-one", accessorymodel.TypeHarborSBOM)+ suite.accMgr.On("List", mock.Anything, hiddenSubjectID(int64(10))).+ Return([]accessorymodel.Accessory{cosign, sbom}, nil)+ suite.accMgr.On("List", mock.Anything, hiddenSubjectID(int64(20))).+ Return([]accessorymodel.Accessory{notation}, nil)++ child := &Artifact{+ Artifact: pkgartifact.Artifact{ID: 1, Digest: "sha256:child"},+ Accessories: []accessorymodel.Accessory{},+ }+ suite.ctl.populateInheritedAccessories(nil, child)++ require.Empty(t, child.Accessories, "parent accessories must not become direct child accessories")+ require.Len(t, child.InheritedAccessories, 2)+ require.Equal(t, cosign, child.InheritedAccessories[0])+ require.Equal(t, notation, child.InheritedAccessories[1])+}