Skip to content

Rename database benchmark to AXeval-Database - #38

Closed
chenmingtang830 wants to merge 1 commit into
mainfrom
pr5-axarena-database-naming
Closed

Rename database benchmark to AXeval-Database#38
chenmingtang830 wants to merge 1 commit into
mainfrom
pr5-axarena-database-naming

Conversation

@chenmingtang830

@chenmingtang830 chenmingtang830 commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Rename the public database benchmark from DAEB to AXeval-Database.
  • Move the canonical benchmark root to ax-arena/benchmark/axeval-database and rename runtime commands to axeval-database-low-pass and axeval-database-production-rerun.
  • Preserve daeb-* commands and both former DAEB roots as deprecated read-only compatibility paths.
  • Publish the frozen legacy suite identity DAEB-1 as AXeval-Database v1 without changing any v1 pack or approval bytes.
  • Update documentation, schemas, release gates, package inventories, workflows, and tests.

Compatibility boundary

The immutable v1 artifacts retain their approved internal DAEB-1 identity. Existing configs and invocations continue to parse through explicit compatibility aliases; all new writes and public output use AXeval-Database.

Validation

  • GitHub CI: all checks passed, including test, typecheck, package, normalized-record diff, and security checks
  • Core local tests: 491 passed
  • Arena targeted migration, frozen-artifact, and publication tests: passed
  • npm run typecheck: passed
  • npm run build: passed
  • npm run pack:check: passed for core and arena
  • npm --cache .npm-cache pack --dry-run: passed
  • Arena package npm pack --dry-run: passed

@vaibhav-i

Copy link
Copy Markdown
Collaborator

Closing this planning-only draft because it contains no implementation and its scope is now represented by the reviewed replacement stack. Keeping an empty PR open would create a false review dependency.

@vaibhav-i vaibhav-i closed this Jul 16, 2026
@chenmingtang830 chenmingtang830 changed the title PR5: Rename DAEB to AXArena Database with compatibility migration Rename database benchmark to AXeval-Database Jul 27, 2026
@chenmingtang830
chenmingtang830 changed the base branch from pr4-authoring-freeze to main July 27, 2026 01:01
@chenmingtang830
chenmingtang830 force-pushed the pr5-axarena-database-naming branch from 6971fec to 8a1631b Compare July 27, 2026 01:02
@github-actions

Copy link
Copy Markdown

Normalized records diff

Overall is a macro-average of participating surface scores. Agents are never averaged together.

Overall by agent

Product Agent Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Agent Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect ranking.

@chenmingtang830

Copy link
Copy Markdown
Owner Author

Superseded by #216, which is a clean branch from current main and includes the coordinated pre-production AXeval-Database path and publication-identity migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants