Skip to content

Opt-in community leaderboard for arena results #13

Description

@rexblade58

Context

The portable result schema (schema v1) already records everything needed for fair comparison. A leaderboard should consume only voluntarily submitted results.

Task

  1. agenteval arena --submit pushes the portable JSON to a leaderboard endpoint (opt-in, explicit)
  2. Leaderboard service (separate repo) validates schema + fairness metadata (commit, environment, agent version)
  3. Never upload private code, prompts, logs, or repo names without explicit consent
  4. Per-benchmark leaderboards: task pack + repo + agent + model + score

Acceptance criteria

  • --submit requires an explicit flag and a LEADERBOARD_URL env var
  • Result payload excludes prompts/logs unless --include-artifacts
  • Documented submission format matching the portable schema

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions