Skip to content

test(benchmark): define benchmark contribution protocol #8

@michael-johnston

Description

@michael-johnston

The goal is to define the multi-stage process for submitting and validating a models benchmarking requirements and for executing the benchmark.

The process must satisfy the requirements in #3

✍️ Scope of the Protocol Document

The document will be structured around the three core stages of a contribution. For each stage, it will describe the user/admin actions and map to the relevant requirements from #3.

Stage 1: The Pull Request (PR) Submission

This stage describes the contributor's responsibility when opening a PR - What do they need to include to have PR been accepted for merging from benchmark.

This will include the various files and their formats etc.

Stage 2: The Pull Request Review & Validation

This stage describes the automated and/or manual checks that will validate that the artefacts provided in Stage 1 meet specific requirements

Stage 3: Post-Merge - Benchmark Execution

This stage describes what happens after a contribution has been approved and merged - again what is automated and what is manual.


✅ Definition of Done:

  • A document (perhaps part of existing CONTRIBUTING_MODEL.md) that clearly defines what the three stages of the contribution process (Submission, Validation, Post-Merge) - what is specifically required as input to each, and what is required to happen with those inputs, what is the criteria to move between stages.

Metadata

Metadata

Labels

documentationImprovements or additions to documentation

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions