The goal is to define the multi-stage process for submitting and validating a models benchmarking requirements and for executing the benchmark.
The process must satisfy the requirements in #3
✍️ Scope of the Protocol Document
The document will be structured around the three core stages of a contribution. For each stage, it will describe the user/admin actions and map to the relevant requirements from #3.
Stage 1: The Pull Request (PR) Submission
This stage describes the contributor's responsibility when opening a PR - What do they need to include to have PR been accepted for merging from benchmark.
This will include the various files and their formats etc.
Stage 2: The Pull Request Review & Validation
This stage describes the automated and/or manual checks that will validate that the artefacts provided in Stage 1 meet specific requirements
Stage 3: Post-Merge - Benchmark Execution
This stage describes what happens after a contribution has been approved and merged - again what is automated and what is manual.
✅ Definition of Done:
The goal is to define the multi-stage process for submitting and validating a models benchmarking requirements and for executing the benchmark.
The process must satisfy the requirements in #3
✍️ Scope of the Protocol Document
The document will be structured around the three core stages of a contribution. For each stage, it will describe the user/admin actions and map to the relevant requirements from #3.
Stage 1: The Pull Request (PR) Submission
This stage describes the contributor's responsibility when opening a PR - What do they need to include to have PR been accepted for merging from benchmark.
This will include the various files and their formats etc.
Stage 2: The Pull Request Review & Validation
This stage describes the automated and/or manual checks that will validate that the artefacts provided in Stage 1 meet specific requirements
Stage 3: Post-Merge - Benchmark Execution
This stage describes what happens after a contribution has been approved and merged - again what is automated and what is manual.
✅ Definition of Done: