Evaluate repositories against these criteria before creating a new benchmark.
| Criteria | Description |
|---|---|
| Standardized interface | CLI (stdin/stdout/args), REST API, gRPC, or file processor. No libraries or GUIs. |
| Adaptable test suite | Tests that can verify behavior through external interface only. |
| Deterministic behavior | Same input → same output, or existing tests handle variability. |
| Self-contained | No external dependencies like databases, authentication services, cloud APIs. |
| Size: 50k - 200k LOC | Large enough to be meaningful, not too large to be impractical. |
- Alternative implementation exists in target language
- Well-documented
- Actively maintained
□ Interface type: CLI / REST API / gRPC / File processor
□ Has good coverage of implementation-agnostic tests (or easily convertible)?
□ Is behavior deterministic (or tests handle variability)?
□ No database, auth, or external service dependencies?
□ LOC between 50k - 200k?
Good candidate: jq (C) - CLI interface, comprehensive test suite, deterministic JSON processing, self-contained, ~50k LOC
Poor candidate: React - Library (no external interface), requires browser environment, massive ecosystem