Skip to content

Make Architect Eval Benchmark Input Schema Configurable #29

@humza-sami

Description

@humza-sami

Currently, the Architect class only supports .jsonl benchmark files with a fixed schema containing "question" and "answer" keys. This limits flexibility for users who have datasets with different key names or structures.

We need to add functionality to allow configuration of these keys, so users can specify which key represents the question and which key represents the reference answer when creating an Architect instance.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions