Skip to content
This repository was archived by the owner on Nov 1, 2024. It is now read-only.
This repository was archived by the owner on Nov 1, 2024. It is now read-only.

Bug with adding a new dataset, failing evaluation if name is misformatted / round is not selected #835

@ktirumalafb

Description

@ktirumalafb

I found a bug when trying to modify eval server, where:
(1) if the name of an uploaded dataset is not [a-ZA-Z] (for example squad_v2)
OR
(2) a round is not selected (the default round number for a dataset is 0)

then batch transform jobs are sent through and the model is evaluated, but the update to the DB fails. So essentially, we do all the work of evaluation, but it still shows up as "failed" and the job metrics are lost. We should either (a) force task owners to correctly format the dataset name + select a round when they upload a dataset, before running evaluation or (b) have a sensible default way of handling this (for example, getting rid of unwanted characters in dataset name to solve (1) + selecting the latest round to solve (2))

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions