You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 1, 2024. It is now read-only.
I found a bug when trying to modify eval server, where:
(1) if the name of an uploaded dataset is not [a-ZA-Z] (for example squad_v2)
OR
(2) a round is not selected (the default round number for a dataset is 0)
then batch transform jobs are sent through and the model is evaluated, but the update to the DB fails. So essentially, we do all the work of evaluation, but it still shows up as "failed" and the job metrics are lost. We should either (a) force task owners to correctly format the dataset name + select a round when they upload a dataset, before running evaluation or (b) have a sensible default way of handling this (for example, getting rid of unwanted characters in dataset name to solve (1) + selecting the latest round to solve (2))
I found a bug when trying to modify eval server, where:
(1) if the name of an uploaded dataset is not [a-ZA-Z] (for example
squad_v2)OR
(2) a round is not selected (the default round number for a dataset is 0)
then batch transform jobs are sent through and the model is evaluated, but the update to the DB fails. So essentially, we do all the work of evaluation, but it still shows up as "failed" and the job metrics are lost. We should either (a) force task owners to correctly format the dataset name + select a round when they upload a dataset, before running evaluation or (b) have a sensible default way of handling this (for example, getting rid of unwanted characters in dataset name to solve (1) + selecting the latest round to solve (2))