Support reporting hyper parameters

[michalwols]

This looks like an awesome project.

Would be great if there was a way to report hyper parameters with each submission.

[RJT1990]

Hey @michalwols.

Yes, metadata related to training is something we were thinking of as a future feature. We have some ideas here, but would be interested to hear your thoughts about a lightweight way to capture this might be? (presently we are focussed on post-training metadata that can be extracted from pretrained models).

[michalwols]

Just adding a params key to the evaluator that takes a dict should work.

If you want to keep it clean you could use a TypedDict (it's available in mypy_extensions) to define the supported set of keys and their types.

class ImageClassificationParams(TypedDict):
  label_smoothing: bool
  optimizer: str
  warmup: bool
  epochs: int
  batch_size: int
  weight_decay: float
  extra_data: bool

from sotabencheval.image_classification import ImageClassificationParams, ImageNetEvaluator

evaluator = ImageNetEvaluator(
  model_name='FixResNeXt-101 32x48d',
  paper_arxiv_id='1906.06423',
  params=ImageClassificationParams(  # could also just be a dict if you annotate the function with params: ImageClassificationParams
    epochs=200,
    batch_size=32,
    weight_decay=.1,
    extra_data=True,
    fixres=True  # new key introduced by this method
  )
)

I guess for enum/choice support something like a dataclass might be better than a typed dict.