Assertion Error in Novel Category Evaluation Due to Category Number Mismatch

[nightgoodl]

When evaluating novel categories using do_test(cfg, model, mode='novel'), an assertion error occurs in omni3d_evaluation.py:

AssertionError: assert len(class_names) == precisions.shape[2]

This error indicates a mismatch between the number of configured categories and the actual prediction categories during evaluation.

Current Implementation

1. The evaluation process is called separately for novel and base categories:

do_test(cfg, model, mode='novel')
do_test(cfg, model, mode='base')

2. In omni3d_evaluation.py, the evaluation uses all category names regardless of mode:

res = self._derive_omni_results(
    evals[mode],
    task,
    mode,
    class_names=self._metadata.get("thing_classes"),  # Using all categories
)

Problem

When evaluating novel categories:

• class_names contains all categories (base + novel)
• But precisions.shape[2] only contains novel categories
• This mismatch triggers the assertion error

Any suggestions would be greatly appreciated.

[nightgoodl]

I am currently testing this on ARKitScenes only

1. In dataset loading (datasets.py):

   • When loading novel datasets, all category mappings are loaded from configs/category_meta.json
   • For ARKitScenes_test_novel, this includes mappings for all possible categories

2. In evaluation (builtin.py):

   • get_omni3d_categories() for ARKitScenes_test_novel only returns one category: {'fireplace'}（because your settings in builtin.py）
   • This creates a mismatch between loaded categories and categories used for evaluation

[nightgoodl]

And I am confused why the novel category is included in both the ARKitScenes training set and the validation set. In my understanding, it should not exist.

[nightgoodl]

I think I may have setup my SFP.yaml file incorrectly

DATASETS:
  TRAIN: ('ARKitScenes_train', 'ARKitScenes_val')
  TEST: ('ARKitScenes_test', 'ARKitScenes_test_novel') 
  CATEGORY_NAMES: ("toilet", "fireplace", "table", "bed", "sofa", "television", "refrigerator", "chair", "cabinet", "stove", "oven", "shelves", "sink", "machine", "bathtub")

I would be grateful for any suggestions you have

[yaojin17]

I think your config should be:

DATASETS:
  TRAIN: ('ARKitScenes_train', 'ARKitScenes_val')
  TEST_BASE : ('ARKitScenes_test',) 
  TEST_NOVEL: ('ARKitScenes_test_novel',)
  CATEGORY_NAMES: ("toilet", "table", "bed", "sofa", "television", "refrigerator", "chair", "cabinet", "stove", "oven", "shelves", "sink", "machine", "bathtub")

[nightgoodl]

So CATEGORY_NAMES is exactly the training category, right?
I made some changes to dataset.py, I think it should have the same effect

    is_novel = dataset_name.endswith("_novel")
    is_test = dataset_name.endswith("_test")
    meta = MetadataCatalog.get(dataset_name)
    

    if is_novel:
        logger.info(f"Loading novel test dataset: {dataset_name}")
        category_path = "configs/category_meta.json"
        metadata = util.load_json(category_path)
        cat_ids = sorted(coco_api.getCatIds(filter_settings['category_names_novel']))
    elif is_test:
        logger.info(f"Loading base test dataset: {dataset_name}")
        cat_ids = sorted(coco_api.getCatIds(filter_settings['category_names_base']))
    else:
        logger.info(f"Loading training dataset: {dataset_name}")
        cat_ids = sorted(coco_api.getCatIds(filter_settings['category_names_base']))
    cats = coco_api.loadCats(cat_ids)
    thing_classes = [c["name"] for c in sorted(cats, key=lambda x: x["id"])]
    meta.thing_classes = thing_classes
    
    logger.info(f"Dataset {dataset_name} loaded with {len(thing_classes)} categories: {thing_classes}")

[yaojin17]

Yes, I remember CATEGORY_NAMES is the training category. In config.py, the category names for evaluation are set separately.