species accuracy

[jpvanek]

First, I love the R interface. Very helpful and I think this has great potential! I wanted to provide feedback on accuracy. It's my understand more training data will be used in the future, but for now, I'm have issues with accuracy. I'm not expecting a fix here, just providing a baseline for the future.

For test data, I'm using 50 camera trap images manually assigned as coyote (Canis latrans), 50 images manually assigned as domestic cat (Felis catus), and 50 assigned as striped skunk (Mephitis mephitis). Each species was analyzed separately. Code for the coyote test provided, only thing that changed between tests was the folder of photos.

Accuracy:
Skunk: 20%
Coyote 4%
Cat: 0%

Coyote Code and Results:

library(CameraTrapDetectoR)
library(tidyverse)

#### COYOTE #####
predictions <-
    deploy_model(
        data_dir = "Cuddeback Coyotes",
        model_type = 'species',
        recursive = FALSE,
        file_extensions = c('.jpg', '.JPG'),
        labeled = FALSE,
        make_plots = TRUE,
        plot_label = TRUE,
        output_dir = NULL,
        sample50 = T,
        write_bbox_csv = FALSE,
        score_threshold = 0.6,
        h = 307,
        w = 408,
        lty = 1,
        lwd = 2,
        col = 'red'
    )

predictions_long <- predictions %>% select(-filename) %>%
pivot_longer(
  cols = 1:77, # columns that should pivot from wide to long (unquoted)
  names_to = "species", # name of the new category column as a quoted string
  values_to = "count" # name of the new value column as a quoted string
)

predictions_long %>% group_by(species) %>% summarize(count = sum(count)) %>% arrange(desc(count))

Cat Results:

Skunk Results:

[matobler]

I just run this on a 230 image test dataset from California. Dominant species in the data are coyote, grey fox, mule deer, mountain lion and cotton tail rabbit. While I have not done any official analysis (the images have not been labeled), looking through the image plots I don't think a single species was identified correctly. The bounding boxes do fit the animals and the model did correctly identify humans and vehicles. Not sure what the issue could be. Maybe the model does not translate well to other environments than the ones where the original data came from? For reference, inference time is 7.14 seconds/image on an i7-3970X loading images form an SSD.