UncertainBird - 🤗

This project addresses the challenge of uncertainty estimation in AI-drivenbird sound classification, essential for the ”DeepBirdDetect” project aimed at harmonizing wind power expansion with avian conservation. We aim to evaluate methods such as Monte Carlo Dropout, Spectral-normalized Neural GaussianProcess, and Focal Loss within deep learning frameworks, assessing their performance across various neural network architectures, including CNNs and Trans-formers, and model scales. Our findings will provide insights into the suitability of these uncertainty estimation techniques for environmental conservation applications, offering a basis for more reliable and transparent AI-based wildlife monitoring.

User Installation

The simplest way to install $\texttt{UncertainBird}$ is to clone this repository.

You can also use the devcontainer configured as as git submodule:

git submodule update --init --recursive

And install python dependencies with uv.

uv sync

To experiment with Perch_v2 install TF specific dependencies separately:

uv pip install -r tf-requirements.txt

Activate virtual environment:

eval ./venv/bin/activate

Dump predictions for calibration analysis

python ./uncertainbird/scripts/dump_predictions.py  --model <model> --dataset <dataset_names> --gpu <gpu_id> --output-dir <output_dir> --num-workers <num_workers> 

For example:

python ./uncertainbird/scripts/dump_predictions.py  --model convnext_bs --datasets NBP HSN --gpu 0 --output-dir ./logs/predictions --num-workers 1

Experiments

Benchmarking the Calibration of Bird sound Classifiers

See notebooks in uncertainbird/benchmarking_calibration for details. This notebook provides an overview of the experiments and results in the paper.

Platt & Temperature Scaling

See notebooks in uncertainbird/notebooks/posthoc_calibration for details.