backward-cp

This repository contains the code for reproducing the experiments and figures presented in the paper Backward Conformal Prediction.

Organization

The repository is structured into three main folders, each corresponding to one of the three experiments presented in the paper: binary classification (Breast Cancer Wisconsin), image classification with constant size constraint rule (CIFAR-10-constant), and image classification with adaptive size constraint rule (CIFAR-10-adaptive). Each folder is self-contained and independent of the others.

Instructions

Binary classification

1. Run the binary-classification.ipynb notebook. This notebook loads and preprocesses the data, trains an XGBoost classifier, applies backward conformal prediction, and compares the results with the standard conformal prediction method.

2. The resulting plot from the backward conformal prediction process is saved in the plots/ directory.

Constant size constraint rule

1. (Optional) Run the model_train.ipynb notebook to re-train the model f. This will download and store the CIFAR-10 dataset in the data/ subfolder.

2. Execute the constant-size-constraint-rule.ipynb notebook to reproduce the experiments from the paper. The notebook will load the CIFAR-10 dataset from the data/ subfolder (downloading it if necessary). You can manually adjust the calibration size and the constant size constraint rule.

3. The resulting adaptive miscoverage values and leave-one-out estimators are saved in the alphas/ subfolder, and the generated plots are stored in the plots/ subfolder.

Adaptive size constraint rule

1. (Optional) Run the model_train.ipynb notebook to re-train the model f. This will download and store the CIFAR-10 dataset in the data/ subfolder.

2. Execute the adaptive-size-constraint-rule.ipynb notebook to reproduce the experiments from the paper. The notebook will load the CIFAR-10 dataset from the data/ subfolder (downloading it if necessary). You can manually adjust the calibration size. The size constraint rule is defined adaptively, following the approach described in the paper. An additional notebook illustration-rule.ipynb is provided to illustrate how the size constraint rule works in practice.

3. The resulting adaptive miscoverage values and leave-one-out estimators are saved in the alphas/ subfolder, and the generated plots are stored in the plots/ subfolder.