Shortcut Audit Toolkit SURCA 2026

Award: https://www.youtube.com/live/b7zUINbHg14?si=S2vh_or8GvM80QNH&t=2296 Website: https://surca.wsu.edu/

Reproducible research pipeline for measuring shortcut-learning susceptibility in computational pathology by injecting a synthetic color-border cue and tracking how strongly models rely on it over training.

Release notes

This public repository is intended to expose the runnable toolkit, documentation, and data-preparation helpers. Large datasets, generated outputs, caches, previews, and draft-only internal writing are intentionally excluded from version control.

What this project does

• Trains binary image classifiers on histopathology datasets with optional class-correlated border cues.
• Evaluates each checkpoint on three aligned test variants:
  • clean
  • matched cue
  • flipped cue
• Logs per-epoch shortcut-emergence metrics and final per-condition summaries.
• Computes shortcut risk scores from emergence speed and final reliance strength.
• Produces plots, including a dataset x model risk heatmap and a speed-vs-strength decomposition chart.

Setup

Create a Python environment and install the dependencies from requirements.txt.

Example on Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Example on macOS or Linux:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Supported datasets and models

Registered datasets:

• CRC
• MHIST
• DHMC

Supported models:

• resnet18
• efficientnet_b0
• googlenet
• densenet121
• convnext_tiny

Current shortcut metrics assume binary classification.

Expected dataset format

Each dataset should be prepared in ImageFolder format:

dataset_root/
  train/
    class0/
    class1/
  val/
    class0/
    class1/
  test/
    class0/
    class1/

Default roots come from the dataset registry:

• CRC -> crc_binary/
• MHIST -> mhist_binary/
• DHMC -> dhmc_binary/

Use --dataset_root to override the default path.

Main workflow

1. Run one condition at a time for a chosen dataset/model/mitigation.
2. Let each run append to output/experiment_results.csv and output/emergence_curves.csv.
3. Compute shortcut risk scores with python score_risks.py.
4. Generate plots and the final heatmap with python -m shortcut_audit.plotting.plot_results.

This is the recommended workflow when you want to run models and datasets independently, then combine everything later into a single heatmap.

Pipeline summary:

• run_experiment.py is the root CLI entry point and forwards into shortcut_audit.experiments.run_experiment.
• The experiment orchestrator resolves dataset/model settings, optional sweep mode, and output locations.
• shortcut_audit.training.injection_crc is the current shared training engine for all registered binary datasets.
• Training writes one final-row table (experiment_results.csv) and one per-epoch table (emergence_curves.csv).
• score_risks.py converts final-run metrics into risk_scores.csv and risk_summary.csv.
• shortcut_audit.plotting.plot_results reads the experiment and scoring CSVs and writes figures into output/plots/.

Quick start

Verify the CLI is available:

python run_experiment.py --help

Run one baseline condition:

python run_experiment.py --dataset CRC --model resnet18

Run MHIST with an explicit dataset root:

python run_experiment.py --dataset MHIST --dataset_root mhist_binary --model googlenet

Prepare MHIST into ImageFolder format before running:

python .data_prep/prepare_mhist_binary.py --images_dir /path/to/mhist/images --annotations /path/to/annotations.csv --out_dir mhist_binary --overwrite

The MHIST prep helper expects binary labels HP and SSA in the annotation file and will raise a clear error if you accidentally point it at a different dataset.

Run DHMC using the default registered root:

python run_experiment.py --dataset DHMC --model resnet18

Run a quick smoke test:

python run_experiment.py --dataset CRC --model resnet18 --smoke --skip_preview --no_checkpoints

That smoke run performs a short 2-epoch validation pass and is the recommended first command to confirm your environment, dataset layout, and model-loading path are working.

Quick MHIST smoke tests after preparation:

python run_experiment.py --dataset MHIST --dataset_root mhist_binary --model resnet18 --epochs 1 --max_samples_per_split 32 --batch_size 8 --num_workers 0 --skip_preview --no_checkpoints
python run_experiment.py --dataset MHIST --dataset_root mhist_binary --model googlenet --epochs 1 --max_samples_per_split 32 --batch_size 8 --num_workers 0 --skip_preview --no_checkpoints

Compute risk scores from accumulated runs:

python score_risks.py

If no results CSV exists yet, the scorer exits cleanly with a message instead of failing.

Generate plots and heatmaps:

python -m shortcut_audit.plotting.plot_results

If no experiment CSVs exist yet, the plotting command exits cleanly after reporting that there is nothing to plot.

CLI flags

Core run selection:

• --dataset: dataset key. Choices are CRC, MHIST, and DHMC.
• --dataset_root: optional path override for the prepared dataset.
• --model: model name. Choices are resnet18, efficientnet_b0, googlenet, densenet121, convnext_tiny.
• --mitigation: mitigation key. Current options are baseline, augmentation, label_smoothing, mixup, strong_weight_decay.

Shortcut injection controls:

• --p_train: probability of injecting the cue into each training image.
• --border_width: reference border width in pixels.
• --alpha: cue blend strength.
• --seed: training seed.
• --seed_test: test cue seed. Defaults to the training seed if omitted.

Runtime controls:

• --epochs: override default epoch count.
• --batch_size: override batch size.
• --lr: override learning rate.
• --num_workers: override dataloader workers.
• --max_samples_per_split: optional cap per split for faster debug runs.
• --smoke: shorthand for a 2-epoch, 500-samples-per-split validation run with checkpoints disabled.

Output controls:

• --results_path: final per-condition CSV. Default output/experiment_results.csv.
• --emergence_path: per-epoch CSV. Default output/emergence_curves.csv.
• --checkpoint_dir: where best/final checkpoints are written. Default output/checkpoints.
• --preview_dir: where preview images are written. Default previews.
• --skip_preview: skip preview image generation.
• --no_checkpoints: do not save best/final checkpoint files.
• --cluster_safe_outputs: write into output/jobs/<job_tag>/... instead of the shared output files.
• --job_tag: explicit per-job tag for --cluster_safe_outputs. Defaults to a SLURM job id when available, otherwise job_local.

Sweep controls:

• --full_grid: run the built-in grid over datasets, mitigations, models, widths, seeds, and p_train.
• --datasets: optional dataset subset to use with --full_grid.

Example commands

Single baseline run:

python run_experiment.py --dataset CRC --model resnet18 --mitigation baseline --p_train 0.25

Independent run for one model-dataset pair:

python run_experiment.py --dataset MHIST --model googlenet --mitigation mixup --p_train 0.5 --seed 1

Custom output paths for incremental collection:

python run_experiment.py --dataset CRC --model resnet18 --results_path output/experiment_results.csv --emergence_path output/emergence_curves.csv

Low-cost debugging run:

python run_experiment.py --dataset CRC --model resnet18 --epochs 1 --max_samples_per_split 64 --batch_size 32 --num_workers 0 --skip_preview --no_checkpoints

Full sweep across all registered datasets:

python run_experiment.py --full_grid

Full sweep for a subset of datasets:

python run_experiment.py --full_grid --datasets CRC MHIST

Outputs

Primary run outputs:

• output/experiment_results.csv
  • one final row per completed condition
  • includes metadata, clean/matched/flipped metrics, SET, susceptibility_slope, and AURC
• output/emergence_curves.csv
  • one row per epoch per run
  • includes emergence dynamics such as flip_rate, delta_flip_auc, prob_shift, and cue_effect_size
• output/checkpoints/
  • best and final checkpoints unless --no_checkpoints is used
• previews/
  • example clean/matched/flipped images unless --skip_preview is used

Risk scoring outputs:

• output/risk_scores.csv
  • per-run scored table with speed_score, strength_score, and risk_score
• output/risk_summary.csv
  • dataset-level and model-level aggregates

Plot outputs are written to output/plots/.

If --cluster_safe_outputs is enabled, the same files are written under output/jobs/<job_tag>/ instead of the shared top-level output paths.

Heatmap workflow

If you want the final dataset x model risk heatmap:

1. Run whichever dataset/model/mitigation conditions you want, independently and in any order.
2. Keep appending to the shared results CSVs.
3. Run python score_risks.py.
4. Run python -m shortcut_audit.plotting.plot_results.
5. Open output/plots/risk_heatmap_dataset_x_model.png.

Related output:

• output/plots/risk_decomposition_speed_vs_strength.png

Notes on reproducibility

• Training, cue sampling, and dataloader shuffle are seeded.
• Matched and flipped test variants are aligned sample-by-sample.
• If the same run_name is written multiple times, scoring and plotting keep the latest row for that run to avoid duplicate-counting.
• score_risks.py will fall back to the legacy file output/results_crc_border.csv if output/experiment_results.csv does not exist yet.

Documentation map

• docs/PROJECT_DETAILS.md: methodology, metrics, scoring, and experiment behavior
• docs/REPO_GUIDE.md: file/folder guide and practical usage flow
• docs/CHANGELOG.md: implementation-focused change history

Architecture snapshot

• run_experiment.py
  • thin root wrapper for the experiment CLI
• score_risks.py
  • thin root wrapper for risk scoring
• shortcut_audit/data/dataset_registry.py
  • dataset registry and default roots
• shortcut_audit/models/model_registry.py
  • model construction and pretrained-weight handling
• shortcut_audit/injectors/border_injector.py
  • synthetic border cue injection utilities
• shortcut_audit/training/injection_crc.py
  • current end-to-end training and evaluation engine
• shortcut_audit/scoring/risk_scorer.py
  • risk-score computation and summary aggregation
• shortcut_audit/plotting/plot_results.py
  • plotting and heatmap generation