Hyperparameter Optimization Framework: FABOLAS vs. Hyperband

This project implements and compares two advanced algorithms for Hyperparameter Optimization (HPO) on deep learning models: FABOLAS and Hyperband. The framework is built in Python using PyTorch for the models, and BoTorch/GPyTorch for the Bayesian Optimization components of FABOLAS.

Table of Contents

1. Overview
2. Implemented Algorithms
3. Project Structure
4. Installation
5. Running Experiments
6. Configuration Examples
7. Results

Overview

Finding optimal hyperparameters is a critical yet computationally expensive step in the machine learning lifecycle. This project provides a flexible framework to compare the efficiency of two powerful HPO strategies on various models and datasets.

• Hyperband: A bandit-based approach that accelerates random search through an efficient early-stopping strategy called Successive Halving.
• FABOLAS: A model-based Bayesian Optimization algorithm that learns a surrogate model of performance and cost as a function of both hyperparameters and dataset subset size.

The primary goal is to benchmark these methods based on their real-world runtime performance in finding high-quality hyperparameter configurations.

Implemented Algorithms

Hyperband

Hyperband prunes poorly performing configurations early, allowing it to allocate more computational resources to promising ones. It achieves this by running multiple brackets of the Successive Halving algorithm, each exploring a different trade-off between the number of configurations evaluated and the resources allocated to each.

FABOLAS

FABOLAS (Fast Bayesian Optimization on Large Datasets) intelligently navigates the hyperparameter space by building two Gaussian Process (GP) models: one to predict model performance (loss) and another to predict computational cost (time). It uses a custom acquisition function to decide which configuration to evaluate on which subset size to maximize the information gained per unit of time.

Project Structure

The codebase is organized into the following key modules:

• main.py: The main entry point to launch all experiments.
• experiments.py: Manages the experiment execution loop, result saving, and checkpointing.
• configs.py: The central configuration file. Defines optimizers, budgets, search spaces, and datasets.
• hpo.py: Contains the base HPO class inherited by specific optimizers.
• hyperband.py: Implementation of the Hyperband algorithm.
• fabolas.py: Implementation of the FABOLAS algorithm.
• MLP.py / ResCNN.py: Definitions of the neural network architectures to be optimized.

Installation

1. Clone the repository:

   git clone [YOUR-REPOSITORY-URL]
   cd [YOUR-REPOSITORY-NAME]

2. Create a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install dependencies:

   pip install torch torchvision scikit-learn numpy matplotlib tqdm gpytorch botorch

Running Experiments

All experiments are controlled via the configs.py file.

1. Configure the Experiment

Modify the configs dictionary in configs.py to specify:

• "OPTIMIZER": A list of the optimizer classes to run (e.g., [HyperbandHPO, FabolasHPO]).
• "RESOURCES": The primary budget type (e.g., "time" for a fair, time-based comparison).
• "OPT_HYPERPARAMS": A list of dictionaries containing parameters for each optimizer.
• "DOMAINS": The hyperparameter search space for your model.
• "DATASET": The dataset to use (e.g., "mnist", "cifar10").
• "N_RUNS": The number of independent repetitions for each experiment.

2. Launch the Experiment

Once configured, run the main script from the project root:

python main.py

The framework will execute all defined experiments, printing progress to the console.

Configuration Examples

Example 1: Running Hyperband on MNIST

This configures Hyperband to run on MNIST, allocating a maximum of 81 epochs to the best configuration within its brackets.

# in configs.py
configs = {
    "OPTIMIZER": [HyperbandHPO],
    "RESOURCES": ["epochs"],
    "OPT_HYPERPARAMS": [{"R": 81, "eta": 3}],
    "DOMAINS": {
        "learning_rate": (1e-4, 1e-1, "continuous"),
        "num_layers": (2, 5, "integer"),
        # ... other hyperparameters ...
    },
    "DATASET": "mnist",
    "N_RUNS": 5,
}

Example 2: Comparing FABOLAS and Hyperband on CIFAR-10

This configuration sets up a direct comparison on CIFAR-10, giving each optimizer a total time budget of 1 hour (3600 seconds).

# in configs.py
configs = {
    "OPTIMIZER": [HyperbandHPO, FabolasHPO],
    "RESOURCES": ["time", "time"],
    "OPT_HYPERPARAMS": [
        {   # Hyperband parameters
            "max_resources": 3600,
            "eta": 3
        },
        {   # FABOLAS parameters
            "max_resources": 3600,
            "n_configs": 20,
            "resource_per_config": 20,
        }
    ],
    "DOMAINS": { # Example for a CNN
        "learning_rate": (1e-5, 1e-2, "continuous"),
        # ... other hyperparameters ...
    },
    "DATASET": "cifar10",
    "N_RUNS": 3,
}

Results

Experiment results are saved as JSON files in the results/ directory. Each file contains detailed information about the run, including the best configuration found, its final performance metrics, and the total runtime. These files can be used for post-hoc analysis and visualization.