[RFC] ConfigPipeline/AutotuneBackend: Separating compilation/benchmarking from search algorithms

[hinriksnaer]

Background

While implementing #1416, it became clear that the current infrastructure is limited in terms of:

• Reproducible autotuning steps across different compilation/benchmarking approaches
• Metrics collection
• Making different strategies search algorithm agnostic.
• Exploring ideas for custom autotuning strategies that support existing search algorithms [RFC] Helion Autotuning service for instantaneous auto-tuning times #1518

Summary

Extract the compile-and-benchmark lifecycle out of BaseSearch into a
standalone ConfigPipeline. Search algorithms hand a batch of candidate
configs to the pipeline and get scored results back. The pipeline owns
how configs are compiled and benchmarked, the search algorithm owns
which configs to try.

Problem

BaseSearch is responsible for a wide range of implementations:

1. Search logic: candidate generation, population management,
   convergence
2. Compile/benchmark infrastructure: compilation, subprocess management
   and GPU benchmarking.
3. Accuracy validation: baseline computation, tolerance checking
   Consequences:

• No standalone benchmarking. You cannot evaluate a fixed set of configs
  without subclassing BaseSearch and replicating its setup ceremony.
• Search coupled with compilation/benchmarking. Noise mitigation
  strategies (CPU pinning, SIGSTOP, etc.) require modifying BaseSearch,
  even though search algorithms don't care.
• Extensibility is tricky. Distributed autotuning, alternative
  benchmarking strategies, or reproducible A/B comparisons all require
  working around the current structure.

Design

Interface

class ConfigPipeline:
    """Compile and benchmark a batch of kernel configs."""
    def __init__(
        self,
        kernel: _AutotunableKernel,
        args: Sequence[object],
        settings: Settings,
        *,
        log: AutotuningLogger | None = None,
        accuracy_check_fn: Callable | None = None,
    ) -> None: ...
    def run(
        self,
        configs: list[Config],
        *,
        desc: str = "Benchmarking",
    ) -> list[BenchmarkResult]: ...

run() takes the full batch. The pipeline internally decides how to
schedule compilation and benchmarking based on settings — sequential,
overlapped, or future strategies. Callers just get list[BenchmarkResult]
back.

What this enables

Custom Pipelines: One search algorithm can support multiple different compilation and benchmarking strategies.

Standalone benchmarking:

with ConfigPipeline(kernel, args, settings) as pipeline:
    results = pipeline.run(fixed_configs)

Strategy comparison (same configs, different evaluation):

seq_results = ConfigPipeline(kernel, args, seq_settings).run(configs)
ovl_results = ConfigPipeline(kernel, args, ovl_settings).run(configs)
compare_rankings(seq_results, ovl_results)

Future distributed autotuning: A DistributedPipeline farms
compilation to remote workers or benchmarks on multiple GPUs. Search
algorithms call pipeline.run(configs) unchanged.
Pluggable noise mitigation: CPU pinning, SIGSTOP scheduling, CUDA
graphs, all become pipeline implementation details, invisible to search.

To do

• Create an abstraction layer for ConfigPipeline
• Refactor existing search algorithms to support modular ConfigPipeline
• Make existing or custom ConfigPipelines easily exchangeable within search algoritms
• Create an interface that supports metrics collection and noise measurement.

[jansel]

I think refactoring benchmarking into its own file makes sense.

Does it need to be a context manager? That makes things harder to use IMO.

ConfigPipeline seems like an odd name. I would expect that do change configs (not benchmark them). Maybe BenchmarkingProvider or Benchmarker or ConfigMeasurement etc.

[hinriksnaer]

I think refactoring benchmarking into its own file makes sense.

This is the general goal of the topic. The autotuner has a lot of responsibilities tangled together in the same place. This introduces a decent amount of overhead when it comes to feature additions or explorations into different compilation/benchmarking/search strategies across multiple search files.

Does it need to be a context manager? That makes things harder to use IMO.

ConfigPipeline seems like an odd name. I would expect that do change configs (not benchmark them). Maybe BenchmarkingProvider or Benchmarker or ConfigMeasurement etc.

I agree with both of these points. There is no need to add more complexity than we have to and I would hope that levels of abstraction can present themselves more organically as I make smaller incremental refactors that are based on frequent patterns we have around different search algorithms.

Once things become more decoupled, we can explore what interchangeable components we would like to support.