Inference Routing and Planning

Inference Routing & Planning

Section: AI & Inference · Media-Playback-System · Home · ONNX-Execution-Providers

Inference routing selects and validates execution providers for pipeline stages. Provider registered ≠ model ready. Windows preference: TensorRT RTX plugin, Windows ML catalog routes, DirectML fallback.

Core Architecture and Components

The planning system is built around a centralized logic that evaluates model metadata against a "Hardware Matrix" to determine the optimal execution path.

Key Components

Component	Description
IRuntimePlanner	The primary interface for determining if a stage can run and selecting the best execution route.
HardwareMatrix	A structured representation of the local machine's capabilities, including detected GPUs and supported Execution Providers.
ModelManifest	Authoritative metadata for a model, including expected_runtime, engine_family, and licensing information.
CommercialSafeEvaluator	Validates that a selected route complies with commercial licensing restrictions.

Runtime Planning Workflow

The following diagram illustrates the decision flow when a pipeline stage requests a runtime plan.

flowchart TD
    Start[Stage Execution Request] --> GetManifest[Fetch Model Manifest]
    GetManifest --> CheckLicense{Commercial Mode?}
    CheckLicense -- Yes --> EvalLicense[Evaluate Model License]
    EvalLicense --> CheckHardware[Query Hardware Matrix]
    CheckLicense -- No --> CheckHardware
    
    CheckHardware --> FilterEPs[Filter Supported EPs]
    FilterEPs --> Ranking[Rank EPs by Performance Tier]
    Ranking --> SmokeTest[Verify Execution Provider Path]
    
    SmokeTest -- Success --> Ready[Return Ready Status]
    SmokeTest -- Missing Files --> Blocked[Return Blocked: Download Required]
    SmokeTest -- Fail --> Fallback[Attempt Legacy Fallback]

Loading

This workflow ensures that engines do not hardcode routing internally but instead rely on the planner to provide a verified execution path.

Hardware Strategy and Execution Providers

Trackdub adopts a platform-specific strategy for hardware acceleration, particularly on Windows, where it distinguishes between legacy and modern acceleration paths.

Windows Execution Providers

On Windows, the system prioritizes Windows ML and TensorRT RTX as modern integration surfaces.

• TensorRT RTX: Utilized via a standalone ONNX Runtime EP ABI plugin (NvTensorRTRTXExecutionProvider).
• Windows ML: Used for certified catalog EPs (MIGraphX, OpenVINO, QNN).
• DirectML: Maintained exclusively as a legacy GPU fallback.

Hardware Matrix Logic

The HardwareMatrix is responsible for fingerprinting the machine and tracking registered EPs.

classDiagram
    class HardwareMatrix {
        +List~GpuInfo~ DetectedGpus
        +List~string~ RegisteredProviders
        +UpdateCapabilities()
        +IsProviderAvailable(string providerId)
    }
    class RuntimePlanner {
        +GetPlan(StageRequest request)
        +VerifyModelReadiness(ModelId id)
    }
    RuntimePlanner --> HardwareMatrix : Queries

Loading

Readiness and Validation States

To prevent "fake readiness," the system tracks fine-grained states for every model and provider combination. A model is not considered ready simply because its class exists; it must pass multiple validation gates.

Readiness Gates

The planner evaluates the following conditions before a stage can execute:

1. Provider Registered: The EP (e.g., DirectML) is known to the runtime.
2. Runtime Installed: Required native binaries (e.g., ONNX DLLs) are present.
3. Model Manifest Present: The model is registered in the authoritative bundled-models.manifest.json.
4. Checksum Verified: Model files in the cache match the manifest hashes.
5. Hardware Provider Available: The selected GPU or NPU is actually accessible.
6. Commercial eligibility: The model license allows for the current project mode.

Status Outcomes

When a plan is generated, it returns structured outcomes rather than simple booleans.

Status	Meaning
Ready	All gates passed; execution can proceed.
Blocked	A mandatory requirement (like a model file) is missing.
Skipped	The stage was bypassed (e.g., low confidence or user toggle).
Fallback	The preferred high-performance route failed, falling back to CPU or DirectML.

Implementation Details (Optional)

The RuntimePlanner utilizes the StageRuntimeRequirementsCatalog to align stages with their necessary hardware and model profiles.

// Example of how the planner might resolve a request
// Path: src/Trackdub.Inference/Runtime/Planning/RuntimePlanner.cs
public async Task<RuntimePlan> ResolvePlanAsync(RuntimeStage stage, CancellationToken ct)
{
    var requirements = _catalog.GetRequirements(stage);
    var manifest = _registry.GetManifest(requirements.DefaultModelId);
    
    // Check hardware matrix and license evaluator
    var ep = _hardwareMatrix.GetBestProvider(manifest.ExpectedRuntimes);
    // ... validation logic ...
}

Conclusion

Inference Routing & Planning provides the necessary abstraction to decouple high-level pipeline logic from low-level hardware concerns. By enforcing strict manifest-driven routing and honest readiness states, it ensures Trackdub remains a stable and predictable workstation regardless of the underlying hardware (CPU, NVIDIA RTX, or AMD). This system is foundational for the "local-first" mission, allowing users to leverage their hardware's full potential while maintaining data privacy.