Write image augmentation as independent transforms. Execute compatible geometry with fewer resampling passes.
fuse-augmentations is a PyTorch tensor-first engine for reducing repeated interpolation in image augmentation pipelines. It recognizes a finite set of Kornia, TorchVision, and Albumentations transforms—or builds a pipeline directly from numeric ranges—then composes compatible transform matrices before pixels are sampled.
You keep the readable pipeline: rotate, scale, shear, translate, flip. The engine finds compatible runs and can replace several geometric warps with one.
Important
This package is Alpha and is not a general drop-in replacement for native Compose containers. It does not guarantee native pixels, input types, target processors, random streams, hooks, or universal speedups.
Warning
Use auxiliary targets only with explicitly supported spatial transforms. An unknown crop, resize, or other spatial passthrough can modify the image while leaving a mask, box, or keypoint tensor stale. Treat every Unknown ... SPATIAL_KERNEL barrier warning as unsafe when data_keys is present.
A conventional chain may interpolate the same pixels after every geometric operation:
Conventional chain — 4 resampling passes
Input ──▶ Rotate ─warp 1─▶ Scale ─warp 2─▶ Shear ─warp 3─▶ Translate ─warp 4─▶ Output
Fused compatible chain — 1 resampling pass
Input ──▶ sample parameters ──▶ M = M_translate · M_shear · M_scale · M_rotate
──▶ one sampling grid ─warp 1─▶ Output
Repeated interpolation adds work, creates intermediate tensors, and can progressively discard high-frequency detail. Matrix composition is cheap by comparison: for a compatible run, the package samples the individual parameters, multiplies their homogeneous matrices in declared order, and evaluates the result through one sampling grid.
This does not mean the fused output is pixel-identical to the native chain. It is a different resampling strategy, and backend centers, fill rules, clipping, and interpolation conventions still matter.
The same fixed Kornia rotation → scale → shear recipe is evaluated below. The native route resamples three times; Fuse Compose composes the geometry and samples once. The red/green/yellow overlay makes local disagreement visible without claiming native-pixel parity.
See three fixed recipes for each of Kornia, TorchVision, and Albumentations, including their exact limits.
| Capability | What is implemented |
|---|---|
| Affine fusion | Registered rotation, affine, shear, translation, scale, and exact discrete operations are grouped within compatible same-backend runs. |
| Exact geometry | Supported flips and discrete operations use lossless tensor paths where the segment contract permits it. |
| Projective fusion | Consecutive registered perspective transforms compose as 3×3 homographies; affine↔projective transitions remain boundaries. |
| Linear color fusion | Supported brightness, contrast, brightness/contrast-only ColorJitter, and standard RGB Normalize paths can collapse into color-matrix segments. |
| Crop-resize | Registered RandomResizedCrop has a dedicated segment; a preceding affine run can be absorbed on Kornia/TorchVision torch paths. |
| Flexible construction | Native numeric ranges, Kornia transforms, TorchVision transforms, Albumentations transforms, or a mixed-backend list. |
| Portable configuration | Frozen TransformSpec values resolve a declarative pipeline against a chosen backend with strict unsupported-operation handling. |
| Auxiliary coordinates | Masks, dense xyxy/xywh boxes, and dense keypoints can follow supported fused matrices, subject to the safety limits below. |
| NumPy bridges | HWC/BHWC NumPy ↔ BCHW torch converters and NumPy output are available; conversion to NumPy detaches and moves data to CPU. |
| Execution controls | Albumentations cv2 or torch execution, optional torch compilation on non-CPU paths, Kornia-dependent downscale antialiasing, interpolation, padding, and color clipping policies. |
| Plan inspection | Human-readable plans, structured descriptors, a warp-saving estimate, and the last matrix-producing segment are exposed. |
| Training integration | Pipelines are nn.Module objects and have tested pickle/serialization paths for common worker use. |
This is an allowlist, not a claim about every upstream transform:
| Backend | Registered geometry | Registered linear color | Crop-resize |
|---|---|---|---|
| Kornia | RandomRotation, RandomAffine, RandomShear, RandomTranslate, H/V flip, RandomRotation90, RandomPerspective |
RandomBrightness, RandomContrast, brightness/contrast-only ColorJitter, 3-channel Normalize |
RandomResizedCrop |
| TorchVision v1/v2 | RandomRotation (expand=False), RandomAffine, H/V flip, RandomPerspective |
Brightness/contrast-only ColorJitter, 3-channel Normalize |
RandomResizedCrop |
| Albumentations | Affine, Rotate, SafeRotate, ShiftScaleRotate, H/V flip, RandomRotate90, D4, Transpose, Perspective |
RandomBrightnessContrast, standard 3-channel Normalize |
RandomResizedCrop with backend-specific execution limits |
| Native/direct | Rotation, scale, x/y shear, x/y translation, H/V flip | Brightness, contrast | Not available |
Per-channel non-linear scalar maps — gamma, solarize, posterize, and supported per-channel equalize — are registered (Kornia RandomGamma/RandomSolarize/RandomPosterize/RandomEqualize; TorchVision RandomSolarize/RandomPosterize/RandomEqualize; Albumentations RandomGamma/Solarize/Posterize/unmasked, by_channels=True Equalize). Static maps compose into a table; equalize builds a separate per-image, per-channel histogram table at its runtime position, so static neighbours remain fused on either side. The uint8 Albumentations native path and the TorchVision equalize table match their native sequential operations exactly; Kornia's installed equalize accepts floating tensors only. The float tensor path retains an interpolation tolerance: it detects large lookup jumps and switches to a sharp step rule there, rather than smearing a discontinuity across a grid cell. Residual non-linear barriers include cross-channel saturation/hue and Albumentations masked or luminance (by_channels=False) equalize, which are not per-channel scalar maps.
Other unknown and nonlinear operations generally become passthrough barriers. That preserves pipeline construction in many image-only cases, but passthrough is not automatically numerically transparent, device-efficient, or auxiliary-target safe.
Gaussian blur is a narrow exception: consecutive Gaussian blurs fold into one operation, and a Gaussian blur commutes to the end of the run when the affine that immediately follows it does not downscale, letting that affine run collapse to a single warp (the surrounding affines then fuse as any affine chain does). Axis-aligned affine scales use Kornia's native Gaussian primitive; rotated (or sheared-and-upscaled) affines use a normalized sampled full-covariance Gaussian with 3-sigma support capped at 63 pixels. A pure shear has a smallest singular value below one, so it is refused like any other downscale — only a shear combined with enough upscale to keep both singular values at or above one reaches the covariance path. The blur stays a barrier when the following affine downscales (its smallest singular value drops below one), and always for projective transforms and non-linear kernels. Kornia's installed sharpness remains a barrier because it clamps intermediate values and restores borders, so it is not a linear shift-invariant kernel chain. The native Albumentations Gaussian path folds consecutive blurs only; it does not commute through affines because that fused path cannot guarantee the backend's random-number stream.
pip install fuse-augmentationsThe base package requires Python 3.10+ and PyTorch 2.2+. It includes the native/direct builder.
Install optional adapter ecosystems only when needed:
pip install "fuse-augmentations[kornia]"
pip install "fuse-augmentations[torchvision]"
pip install "fuse-augmentations[albumentations]"
pip install "fuse-augmentations[all]"import torch
from fuse_augmentations import Compose, ReorderPolicy
torch.manual_seed(7)
augment = Compose.from_params(
rotation=(-15.0, 15.0),
scale=(0.9, 1.1),
shear_x=(-4.0, 4.0),
translate_x=(-8.0, 8.0),
hflip_p=0.5,
reorder=ReorderPolicy.NONE,
)
images = torch.rand(4, 3, 128, 128, dtype=torch.float32)
augmented, matrix = augment(images, return_matrix=True)
assert augmented.shape == images.shape
assert augmented.dtype == images.dtype
assert matrix is not None
assert matrix.shape == (4, 3, 3)
print(augment.fusion_plan)
print(
[
(descriptor.kind, descriptor.n_warps_saved)
for descriptor in augment.fusion_plan_descriptors
]
)Fusion plan and saved warp count for the quick-start pipeline
fused(_DirectParamTransform, _DirectFlipTransform)
[('fused', 1)]
The common fused input contract is a floating BCHW torch tensor. Both fuse_augmentations and the shorter fuse_aug import expose the same public objects.
import torch
import torchvision.transforms.v2 as T
from fuse_augmentations import Compose, ReorderPolicy
augment = Compose(
[
T.RandomRotation(15.0),
T.RandomAffine(degrees=0.0, scale=(0.9, 1.1)),
T.RandomHorizontalFlip(p=0.5),
],
reorder=ReorderPolicy.NONE,
)
output = augment(torch.rand(8, 3, 224, 224))Kornia and Albumentations transform objects follow the same BCHW entry point when used in tensor pipelines. Albumentations also has an image-only HWC NumPy compatibility call, but it is not a replacement for native multi-target dictionaries and processors.
Mixed-backend pipelines are supported, with every backend change acting as a hard fusion boundary.
These numbers are from the 2026-07-12 local audit on macOS arm64, fuse-augmentations 0.9.0.dev0, Python 3.12, PyTorch 2.10, 256×256 inputs. They demonstrate the shape of the opportunity—not a release-wide promise.
| Measurement | Observed result | Interpretation |
|---|---|---|
| Fixed 45-case CPU, batch-1 score | 1.79× geometric mean of native/fused latency | Real aggregate gain for the synthetic bank; mixed cases use opt-in reordering. |
| Five-op geometric chain, CPU batch 1 | 6.52× Kornia, 14.48× TorchVision in one quick run | Long geometric chains are the strongest fit; quick-run effect sizes are noisy. |
| Sampled CPU tensor peak memory | Lower in 12/12 compared pairs | Fewer intermediate warped tensors can reduce peak; allocation count improved in only 8/12. |
| TorchVision 3-op, CPU batch 8 peak | 117.5 MB → 38.0 MB (~3.1× lower) | One profiler result, not a universal memory ratio. |
| Apple MPS quick sweep | Faster in only 9/28 comparable Kornia/TorchVision pairs | GPU-capable does not mean faster; TorchVision MPS often regressed here. |
| CUDA | Not tested | No CUDA performance claim is made by this audit. |
Single operations can be slower because there is no resampling to eliminate. Color-heavy pipelines retain color cost. Small accelerator workloads can be dominated by launch, sampling, compilation, or conversion overhead. Always benchmark the exact production pipeline and keep a correctness/parity gate beside the timing.
Fusing geometry changes when interpolation happens. That can preserve more detail than repeatedly warping an already warped image, but it also means the result is not a byte-for-byte substitute for the native chain.
For research or parity-sensitive use:
- set
ReorderPolicy.NONEexplicitly; - pair or record sampled parameters and matrices;
- compare output coordinates and task metrics, not only shapes;
- report backend, device, versions, dtype, image size, and batch size;
- separate compile warmup from steady-state timing;
- publish losses and skips alongside wins.
POINTWISE and AGGRESSIVE reordering are behavior-changing optimizations: moving color across geometry can alter border and clipped pixels. AGGRESSIVE currently behaves like POINTWISE.
Registered fused geometry can route:
- masks as BCHW tensors with nearest or bilinear sampling;
- boxes as dense
(B, N, 4)xyxy or xywh tensors; - keypoints as dense
(B, N, 2)tensors.
Important boundaries:
- unknown spatial transforms may desynchronize image and targets;
- mask padding is always zero even when image padding is border/reflection;
- nearest masks are intentionally detached; bilinear requires floating soft masks;
- boxes are AABB-wrapped after rotation but are not clipped or filtered;
- labels, visibility, ragged instances, invalid-box removal, and keypoint validity are application responsibilities;
- the Albumentations HWC NumPy path is image-only.
For detection and segmentation, validate every transform class and warning before training.
fusion_plandescribes the current segment structure.fusion_plan_descriptorsprovides structured, serializable segment metadata.n_warps_savedis a plan estimate, not a literal native interpolation counter for exact operations.return_matrix=Trueandtransform_matrixexpose the last matrix-producing segment, not an automatic whole-pipeline matrix across backend, projective, crop, or passthrough boundaries.
Use per-call matrix return when output and transform provenance must stay paired.
For one fused affine or projective geometric segment, pass the matrix returned by the same call to inverse to map a prediction back into the original frame. This pairing is safe for concurrent calls; inverse deliberately does not read the mutable transform_matrix property.
import torch
from fuse_augmentations import Compose
augment = Compose.from_params(translate_x=(2.0, 2.0))
images = torch.rand(1, 3, 16, 16)
prediction_augmented, matrix = augment(images, return_matrix=True)
prediction_original = augment.inverse(prediction_augmented, matrix=matrix)
assert prediction_original.shape == images.shapeWith data_keys, pass the augmented auxiliary targets in the same positional order; masks use the matching sampling grid and boxes/keypoints use the inverse pixel matrix. Keypoints and masks recover to sampling precision, but bounding boxes are axis-aligned: a forward-then-inverse box is exact only for axis-aligned transforms (flip, scale, translation) and inflates under a rotation, shear, or projective warp. inverse raises instead of guessing for crop-resize (cropped pixels are lost), color/LUT/blur or passthrough segments, exact-only segments, multiple segments, or a missing paired matrix. It is geometric-only and cannot recover values discarded by interpolation or padding.
Use fuse-augmentations when:
- data is already a BCHW torch tensor;
- the pipeline contains several registered geometric transforms;
- repeated resampling is a fidelity, memory, or throughput concern;
- you can validate output semantics for the exact backend and task.
Prefer the native backend container when you require:
- PIL or unbatched CHW input;
- full Albumentations dictionary processors;
- exact native centers, fills, pixels, hooks, or RNG behavior;
- unsupported spatial transforms with masks, boxes, or keypoints;
- backend-specific per-transform interpolation semantics; per-transform border modes are available with
padding_mode="per_transform"when they map exactly, while opaque modes stay native boundaries with a warning and the default override is unchanged.
The repository includes a complete MkDocs Material site:
- Overview
- Installation and backend-free quickstart
- How fusion works
- Exact capabilities
- Backend and configuration guides
- Auxiliary-target safety
- Reproducibility
- Quality and benchmark evidence
- Research methodology
- Known limitations
- FAQ
- Application walkthroughs
- generated references for the notable public API
The site configuration provides local search, per-page descriptions, canonical/Open Graph metadata, sitemap and crawler files, an llms.txt agent index, and GitHub Pages publication automation.
Benchmark and memory scripts live in experiments/. They expose cases where fusion loses as well as wins.
uv run --all-extras --group benchmark python experiments/optimize_score.py
uv run --all-extras --group benchmark python experiments/bench_gpu_batch.py --quick
uv run --all-extras --group benchmark python experiments/bench_memory.py --quickTreat quick runs as smoke evidence. Release-grade comparisons need independent processes, uncertainty intervals, paired RNG state, output-parity assertions, and full environment provenance.
Bug reports and focused pull requests are welcome. Open an issue before a public API or architecture change.
Documentation example authoring and generated-test instructions are in CONTRIBUTING.md. Generated documentation tests are recreated in CI and should not be committed.
Build the docs locally with:
uv sync --group docs
uv run --group docs mkdocs build --strictApache-2.0 © 2025–2026 Jiri Borovec.