Train robot policies with up to 75% less data.
Calibra helps robotics teams build smaller, higher-quality training sets — catching bad demonstrations before they waste GPU time, then selecting the episodes that actually matter.
| Dataset | Quality Score | Best Retention | Result |
|---|---|---|---|
PushT (lerobot/pusht) |
76.7 | 25% | 99.5% of full-data performance with 75% less training data |
DROID-100 (lerobot/droid_100) |
77.0 | 75% | Outperformed full-data baseline (+3%) |
ALOHA sim (lerobot/aloha_sim_insertion_human) |
87.3 | Higher | Smaller gains — already a clean dataset |
xArm lift (lerobot/xarm_lift_medium) |
82.7 | — | Little benefit — already a high-quality simulation dataset |
Across four public robotics datasets, Calibra consistently preserved more rare behaviors than random selection. The magnitude of training-data reduction depended on the dataset's quality and redundancy.
Across three datasets and three policy families (BC-MLP, ACT, Diffusion Policy) at 30% retention, Calibra improves over random by +24.5% on average.
→ Full benchmark results, ablation tables, and limitations
The reason Calibra can remove 75% of demonstrations without hurting performance is that most robotics datasets contain two distinct problems: bad episodes (jerk spikes, dropped frames, sync errors) and redundant episodes (near-duplicate demonstrations of the same behavior). Calibra removes both.
The pipeline:
| Step | Question | Command |
|---|---|---|
| 1. Integrity | Can I trust this dataset? | calibra integrity |
| 2. Quality | Which episodes are clean? | calibra audit |
| 3. Coverage | Which episodes are distinct? | calibra review |
| 4. Select | Keep only what matters. | calibra prune |
$ calibra integrity /data/my_demos.h5
─── Dataset Integrity ────────────────────────────────────
my_demos · 120 episodes
Critical (1)
❌ camera_freeze_events: 1 of 120 episodes (0.8%) contain a run of ≥5
consecutive near-identical camera frames (episode ep_17).
Warnings (1)
⚠️ blurry_episode_fraction: camera frames markedly blurrier than the
rest of the dataset in 1 episode.
Passed (8)
✅ timestamp_jitter_cv ✅ timestamp_dropout_rate ✅ short_episode_fraction
✅ action_dropout_rate ✅ duplicate_frame_rate ✅ ldlj
✅ jerk_spike_rate ✅ velocity_discontinuity_rate
Integrity Score: 85/100 · Status: Warning
pip install calibra-robotics
calibra integrity /data/my_demos.h5
calibra audit lerobot/pusht
calibra prune lerobot/pusht --keep 0.25 --report results/pusht/latest.jsonNo installation required.
🔗 Calibra — Dataset Integrity (Hugging Face Space)
- Check any LeRobot dataset's integrity — timestamps, sync, completeness, duplicate/frozen/blurry frames, jittery motion
- See its Quality & Coverage score and percentile
- Compare against community benchmarks
- Download a full audit report
On real PushT data: at 10% retention, Calibra achieves lower prediction error than training on the full dataset, while random selection degrades sharply.
Ablation across 5 seeds on ALOHA mobile (keep 30%): Calibra full pipeline and diversity-only both outperform all published baselines.
Mean improvement over random selection (5 seeds, 30% retention, 3 datasets):
| Method | BC-MLP | ACT | Diffusion Policy |
|---|---|---|---|
| Diversity-only | +29.5% | +26.5% | +11.9% |
| Calibra full | +24.5% | +23.7% | +13.8% |
| K-Center | +24.0% | +23.1% | +10.1% |
| Facility Location | +21.5% | +18.4% | +8.7% |
| Random | 0.0% | 0.0% | 0.0% |
Method rankings are stable across all three policy families (Spearman ρ ≥ 0.86).
→ Full benchmarks and ablations
Calibra can record measured training results from real experiments and connect them to benchmark reports.
calibra experiment record --experiment-id my-run --condition calibra --retention 25 \
--gpu-hours 6.2 --eval-success-rate 0.88
calibra experiment list --experiment-id my-run
calibra experiment report --experiment-id my-runRun a retention sweep:
calibra benchmark --sweepConnect measured results to the benchmark:
calibra benchmark --sweep --experiment-id my-runReports distinguish simulated, partially measured, and validated case-study results so estimated compute savings are not confused with measured results.
Random selection picks a clustered subset. Calibra's coverage-based selector spreads selections across the behavioral space — ensuring the policy sees every behavioral mode, even rare ones.
Inspect dataset health, identify problematic demonstrations with root causes, and generate a training-ready coreset — all from one interface. Generated with calibra audit lerobot/columbia_cairlab_pusht_real --html-out report.html.
# 1. Record demos
lerobot-record --robot-type so100 --repo-id $HF_USER/my_dataset
# 2. Curate and write the report
calibra prune /path/to/my_dataset --keep 0.3 --report results/my_dataset/latest.json
# 3. Train on the coreset
lerobot-train policy=act dataset_repo_id=./my_dataset_coresetfrom calibra.integrations.lerobot import load_dataset
ds = load_dataset("lerobot/pusht", report_path="results/pusht/latest.json")
# ds is a datasets.Dataset with only Calibra-approved episodesfrom calibra.integrations.isaac_lab import export_gr00t_manifest, filter_hdf5
export_gr00t_manifest("results/franka/latest.json", demos_path="demos.hdf5")
filter_hdf5("demos.hdf5", "results/franka/latest.json", "demos_coreset.hdf5")calibra prune demos.hdf5 --keep 0.3 --policy gr00t --report results/franka/latest.json
python -m gr00t.train --manifest gr00t_manifest.json --demo-file demos_coreset.hdf5from calibra.ingestion.registry import load
from calibra.pipeline import Pipeline
from calibra.pruning import CoresetSelector
batch = load("lerobot/pusht")
report = Pipeline().run(batch, policy_family="diffusion")
selector = CoresetSelector(keep_fraction=0.3)
result = selector.select(batch, report)
# result.keep_episode_ids → filter your dataset| Command | Description |
|---|---|
calibra integrity |
"Can I trust this dataset?" — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion (--decode-images for LeRobot v1) |
calibra audit |
Full diagnostic report with bootstrap CIs and per-episode outlier detection |
calibra review |
Ranked episode review queue — separates anomaly, quality-risk, and coverage-value signals |
calibra prune |
Two-stage coreset: quality filter + greedy max-coverage selection |
calibra certify |
Structured CERTIFIED / PROVISIONAL / NOT CERTIFIED; --json for CI |
calibra predict |
Estimate training outcome before spending GPU time |
calibra watch |
Real-time quality feedback during teleoperation |
calibra score |
Composite 0–100 score across Quality, Synchrony, Coverage, Task Structure |
calibra compare |
Evidence-backed cross-dataset comparison with falsifiable claims |
calibra corrupt |
Inject synthetic corruptions to validate metric sensitivity |
calibra card |
Generate a HuggingFace dataset quality card |
calibra sim2real |
Quantify sim-to-real distribution gap and transfer risk |
calibra transfer |
Cross-embodiment compatibility scoring |
calibra cure |
Automatic data remediation (smoothing, resampling, trimming) |
calibra audit-all |
Bulk-audit an entire HF org; writes CalibraReport JSONs |
calibra site |
Generate a static leaderboard website from audit results |
calibra serve |
Local REST API server and web dashboard |
calibra benchmark |
Compare full, random, and Calibra-selected datasets across training-data retention levels |
calibra experiment |
Record and report measured training results such as GPU-hours and evaluation success |
v0.7.1 (current) — Dataset Integrity: calibra integrity — timestamps, sync, episode completeness, duplicate/frozen/blurry camera frames, jittery/jerky motion, with LeRobot v1 image support via --decode-images. Full details in CHANGELOG.md.
Next — Vision Integrity for video-backed LeRobot (v2/v3): decode sampled frames from LeRobot's mp4-encoded v2/v3 datasets so duplicate-frame/camera-freeze/blur detection work there too.
PyPI package name:
calibra-robotics(thecalibraname on PyPI is an unrelated package)
pip install calibra-robotics # core (numpy + pydantic only)
pip install 'calibra-robotics[lerobot]' # LeRobot / HuggingFace Hub (recommended)
pip install 'calibra-robotics[hdf5]' # HDF5 (Isaac Lab, Robomimic)
pip install 'calibra-robotics[rlds]' # RLDS / TF Datasets
pip install 'calibra-robotics[mcap]' # MCAP / ROS2 bags
pip install 'calibra-robotics[all]' # everythingFormats supported: LeRobot v1/v2/v3 (Parquet), HuggingFace Hub IDs, HDF5 (Isaac Lab, Robomimic), RLDS/TF Datasets, MCAP/ROS2 bags.
Camera-frame checks (duplicate_frame_rate, camera_freeze_events, blurry_episode_fraction in calibra integrity) work out of the box on HDF5/Isaac Lab/robomimic data, and on LeRobot v1 datasets via calibra integrity <path> --decode-images (opt-in — decodes HuggingFace Image-feature columns, off by default since it increases load time/memory). Not yet supported for LeRobot v2/v3 (video-encoded).
Coming soon. The central empirical finding — that the optimal coreset selection strategy depends on the data-retention budget — will be described in full detail.
Available after paper release.
Calibra is not open to external pull requests or contributions at this time.
git clone https://github.com/omertt27/Calibra
pip install -e '.[all,dev]'
pytest # 679 tests
ruff check . # zero errors expectedBusiness Source License 1.1 — free for research and internal use, converts to Apache 2.0 on 2030-06-30. Commercial hosting requires a separate license. See LICENSE and LICENSING.md. Contact: omertahtaci05@gmail.com