Original Git repository: https://github.com/PhysicalAI-AIM/Robot_synthetic_data_generation_workshop.git
End-to-end pipeline for robot manipulation on AMD GPUs (ROCm): Synthetic Data Generation → VLA Training → Simulation Evaluation.
Verified on CDNA3 (MI300 series), RDNA4 (Radeon AI PRO R9700), and RDNA3.5 (Radeon PRO W7900).
┌──────────────────────────────┐ ┌─────────────────────┐ ┌──────────────────────────┐
│ 01_gen_data_custom_scene.py │ │ 02_train_vla.py │ │ 03_eval_custom_scene.py │
│ kitchen GLB + floor_origin │ │ │ │ │
│ up + wrist cameras │─▶│ SmolVLA fine-tune │─▶│ Closed-loop eval │
│ 100 episodes, GPU render │ │ on LeRobot dataset │ │ in Genesis kitchen sim │
│ │ │ HF checkpoint out │ │ success rate + video │
└──────────────────────────────┘ └─────────────────────┘ └──────────────────────────┘
Franka 7-DOF lerobot/smolvla_base render → VLA → PD
pick red cube freeze vision encoder action chunking
2 cameras (up/wrist) train expert + state_proj randomized cube pos
| Path A: CDNA3 (MI300 series) | Path B: RDNA4/3.5 (R9700 / W7900) | |
|---|---|---|
| Data Generation | Skip — use pre-built HuggingFace dataset | End-to-end 01_gen_data_custom_scene.py kitchen scene (100 episodes) |
| Training | Local training | Local training |
| Evaluation | CPU render (llvmpipe) | GPU render (radeonsi) |
| Total time | ~20 min (train + eval) | ~30 min (gen + train + eval) |
For MI300/MI325 nodes without GPU graphics pipeline. The 100-episode dataset is pre-generated on RDNA4 and pulled from HuggingFace.
For RDNA nodes with GPU graphics pipeline. All three steps run live during the pipeline — no external dataset needed.
SmolVLA models are downloaded from ModelScope (China-friendly mirror):
| Model | ModelScope URL |
|---|---|
| SmolVLA base (450M) | https://modelscope.cn/models/lerobot/smolvla_base |
| SmolVLM2-500M backbone | https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-500M-Video-Instruct |
The pre-built image
crpi-zi7ws1z1ui8z86ek.cn-shanghai.personal.cr.aliyuncs.com/amd_image/genesis-amd:latestalready caches both models. do not need to download manually.
docker exec -it genesis-amd bash
# Step 0: Download kitchen scene assets (~130 MB, first time only)
python scripts/00_download_kitchen.py --mesh-only
# Step 1: Data Generation — kitchen scene + up/wrist cameras (~10-20 min)
python scripts/01_gen_data_custom_scene.py \
--scene rustic_kitchen --anchor floor_origin \
--camera-layout up_wrist \
--n-episodes 100 --seed 42 \
--repo-id local/franka-kitchen-wrist-100ep
# Step 2: Training (~10-20 min)
python scripts/02_train_vla.py \
--dataset-id local/franka-kitchen-wrist-100ep \
--pretrained lerobot/smolvla_base \
--n-steps 8000 --batch-size 4 --num-workers 4 \
--run-name smolvla_kitchen_wrist
# Step 3: Evaluation — kitchen scene (~5 min)
python scripts/03_eval_custom_scene.py \
--checkpoint output/train/smolvla_kitchen_wrist/final \
--dataset-id local/franka-kitchen-wrist-100ep \
--scene rustic_kitchen --anchor floor_origin \
--camera-layout up_wrist \
--n-episodes 20 --seed 99 --record-video| Metric | R9700 (RDNA4) | W7900D (RDNA3.5) |
|---|---|---|
| Data gen success | 100% | 100% |
| Loss (start → end) | 0.671 → 0.011 | 0.67 → 0.011 |
| Peak VRAM | 2.33 GB | 2.27 GB |
| Eval success rate (GPU render, kitchen) | ~75% | ~70% (3 seeds pooled) |
Note: Both GPUs use the same kitchen+wrist scene — results are directly comparable.
The training dataset is pre-generated on RDNA4 and published on HuggingFace. The pre-built Docker image already caches it:
| Item | Value |
|---|---|
| Scene | Rustic kitchen GLB + Franka Panda picking a red cube |
| Camera layout | up (overhead) + side (wrist-mounted, eye-in-hand), 640×480 |
| Episodes / Frames | 100 / 13,500 |
| Size | ~200 MB (AV1 video, LeRobot v3.0) |
| Action space | 9-DoF joint position (7 arm + 2 finger) |
| Generated on | RDNA4 (Radeon AI PRO R9700), Genesis 0.4.5, seed=42 |
⚠️ The tensor keyobservation.images.sidestores the wrist (eye-in-hand) camera, not a world-fixed side view. Do not mix with the legacyup+world-sidedataset — key names collide but semantics differ.
git clone https://github.com/AMD-AIM/Physical_AI_Challenge.git
cd Physical_AI_Challenge
bash docker/build.sh # → genesis-amd:latestdocker run -it \
--device=/dev/kfd --device=/dev/dri --group-add video --ipc=host \
--network=host \
-e HF_HUB_OFFLINE=1 \
genesis-amd:latest bashjupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-rootAccess the notebook via notebooks.amd.com. All outputs are written to output/ under the directory.
| Notebook | Target Hardware | Description |
|---|---|---|
rdna_pipeline.ipynb |
R9700 / W7900 (RDNA4/3.5) | End-to-end pipeline, GPU render |
| Section | rdna_pipeline.ipynb |
|---|---|
| 0. Environment Setup | GPU detection + kitchen GLB download |
| 1. Data Generation | 100 ep kitchen scene generation |
| 2. VLA Training | SmolVLA post-training |
| 3. Evaluation | GPU-render closed-loop eval |
| 4. Summary | PNG / MP4 / JSON |
All outputs (checkpoints, plots, eval videos) are written to output/ in the file browser.
| Package | Version | Purpose |
|---|---|---|
genesis-world |
≥1.2.0 | Physics simulation + rendering (Taichi backend, ROCm native). |
lerobot |
≥0.6.0 | Dataset format + SmolVLA model |
torch |
≥2.1 (ROCm) | Training and inference |
transformers |
≥4.40 | SmolVLA backbone (Idefics3) |
accelerate |
latest | HuggingFace model loading |
num2words |
latest | Required by transformers SmolVLM processor |
numpy |
==2.1.2 | Required by Genesis; must match scikit-image C extension ABI |
scikit-image |
≥0.22 | Must be recompiled against numpy==2.1.2 |
xvfb |
system | Headless rendering (apt-get install) |
ffmpeg |
system | Video encoding (apt-get install) |
Hardware: CDNA3 (AMD Instinct MI300/MI325 series) or RDNA3 (AMD Radeon PRO W7900) or RDNA4 (AMD Radeon AI PRO R9700); ≥4 GB VRAM on either.
Genesis Scene LeRobot Dataset SmolVLA
┌──────────────┐ ┌───────────────┐ ┌──────────────┐
│ Franka Panda │ │ observation │ │ Vision │
│ Red Cube │──IK plan──────▶│ .state [9D] │──train──────▶│ Encoder │
│ 2 Cameras │ joint lerp │ .images.up │ │ (frozen) │
│ │ render │ .images.side │ │ │
│ Physics sim │ │ action [9D] │ │ Expert │
│ (Genesis) │ │ task (text) │ │ Layers │
└──────────────┘ └───────────────┘ │ (trainable) │
▲ scene source: │ │
│ (a) flat plane (01) │ → action │
│ (b) kitchen GLB (02) same LeRobot format │ chunk [50] │
│ │
Eval Loop: │ │
render ─────────────────────────────────── inference ─────── │ │
observe state ──────────────────────────── predict ───────── │ │
execute action[0] ──────── PD control ──── scene.step() └──────────────┘
| Architecture | EGL Renderer | Type |
|---|---|---|
| CDNA3 (MI300/MI325 series) | llvmpipe | CPU software rasterization |
| RDNA4 (R9700) | radeonsi | GPU hardware rasterization |
CDNA3 has no graphics pipeline — Genesis falls back to CPU llvmpipe for camera rendering. RDNA4 has a full graphics pipeline (radeonsi, hardware-accelerated), which is the primary source of the 3-4× data-generation speedup and eliminates the render-gap bias at evaluation.
CPU-render evaluation bias (MI300/MI325): CPU and GPU rasterizers produce visually different frames. A policy trained on GPU-rendered data but evaluated with CPU rendering shows a systematic ~20 pt lower success rate. On the kitchen+wrist main path: This is expected behaviour, not a bug. For benchmark-quality numbers, evaluate on an RDNA4 or another GPU-render node.
| Issue | Fix |
|---|---|
numpy / scikit-image ABI mismatch (numpy.dtype size changed) |
pip install --force-reinstall "scikit-image>=0.22" "numpy==2.1.2" |
| torchcodec pip wheel links CUDA libs, fails on ROCm | bash setup_torchcodec.sh (CPU-only build) |
- LeRobot — Robot learning framework (dataset + policies)
- Genesis — GPU-accelerated physics simulation (ROCm native via Taichi)
- SmolVLA — Vision-Language-Action model
- World Labs Marble — 3D scene generation for custom environments
- AMD ROCm Documentation