Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions docs/explanation/cameras.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,42 @@ Camera instances are not thread-safe. Use one thread per camera instance or add

## SharedCamera

For multi-process or multi-thread access, use `SharedCamera`. It wraps any camera and handles IPC transport automatically.
`SharedCamera` lets one publisher subprocess own a camera's exclusive hardware
connection while any number of subscribers read frames over iceoryx2. It
satisfies the same `Camera` protocol as a direct driver.

Construction uses `camera=` or `from_config()`, matching `SharedRobot`. Prefer
`from_config()` (or YAML) when sharing. A disconnected `@export_config` camera
can be passed directly to `camera=`; this exports its recipe rather than
handing an open device into the child. The publisher always opens fresh. Never
keep a direct camera connected to the same device while sharing: another
connected holder causes open failure.

```python
from physicalai.capture import SharedCamera, UVCCamera
from physicalai.capture import SharedCamera
from physicalai.config import to_config

shared = SharedCamera(UVCCamera(device="/dev/video0"))
# Prefer config-only / disconnected export — no live direct camera held open
shared = SharedCamera.from_config(
{
"class_path": "physicalai.capture.UVCCamera",
"init_args": {"device": "/dev/video0", "backend": "v4l2"},
},
)
# Equivalent after to_config(disconnected_driver):
# shared = SharedCamera.from_config(to_config(driver))
shared.connect()

# Safe to read from multiple threads/processes
frame = shared.read_latest()
```

`create_camera(..., shared=True)` remains a convenience for shareable built-ins
(`uvc`, `realsense`, `basler`) that packs a type into `SharedCamera(camera=...)`.

`from_config()` derives `service_name` as
`physicalai/camera/<ClassName>/<device_id>/frame` for any `class_path`,
including third-party drivers. Pass `service_name=` explicitly if two camera
classes share a class name and a device id.

`SharedCamera` is the recommended approach for production deployments where multiple consumers need camera frames. It avoids the need for manual synchronization and handles frame distribution efficiently.
10 changes: 5 additions & 5 deletions examples/capture/shared_camera.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"the camera, buffer management, or copy overhead.\n",
"\n",
"Because the transport layer is [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2),\n",
"any language with iceoryx2 bindings (Rust, C, C++, Python, ) can subscribe to\n",
"any language with iceoryx2 bindings (Rust, C, C++, Python, \u2026) can subscribe to\n",
"the same camera stream.\n",
"\n",
"**Why use SharedCamera:**\n",
Expand Down Expand Up @@ -81,16 +81,16 @@
"metadata": {},
"outputs": [],
"source": [
"from physicalai.capture import SharedCamera\n",
"from physicalai.capture import SharedCamera, create_camera\n",
"\n",
"# --- Edit these to match your setup ---\n",
"CAMERA_TYPE = CameraType.UVC\n",
"DEVICE_KWARGS = {\"device\": 0} # or {\"serial_number\": \"123456\"} for realsense/basler\n",
"# ---------------------------------------\n",
"\n",
"cam = SharedCamera(CAMERA_TYPE, zero_copy=True, **DEVICE_KWARGS)\n",
"cam: SharedCamera = create_camera(CAMERA_TYPE, shared=True, zero_copy=True, **DEVICE_KWARGS) # type: ignore[assignment]\n",
"cam.connect()\n",
"print(f\"Connected to {cam.device_id}\")"
"print(f\"Connected to {cam.device_id}\")\n"
]
},
{
Expand Down Expand Up @@ -219,7 +219,7 @@
"outputs": [],
"source": [
"# A second script / process can attach to the same publisher.\n",
"cam = SharedCamera.from_publisher(\"physicalai/camera/uvc/0/frame\", zero_copy=True)\n",
"cam = SharedCamera.from_publisher(\"physicalai/camera/UVCCamera/0/frame\", zero_copy=True)\n",
"cam.connect()\n",
"frame = cam.read_latest()\n",
"print(f\"Attached to existing publisher, seq={frame.sequence} shape={frame.data.shape}\")"
Expand Down
42 changes: 21 additions & 21 deletions examples/tutorials/collect_train_deploy.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@
"id": "7c7e915d",
"metadata": {},
"source": [
"# Collect Train Deploy with SO-101\n",
"# Collect \u2192 Train \u2192 Deploy with SO-101\n",
"\n",
"End-to-end workflow: collect teleoperation demos on an **SO-101** arm, fine-tune a **π0.5** policy, and deploy it back on the robot using `physicalai` runtime.\n",
"End-to-end workflow: collect teleoperation demos on an **SO-101** arm, fine-tune a **\u03c00.5** policy, and deploy it back on the robot using `physicalai` runtime.\n",
"\n",
"1. **Collect** demonstration data (teleoperation with leader/follower arms)\n",
"2. **Train** a π0.5 visuomotor diffusion policy\n",
"2. **Train** a \u03c00.5 visuomotor diffusion policy\n",
"3. **Deploy** the trained policy on hardware via `physicalai` runtime\n",
"\n",
"---"
Expand All @@ -28,16 +28,16 @@
"| | Requirement |\n",
"|--|-------------|\n",
"| **Robot** | SO-101 follower arm (+ leader arm for data collection) |\n",
"| **Cameras** | 1–2 USB cameras (UVC/RealSense/Basler compatible) |\n",
"| **Cameras** | 1\u20132 USB cameras (UVC/RealSense/Basler compatible) |\n",
"| **Training GPU** | 40 GB+ VRAM (e.g. 2*B70 or A100) |\n",
"| **Deploy machine** | Intel CPU, GPU, or NPU with OpenVINO support |\n",
"| **OS** | Linux (Ubuntu 22.04+ recommended) |\n",
"| **Python** | 3.12 or newer |\n",
"\n",
"### Install packages\n",
"\n",
"- **`physicalai`** runtime for deployment (Step 3)\n",
"- **`physicalai-train`** fine-tuning library (Step 2), distributed via [Physical AI Studio](https://github.com/open-edge-platform/physical-ai-studio)"
"- **`physicalai`** \u2014 runtime for deployment (Step 3)\n",
"- **`physicalai-train`** \u2014 fine-tuning library (Step 2), distributed via [Physical AI Studio](https://github.com/open-edge-platform/physical-ai-studio)"
]
},
{
Expand Down Expand Up @@ -87,7 +87,7 @@
"\n",
"> **Note:** LeRobot uses `so100` as the robot type for both SO-100 and SO-101 hardware.\n",
"\n",
"Aim for **50–100 demonstrations** of the target task with varied object positions."
"Aim for **50\u2013100 demonstrations** of the target task with varied object positions."
]
},
{
Expand All @@ -97,11 +97,11 @@
"source": [
"## Step 2: Train a Policy\n",
"\n",
"Train a π0.5 (pi-zero-point-five) visuomotor diffusion policy on the collected demonstrations.\n",
"Train a \u03c00.5 (pi-zero-point-five) visuomotor diffusion policy on the collected demonstrations.\n",
"\n",
"### Option A: Physical AI Studio (recommended)\n",
"\n",
"Launch a training job from the web UI select your dataset, choose the π0.5 architecture, and configure training parameters visually.\n",
"Launch a training job from the web UI \u2014 select your dataset, choose the \u03c00.5 architecture, and configure training parameters visually.\n",
"\n",
"Recommended training parameters:\n",
"| Parameter | Value |\n",
Expand All @@ -113,7 +113,7 @@
"| Compile model | true |\n",
"\n",
"\n",
"> **Note:** Training π0.5 requires at least **40 GB of VRAM** (e.g. NVIDIA A100).\n",
"> **Note:** Training \u03c00.5 requires at least **40 GB of VRAM** (e.g. NVIDIA A100).\n",
"\n",
"### Option B: `physicalai-train` library"
]
Expand All @@ -131,13 +131,13 @@
"from physicalai.policies import Pi05\n",
"from physicalai.train import Trainer\n",
"\n",
"# ── Config ──\n",
"# \u2500\u2500 Config \u2500\u2500\n",
"# Physical AI Studio: ~/.cache/physicalai/datasets/<dataset-id>/<dataset-name>\n",
"# LeRobot: data/<repo_id> (relative to where you ran the collection script)\n",
"DATASET_PATH = \"~/.cache/physicalai/datasets/<dataset-id>/<dataset-name>\"\n",
"CHECKPOINT_DIR = \"./checkpoints\"\n",
"\n",
"# Fine-tune π0.5 from the pretrained base\n",
"# Fine-tune \u03c00.5 from the pretrained base\n",
"model = Pi05(\n",
" pretrained_name_or_path=\"lerobot/pi05_base\",\n",
" dtype=\"bfloat16\",\n",
Expand Down Expand Up @@ -182,7 +182,7 @@
"source": [
"### Export to OpenVINO\n",
"\n",
"If using **Physical AI Studio**, the model is automatically exported to OpenVINO format when you download it no extra step needed.\n",
"If using **Physical AI Studio**, the model is automatically exported to OpenVINO format when you download it \u2014 no extra step needed.\n",
"\n",
"If training via the library, export manually:"
]
Expand Down Expand Up @@ -226,15 +226,15 @@
"metadata": {},
"outputs": [],
"source": [
"# Discover available cameras use the device IDs from this output in the config below\n",
"# Discover available cameras \u2014 use the device IDs from this output in the config below\n",
"from physicalai.capture import discover_all\n",
"\n",
"for driver, devices in discover_all().items():\n",
" if not devices:\n",
" continue\n",
" print(f\"\\n[{driver}]\")\n",
" for dev in devices:\n",
" print(f\" {dev.device_id} {dev.name}\")"
" print(f\" {dev.device_id} \u2014 {dev.name}\")"
]
},
{
Expand All @@ -256,7 +256,7 @@
"# LeRobot: ~/.cache/calibration/so101_follower.json\n",
"SO101_CALIBRATION = \"~/.local/share/physicalai/robots/<robot-uuid>/calibrations/<cal-uuid>.json\"\n",
"\n",
"# Cameras use the same names and setup as during dataset collection\n",
"# Cameras \u2014 use the same names and setup as during dataset collection\n",
"CAMERAS = [\n",
" (\"overhead\", \"uvc\", \"/dev/video0\"),\n",
" (\"arm\", \"uvc\", \"/dev/video1\"),\n",
Expand Down Expand Up @@ -308,14 +308,14 @@
"source": [
"from typing import Any\n",
"\n",
"from physicalai.capture import SharedCamera\n",
"from physicalai.capture import SharedCamera, create_camera\n",
"from physicalai.robot import SO101\n",
"\n",
"# Cameras\n",
"cameras = {}\n",
"for name, driver, device_id in CAMERAS:\n",
" kwargs: dict[str, Any] = {\"serial_number\": device_id} if driver == \"realsense\" else {\"device\": device_id}\n",
" cam = SharedCamera(driver, **kwargs, width=CAMERA_WIDTH, height=CAMERA_HEIGHT, fps=CAMERA_FPS)\n",
" cam: SharedCamera = create_camera(driver, shared=True, width=CAMERA_WIDTH, height=CAMERA_HEIGHT, fps=CAMERA_FPS, **kwargs) # type: ignore[assignment]\n",
" cam.connect()\n",
" cameras[name] = cam\n",
" print(f\"Camera '{name}' connected: {cam.actual_width}x{cam.actual_height} @ {cam.actual_fps}fps\")\n",
Expand All @@ -333,7 +333,7 @@
"source": [
"### Build and run the policy runtime\n",
"\n",
"`RobotRuntime` runs the control loop: observe infer send actions at the target FPS. The action source (`PolicySource`) adapts the model + execution + action-queue pipeline.\n",
"`RobotRuntime` runs the control loop: observe \u2192 infer \u2192 send actions at the target FPS. The action source (`PolicySource`) adapts the model + execution + action-queue pipeline.\n",
"\n",
"> **Safety:** The robot will begin moving immediately. Keep your hand near the power switch or e-stop."
]
Expand Down Expand Up @@ -361,10 +361,10 @@
")\n",
"\n",
"with runtime:\n",
" print(f\"Running policy at {FPS} fps for {DURATION_S}s task: {TASK!r}\")\n",
" print(f\"Running policy at {FPS} fps for {DURATION_S}s \u2014 task: {TASK!r}\")\n",
" steps = runtime.run(duration_s=DURATION_S)\n",
"\n",
"print(f\"\\nDone {steps} steps, {policy_source.action_queue.total_pops} actions popped\")"
"print(f\"\\nDone \u2014 {steps} steps, {policy_source.action_queue.total_pops} actions popped\")"
]
},
{
Expand Down
8 changes: 5 additions & 3 deletions src/physicalai/capture/camera.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,16 @@ class Camera(ABC):
def __init__(
self,
*,
color_mode: ColorMode = ColorMode.RGB,
color_mode: ColorMode | str = ColorMode.RGB,
) -> None:
"""Store requested capture parameters.

Args:
color_mode: Pixel format for colour image reads.
color_mode: Pixel format for colour image reads. Accepts
:class:`ColorMode` or its string value so
:func:`~physicalai.config.instantiate` round-trips work.
"""
self._color_mode = color_mode
self._color_mode = ColorMode(color_mode)
self.__executor: ThreadPoolExecutor | None = None

# ------------------------------------------------------------------
Expand Down
2 changes: 2 additions & 0 deletions src/physicalai/capture/cameras/basler/_camera.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@
from physicalai.capture.camera import Camera, ColorMode
from physicalai.capture.errors import CaptureError, CaptureTimeoutError, NotConnectedError
from physicalai.capture.frame import Frame
from physicalai.config import export_config

if TYPE_CHECKING:
import numpy as np

from physicalai.capture.discovery import DeviceInfo


@export_config(class_path="physicalai.capture.BaslerCamera")
class BaslerCamera(Camera):
"""Basler camera using pypylon SDK."""

Expand Down
2 changes: 2 additions & 0 deletions src/physicalai/capture/cameras/realsense/_camera.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,13 @@
from physicalai.capture.errors import CaptureError, CaptureTimeoutError, NotConnectedError
from physicalai.capture.frame import Frame
from physicalai.capture.mixins.depth import DepthMixin
from physicalai.config import export_config

if TYPE_CHECKING:
from physicalai.capture.discovery import DeviceInfo


@export_config(class_path="physicalai.capture.RealSenseCamera")
class RealSenseCamera(DepthMixin, Camera):
"""RealSense color and depth camera."""

Expand Down
2 changes: 2 additions & 0 deletions src/physicalai/capture/cameras/uvc/_camera.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,15 @@

from physicalai.capture.camera import Camera, ColorMode
from physicalai.capture.cameras.uvc._camera_setting import CameraSetting # noqa: PLC2701
from physicalai.config import export_config

if TYPE_CHECKING:
from physicalai.capture.cameras.uvc.v4l2 import V4L2Camera
from physicalai.capture.cameras.uvc.v4l2._controls import V4L2CameraControls
from physicalai.capture.frame import Frame


@export_config(class_path="physicalai.capture.UVCCamera")
class UVCCamera(Camera):
"""Camera facade for UVC devices (USB Video Class).

Expand Down
Loading
Loading