diff --git a/.agents/skills/runtime-adding-a-robot b/.agents/skills/runtime-adding-a-robot new file mode 120000 index 0000000..d796e82 --- /dev/null +++ b/.agents/skills/runtime-adding-a-robot @@ -0,0 +1 @@ +../../skills/runtime-adding-a-robot \ No newline at end of file diff --git a/.agents/skills/runtime-adding-an-inference-backend b/.agents/skills/runtime-adding-an-inference-backend new file mode 120000 index 0000000..fce9426 --- /dev/null +++ b/.agents/skills/runtime-adding-an-inference-backend @@ -0,0 +1 @@ +../../skills/runtime-adding-an-inference-backend \ No newline at end of file diff --git a/.claude/skills/runtime-adding-a-robot b/.claude/skills/runtime-adding-a-robot new file mode 120000 index 0000000..d796e82 --- /dev/null +++ b/.claude/skills/runtime-adding-a-robot @@ -0,0 +1 @@ +../../skills/runtime-adding-a-robot \ No newline at end of file diff --git a/.claude/skills/runtime-adding-an-inference-backend b/.claude/skills/runtime-adding-an-inference-backend new file mode 120000 index 0000000..fce9426 --- /dev/null +++ b/.claude/skills/runtime-adding-an-inference-backend @@ -0,0 +1 @@ +../../skills/runtime-adding-an-inference-backend \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..17423a2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,38 @@ +# Physical AI Runtime Agent Guide + +Physical AI Runtime is the deployment-side package for loading exported policies, connecting cameras and robots, and running control loops on hardware. + +## Repository Layout + +- `src/physicalai/capture/`: camera interfaces, concrete camera integrations, discovery, and transport support. +- `src/physicalai/robot/`: robot protocol, concrete robot integrations, connection helpers, and verification. +- `src/physicalai/inference/`: exported policy loading, adapter registry, backend adapters, preprocessing, callbacks, and runners. +- `src/physicalai/runtime/`: `PolicyRuntime`, execution modes, action queues, and control-loop behavior. +- `docs/`: mkdocs documentation. +- `tests/`: unit and integration tests. +- `skills/`: canonical agent skills for repo-specific workflows. Client adapters may expose these through `.claude/skills/`, `.agents/skills/`, or other client paths. + +## Setup + +- Install dependencies with `uv sync` from the repo root. +- Include hardware extras only when needed, such as `physicalai[realsense]`, `physicalai[basler]`, `physicalai[so101]`, or `physicalai[trossen]`. + +## Build, Test, Lint + +- Run repo hooks with `prek run --all-files`. +- Run tests with `uv run pytest`. +- Build docs with `uv run mkdocs build` when documentation changes are involved. + +## Cross-Repo Rules + +- Runtime owns the `physicalai` executable and `pai` alias. +- Training packages, including Studio's `physicalai-train`, contribute CLI subcommands through `physicalai.cli.subcommands`. +- Runtime owns the load side of the export/load contract. Studio produces artifacts; Runtime consumes them with `InferenceModel.load(...)`. +- Runtime core ships ONNX and OpenVINO inference adapters. Additional backends may come from companion distributions. +- Respect Preview markers in docs and APIs. Do not present planned APIs as shipped behavior. + +## Contribution Notes + +- Use Conventional Commits for PR titles and commits. +- Sign commits when committing changes. +- Follow `.github/copilot-instructions.md` for detailed coding standards when present. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..22d682a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +# Claude Code Instructions + +Refer to [AGENTS.md](./AGENTS.md) for repository guidance, build/test commands, and agent instructions. diff --git a/skills/runtime-adding-a-robot/SKILL.md b/skills/runtime-adding-a-robot/SKILL.md new file mode 100644 index 0000000..6290977 --- /dev/null +++ b/skills/runtime-adding-a-robot/SKILL.md @@ -0,0 +1,28 @@ +--- +name: runtime-adding-a-robot +description: Add or modify a Physical AI Runtime robot integration. Use when implementing robot hardware support, the Robot protocol, connect/disconnect behavior, get_observation, send_action, joint_names, verification, safety checks, SO-101, Trossen/WidowX, or other robot adapters. +license: Apache-2.0 +--- + +# Adding a Runtime Robot + +Use this skill for changes under `src/physicalai/robot` or tests/docs for robot integrations. + +## Workflow + +1. Inspect the `Robot` protocol and existing concrete robot packages before adding a new implementation. +2. Prefer structural typing. A robot integration must satisfy the protocol; inheritance is not required unless nearby code already uses it. +3. Implement connection lifecycle explicitly: `connect()`, `disconnect()`, and safe cleanup on errors. +4. Implement `get_observation()` with stable joint ordering and timestamp/metadata conventions that match existing robots. +5. Implement `send_action(...)` with clear units, limits, timing semantics, and safe failure behavior. +6. Add or update verification helpers and tests so users can validate hardware before running a policy. + +## Safety Requirements + +- Never skip joint limit, workspace, speed, or emergency-stop considerations in user-facing examples. +- Prefer reduced-speed first-run instructions in docs. +- Make hardware dependency errors actionable: mention missing packages, permissions, ports, udev rules, or drivers when relevant. + +## References + +- See `references/robot-protocol.md` for the protocol checklist. diff --git a/skills/runtime-adding-a-robot/references/robot-protocol.md b/skills/runtime-adding-a-robot/references/robot-protocol.md new file mode 100644 index 0000000..79d8683 --- /dev/null +++ b/skills/runtime-adding-a-robot/references/robot-protocol.md @@ -0,0 +1,18 @@ +# Robot Protocol Checklist + +Runtime robots are protocol-based. A concrete robot should provide the methods and properties used by `PolicyRuntime` and verification helpers. + +## Required Behavior + +- `connect()`: initialize hardware communication and fail with actionable errors. +- `disconnect()`: release hardware resources and be safe to call during cleanup. +- `get_observation()`: return joint state using stable joint ordering. +- `send_action(...)`: command the robot using documented units and timing semantics. +- `joint_names`: expose the exact joint order expected by observations and actions. + +## Implementation Notes + +- Keep hardware SDK imports lazy or guarded when possible so optional extras remain optional. +- Do not require inheritance unless needed by existing code. Structural typing is the intended extension mechanism. +- Ensure observation and action dimensions match `joint_names`. +- Keep verification code conservative. Users should validate joints individually before running policies. diff --git a/skills/runtime-adding-an-inference-backend/SKILL.md b/skills/runtime-adding-an-inference-backend/SKILL.md new file mode 100644 index 0000000..d8707bf --- /dev/null +++ b/skills/runtime-adding-an-inference-backend/SKILL.md @@ -0,0 +1,36 @@ +--- +name: runtime-adding-an-inference-backend +description: Add or modify a Physical AI Runtime inference backend adapter. Use when implementing RuntimeAdapter classes, backend_registry registration, ONNX, OpenVINO, Torch, ExecuTorch companion adapters, file extension detection, InferenceModel.load(...), exported artifact metadata, or the Runtime side of the Studio export/load contract. +license: Apache-2.0 +--- + +# Adding a Runtime Inference Backend + +Use this skill for changes under `src/physicalai/inference/adapters`, `InferenceModel.load(...)`, backend discovery, or exported artifact compatibility. + +## Workflow + +1. Identify the exported artifact format, file extensions, optional dependencies, and target hardware/runtime. +2. Inspect the existing ONNX and OpenVINO adapters before adding a new adapter. +3. Implement the `RuntimeAdapter` contract with explicit load, predict, and cleanup behavior. +4. Register the backend with `backend_registry.register(...)` or `register_lazy_module(...)` so auto-detection can find it without importing heavy dependencies too early. +5. Validate the adapter against the shared export/load contract in `references/export-contract.md`. +6. Add tests for registration, extension detection, missing dependency errors, and at least one load/predict path using mocks when real runtime dependencies are heavy. + +## Runtime Packaging Rules + +- Runtime core ships ONNX and OpenVINO adapters. +- Additional backends should keep optional dependencies isolated, ideally through extras or companion distributions. +- Do not add user-facing support claims unless installation and loading are documented. + +## Required Checks + +- Backend names and extensions are stable and unique. +- Missing optional dependencies raise helpful installation guidance. +- The adapter consumes Studio export metadata without changing field semantics unilaterally. +- Auto-detection remains deterministic when multiple model files are present. + +## References + +- See `references/backend-adapters.md` for adapter implementation patterns. +- See `references/export-contract.md` for the shared Studio/Runtime artifact contract. diff --git a/skills/runtime-adding-an-inference-backend/references/backend-adapters.md b/skills/runtime-adding-an-inference-backend/references/backend-adapters.md new file mode 100644 index 0000000..d77a5ad --- /dev/null +++ b/skills/runtime-adding-an-inference-backend/references/backend-adapters.md @@ -0,0 +1,27 @@ +# Backend Adapter Patterns + +Runtime adapters bridge exported model artifacts to `InferenceModel`. + +## Registration + +- Use `backend_registry.register("name", extensions=(".ext",))` for adapters safe to import eagerly. +- Use `backend_registry.register_lazy_module(...)` when the adapter has heavy or optional dependencies. +- Keep extensions available without importing heavy dependencies so export directories can be probed cheaply. + +## Adapter Behavior + +- Load model artifacts from the export directory or explicit model path. +- Convert Runtime-prepared inputs into backend-specific input objects. +- Return outputs in the shape and naming expected by `InferenceModel` postprocessing. +- Release backend resources in cleanup/close paths when the backend needs it. + +## Dependency Errors + +- Missing dependencies should raise clear messages with the package or extra to install. +- Do not import optional runtime SDKs at module import time unless they are required by the base package. + +## Testing + +- Test registry name and extension behavior. +- Test missing dependency messages. +- Mock heavy backend runtimes when CI cannot install or execute them. diff --git a/skills/runtime-adding-an-inference-backend/references/export-contract.md b/skills/runtime-adding-an-inference-backend/references/export-contract.md new file mode 100644 index 0000000..85c68f6 --- /dev/null +++ b/skills/runtime-adding-an-inference-backend/references/export-contract.md @@ -0,0 +1,22 @@ +# Export/Load Contract + +Studio produces deployment artifacts; Runtime loads them with `InferenceModel.load(...)`. + +## Required Artifact Properties + +- A backend-identifying model file exists in the export directory. +- Metadata describes the policy, backend, expected inputs, expected outputs, preprocessing/postprocessing requirements, and action chunk semantics. +- Runtime can either auto-detect the backend from file extensions or load it when the backend is explicitly provided. +- Optional backend dependencies fail with clear installation guidance. + +## Backend Ownership + +- Studio owns export implementation and export metadata generation. +- Runtime owns adapter discovery, backend loading, preprocessing, inference execution, and action selection from exported artifacts. +- Studio and Runtime must keep this contract synchronized. If this file is mirrored from Studio, CI should fail when copies diverge. + +## Compatibility Rules + +- Do not change metadata field names or semantics without coordinating Studio changes. +- Do not add a backend to user-facing instructions unless Runtime can load it in the current package or via a documented companion distribution. +- Treat numerical parity and latency validation as separate checks: parity proves correctness; latency proves deployment viability.