Cognitive-Physical Decoupling — A Session-Centered Runtime for Embodied Intelligence

_{English · 中文}

---

📢 Changelog

Version	Date	Update
	2026-07-5	Supports benchmarking for both the policy loop and target native builtin paths; Incorporates agent verification and failure recovery server
	2026-06-27	Support for Behavior 1K; SessionVerfier for Agent verification; VerifySessionTool
	2026-06-11	Cleaned protocol files and docs; game scenario separated to general-game-agent branch; main branch now focused on sim & real
	2026-06-5	Optimize the user-friendly onboarding process; Communication Protocol Specification; More reasonable coding standards; Game Agent & Benchmarking ready
	2026-05-25	Strict separation of PolicySkillRuntime / BuiltinSkillRuntime; Game Agent & Benchmarking ready
	2026-05-20	Perception plugin system: SensorConfig / PerceptionConfig YAML + EnvironmentWriter auditable writeback
	2026-05-18	Session-Centered Runtime MVP: DummySimTarget + DummyAdapter + DummyClient serial pipeline
	2026-04-29	Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline

---

🤔 Why PhyAgentOS?

Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:

🔌	One Codebase, Any Hardware — Adding a new robot means implementing one Target Adapter (~100 lines); zero changes to the scheduling layer.
🛡️	Three Safety Layers — Critic validation → Strict Preflight → Target-side SafetyGuard; mandatory for real-robot deployment.
📋	Fully Auditable — State, actions, and perception results are written to Markdown + YAML files; every step is traceable and reproducible.
🔄	Zero-Friction Migration — The same Session protocol runs identically across sim and real targets.

_{▲ Session-Centered Runtime Architecture Overview}

---

✨ Core Features

🔄	Session-Centered Runtime	WatchdogSupervisor → SessionRunner → SkillRuntime → TargetSessionHandle execution pipeline, replacing the legacy Driver-Center architecture
🎯	Target-Configured	Three target kinds — debug / simulation / real_robot — registered in TARGETS.md, adapters attached on demand
🧩	Adapter + Bridge	TargetAdapter + PolicyAdapter + ActionBridge three-way decoupling with explicit observation/action contracts; AdapterPlan auto-composed, eliminating target×skill combinatorial explosion
⚡	Dual Skill Runtimes	PolicySkillRuntime maintains the policy closed loop + BuiltinSkillRuntime manages built-in loops such as target-native benchmarks and constrained agent interaction
🛡️	Strict Preflight	Runtime validation checks (target / sensor / perception / adapter contract / action contract / tool); failures are rejected before execution starts
📝	File Protocol Matrix	TARGETS.md · SKILLRUNTIME.md · SESSIONS.md · ENVIRONMENT.md · LESSONS.md + external YAML configs
🔐	Multi-Layer Safety	Critic validation → Preflight contract checks → Target-side SafetyGuard → Operator Override
🌐	Fleet Mode	Multi-robot coordination with shared + per-robot workspaces, priority-based serial scheduling

---

🚀 5-Minute Quick Start

1	Install git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS pip install -e . # Python ≥ 3.11 pip install -e ".[dev]" # Dev dependencies
2	Initialize Workspace paos onboard
3	Start Agent paos agent
4	Optional: Connect Runtime Services The following example evaluates the official PI0.5 policy on the libero_spatial suite through LiberoBenchmarkSkillRuntime, with episode replanning enabled. Run the three terminals on the same host from the repository root. # Terminal 1: LIBERO TargetWS MUJOCO_GL=egl PYTHONWARNINGS=ignore \ conda run --no-capture-output -n libero \ python PhyAgentOS/runtime/targets/remote/libero/server.py \ --host 0.0.0.0 --port 9002 \ --camera-height 256 --camera-width 256 \ --max-steps 300 --num-steps-wait 10 \ --control-mode relative --seed 7 # Terminal 2: official OpenPI PI0.5 policy server conda run --no-capture-output -n openpi \ python -m PhyAgentOS.runtime.policy.openpi.native_openpi_server \ --policy-config pi05_libero \ --checkpoint-dir gs://openpi-assets/checkpoints/pi05_libero \ --host 0.0.0.0 --port 8000 # Terminal 3: Agent, Watchdog, and Verification Service paos agent --workspace ~/.PhyAgentOS/workspace -m \ "Evaluate PI0.5 on LIBERO suite libero_spatial. Use target libero_real_remote, the libero_target_benchmark builtin runtime, target_native execution, recovery verification, task ids 0-9, init-state ids 0-49, max_steps 300, replan_every_steps 5, and policy endpoint openpi://127.0.0.1:8000."

Before starting Terminal 3, enable libero_real_remote, configure Runtime verification, and set the Target configuration to retry_instruction_mode: original. The latter keeps the original task instruction on recovery attempts; use verifier_rewrite when the policy should receive the verifier's nonempty rewrite instead. See the Runtime configuration reference for the complete global, Target, SkillRuntime, Session, benchmark, verification, and remote-host parameter reference. The LIBERO setup is included as an example.

---

🗂️ Protocol Files

Context Loading	File	Owner	Purpose
Always loaded into the agent system prompt	AGENTS.md	Agent workspace	Project-level operating rules for the agent
Always loaded into the agent system prompt	SOUL.md	Agent workspace	Identity, high-level behavior, and assistant style
Always loaded into the agent system prompt	USER.md	Agent workspace	User preferences and durable profile notes
Always loaded into the agent system prompt	TOOLS.md	Agent workspace	Tool usage policy and available tool guidance
Always loaded into the agent system prompt	SKILLS.md	Agent workspace	Agent-facing skill discovery and loading rules
Loaded when present; filtered by enabled runtime targets where applicable	EMBODIED.md	Agent workspace	Human-readable target capability descriptions
Loaded when present as state, not bootstrap policy	ENVIRONMENT.md	Agent/runtime workspace	Current target and scene/environment state
Loaded when present as memory/state	LESSONS.md	Agent workspace	Operational lessons and failure notes
Loaded when present as task state	TASK.md	Agent workspace	Multi-step task decomposition and progress
Runtime protocol; read before scheduling sessions	RUNTIME.md	Runtime workspace	Instructions for writing valid runtime sessions
Runtime protocol; read before scheduling sessions	TARGETS.md	Runtime workspace	Enabled targets, endpoint/adapter/config references, supported skill runtimes
Runtime protocol; read before scheduling sessions	SKILLRUNTIME.md	Runtime workspace	Policy/builtin skill runtime registry and execution contracts
Runtime queue/state; written by Agent and watchdog	SESSIONS.md	Runtime workspace	Pending/running/completed execution sessions and results

SKILLS.md is for agent capabilities and skill discovery. SKILLRUNTIME.md is for runtime execution contracts; it is paired with TARGETS.md and SESSIONS.md.

---

📦 Project Structure

PhyAgentOS/
│
├── PhyAgentOS/agent/          # Track A  ─  Planner / Critic / Memory
│
├── PhyAgentOS/runtime/        # Track B  ─  Execution Plane
│   ├── watchdog/              #   WatchdogSupervisor
│   ├── sessions/              #   SessionRunner / TargetSessionHandle
│   ├── targets/               #   RolloutTarget (debug·sim·real)
│   │   └── remote/libero/     #   LIBERO benchmark TargetWS server + proxy
│   ├── skillruntime/          #   PolicySkillRuntime / BuiltinSkillRuntime
│   ├── adapters/              #   TargetAdapter / PolicyAdapter / Bridge
│   │   ├── libero/            #   LIBERO target adapter
│   │   └── openpi/            #   OpenPI policy adapters
│   ├── policy/openpi/         #   OpenPI client + LeRobot pi0-family server
│   ├── perception/            #   Perception Runtime / EnvironmentWriter
│   ├── preflight/             #   RuntimeCompatibilityPreflight
│   └── schemas/               #   Pydantic Schema
│
├── configs/runtime/           # Sensor / Perception / Contract YAML
├── scripts/                   # Utility scripts
├── workspace/                 # Agent workspace; runtime files may share it by config
├── docs/                      # Documentation
└── tests/                     # Tests

---

🏷️ Supported Targets

	Kind	Location	Examples
🐛	debug	Local	echo / mock / dry-run — zero-hardware protocol pipeline validation
🧪	simulation	Remote	RoboCasa, LIBERO — benchmark evaluation & batch experience mining
🤖	real_robot	Remote	Franka, Go2, XLeRobot, AgileX PIPER — real-world deployment

All targets are registered in TARGETS.md, identified by target_adapter:// URI. More examples & demos → Project Website

---

📖 Documentation

Document	Audience	Description
🌐 Website	Everyone	Full docs, architecture details, demos
📘 User Manual	Users	Installation, deployment, and operation guide
📙 Dev Guide	Developers	Secondary development, hardware integration, plugin authoring

---

🤝 Contributing

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

---

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory & X-Era Lab

       
_{MIT License · Copyright © 2025-2026 PhyAgentOS}