Dynamic person tracking new detections #690
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
653
to
657
| # Connect person tracker inputs | ||
| if self.person_tracker: | ||
| self.person_tracker.image.connect(self.connection.color_image) | ||
| self.person_tracker.detections.connect(self.detection_module.detections) | ||
| self.person_tracker.target.connect(self.local_planner.path) |
There was a problem hiding this comment.
self.local_planner.path is already connected to the global planner earlier in _deploy_navigation (line 531). The new connection self.person_tracker.target.connect(self.local_planner.path) overwrites that single In connection, so the local planner no longer receives the global path for ordinary navigation. After this change the robot cannot follow normal goal paths unless the person tracker is actively publishing, effectively breaking standard navigation.
Useful? React with 👍 / 👎.
Comment on lines
284
to
287
| def _publish_tf(self, msg): | ||
| self.odom.publish(msg) | ||
| self.tf.publish(*self._odom_to_tf(msg)) |
There was a problem hiding this comment.
_odom when publishing TF
_publish_tf now publishes odometry but no longer assigns self._odom. get_odom() still returns self._odom (line 300), which remains None after startup. Any skill calling UnitreeGo2.get_odom() or NavigationSkillContainer.tag_location will fail because there is never a cached pose.
Useful? React with 👍 / 👎.
Comment on lines
244
to
246
| # Publish camera info and pose synchronized with video | ||
| timestamp = msg.ts if msg.ts else time.time() | ||
| self._publish_camera_info(timestamp) |
There was a problem hiding this comment.
_on_video used to call _publish_camera_pose; the new version omits that call, so /go2/camera_pose is never emitted even though downstream modules expect it (SpatialMemory wires connection.camera_pose at lines 575‑580). Consumers depending on camera pose data will silently stop receiving updates.
Useful? React with 👍 / 👎.
spomichter
commented
Nov 19, 2025
| logger = setup_logger(__name__) | ||
|
|
||
|
|
||
| class WebInput(Module): |
…gration-paul # Unitree Agents2 Skill Integration ## Overview This PR integrates the Agents2 framework with the Unitree robot skill system, enabling AI agent-based control and human interaction capabilities. ## Key Changes ### Agent Framework Enhancements - **Agent Core Improvements** (`dimos/agents2/agent.py`) - Improved message handling with support for human messages and special outputs - Enhanced state serialization with datetime support - Added agent lifecycle management with proper stop mechanisms ### Human Interaction CLI - **New Human CLI Interface** (`dimos/agents2/cli/human_cli.py`, `dimos/agents2/cli/human.py`) - Interactive command-line interface for human operators - Real-time skill monitoring and control - Support for direct human input into agent conversations - Visual feedback for skill execution states ### Skill Coordination Improvements - **Enhanced Skill Coordinator** (`dimos/protocol/skill/coordinator.py`) - Improved async event handling with lazy initialization - Better loop context management for multi-threaded environments - Added proper cleanup and unsubscribe mechanisms - Enhanced error reporting and debugging capabilities ### Hardware Integration - **New Webcam Module** (`dimos/hardware/webcam.py`) ### Testing Infrastructure - **Comprehensive Test Suite** - Added fixture-based testing with recorded agent conversations - New test files for agent functionality (`test_agent.py`, `test_agent_direct.py`, `test_agent_fake.py`) - Integration tests for Unitree skill container - Webcam agent testing capabilities ### Unitree Integration Components - **Skill Container** (`dimos/robots/unitree_webrtc/unitree_skill_container.py`) - Bridge between agent framework and Unitree robot - WebRTC-based communication support - Skill deployment and management ### Utilities and Infrastructure - **Core Module Updates** (`dimos/core/module.py`) - Improved module lifecycle management - Better async/sync boundary handling - **LCM Protocol Updates** (`dimos/protocol/pubsub/lcmpubsub.py`) - Enhanced message serialization - Better error handling for None messages ## Bug Fixes - Resolved test loop synchronization problems - Fixed JSON encoding for datetime objects - Improved error handling in skill execution Former-commit-id: 9706cbe [formerly b41850b] Former-commit-id: 3ff0cc8
* zed local node * Update dimos/hardware/webcam.py Co-authored-by: Paul Nechifor <paul@nechifor.net> * Update dimos/hardware/webcam.py Co-authored-by: Paul Nechifor <paul@nechifor.net> * fix missing import --------- Co-authored-by: Paul Nechifor <paul@nechifor.net> Former-commit-id: 947b3cd [formerly 78dc8c9] Former-commit-id: 6a55068
…odel standardization, reid system
detection pipeline rewrite, embedding, vl model standardization, reid system
## small features around this
- Implements a retry decorator
- Better VL model JSON query system with retry, validation & extraction
- Better VL model bounding box query parsing, Detection2D output directly from vl model
- New modular detection type structure, moved detection2d to detection/ since they are 3d also
```
├── detection2d
│ ├── base.py
│ ├── bbox.py
│ └── person.py
└── detection3d
├── base.py
├── bbox.py
└── pointcloud.py
```
## embedding model standardization
`models/embedding/`
```py
class EmbeddingModel(ABC, Generic[E]):
@AbstractMethod
def embed(self, *images: Image) -> E | list[E]:
def embed_text(self, *texts: str) -> E | list[E]:
def compare_one_to_many(self, query: E, candidates: list[E]) -> torch.Tensor:
def compare_many_to_many(self, queries: list[E], candidates: list[E]) -> torch.Tensor:
def query(self, query_emb: E, candidates: list[E], top_k: int = 5) -> list[tuple[int, float]]:
def warmup(self) -> None:
```
implements
`mobileclip.py`
`clip.py`
`treid.py (torch re-id)`
- all of those conform to the same interface type and are passing the same tests
## vlm image query standardization
`models/vl`
requires drivers to implement:
```py
class VlModel:
def warmup(self) -> None
def query(self, image: Image, query: str, **kwargs) -> str:
```
provides:
```py
class VlModel:
def query_detections(self, image: Image, query: str) -> ImageDetections2D:
```
- JSON validation, extraction and retry system (in case of goofy responses)
- vlm queries now return standard Detction2D types directly
- adds new moondream model integration
`qwen.py`
`moondream.py`
## initial re-id system and module
Module takes detections and image
Outputs annotations and stable detections
```py
class ReidModule(Module):
detections: In[Detection2DArray] = None # type: ignore
image: In[Image] = None # type: ignore
annotations: Out[ImageAnnotations] = None # type: ignore
stabledetections: Out[Detection2DArray] = None # TODO! easy but not in yet - for people follow etc to have a stable target
```
can plug in any IDSystem
```py
class IDSystem(ABC):
def register_detection(self, detection: Detection2DBBox) -> int: # returns a long term id
```
EmbeddingIDsystem can host anything that takes an image and outputs a multidimensional vector, uses this to cluster frames detections.
Uses timestamp syncing to reconstruct Detection2D types from ros detection type + actual image frame
## random
includes different .direnv files (to auto enter nix env, auto enter venv etc)
Former-commit-id: b8eec2e [formerly 3e44e70]
Former-commit-id: 7ef7fe0
spomichter
force-pushed
the
dynamic-person-tracking-new-detections
branch
from
b38df79 to
0057747
Compare
|
Too many files changed for review. |
1 similar comment
|
Too many files changed for review. |
spomichter
force-pushed
the
dynamic-person-tracking-new-detections
branch
from
0057747 to
e40f213
Compare
|
Too many files changed for review. |
1 similar comment
|
Too many files changed for review. |
spomichter
force-pushed
the
dynamic-person-tracking-new-detections
branch
from
e40f213 to
17bb897
Compare
|
Too many files changed for review. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.