Multi Object Pick and Place

[noshluk2]

1. Environment Description

File : ros2 launch panda panda_multi_object_sequence.launch.py in m6_to_m8 branch

• Simulation environment containing:

  • A robotic arm mounted on a table.
  • Three target objects placed with slight spacing:
    • Cube
    • Rectangular block
    • Square block
  • Additional small spherical objects placed randomly in front of or between the main objects to act as clutter/obstacles.

• A depth camera mounted on the end-effector (hand) of the robotic arm, oriented downward toward the table.

• Point Cloud Templates (PCD files) will be provided for each target object shape (cube, rectangle, square).

---

2. Task Objective

Develop a perception-driven manipulation pipeline that:

• Detects and identifies target objects using point cloud matching with provided PCD templates.
• Builds a scene 3d map from the depth camera points.
• Inserts non-target objects into the MoveIt planning scene as obstacles.
• Plans and executes a pick-and-place sequence for the target objects.

---

3. Expected System Capabilities

• Perform point cloud segmentation to isolate individual objects on the table.
• Identify objects by matching segmented point clouds with provided PCD templates.
• Determine object order based on spatial sequence (e.g., left → right).
• Maintain a scene representation for motion planning.

---

4. Tentative Procedure

• Acquire depth point cloud from the end-effector camera and build 3d map.

• Perform point cloud preprocessing:

  • Downsampling
  • Plane segmentation to remove table

• Perform object segmentation to isolate clusters.

• For each cluster:

  • Match against provided PCD templates.
  • Classify object type (cube, rectangle, square).

• Add **non-target clusters ** to the MoveIt planning scene as collision obstacle objects.

• Determine object picking order (e.g., leftmost first).

• For each identified target object:

  • Compute grasp pose
  • Plan motion using MoveIt
  • Execute pick operation
  • Place object at specified target pose

• Repeat until all target objects are placed.

---

5. Deliverables

• Working simulation demonstrating perception + manipulation pipeline.
• Integration with MoveIt planning scene for obstacle handling.
• Demonstration of sequential pick and place of all target objects.