Open-set segmentation runs but no image/scene graph generated in RViz (Ubuntu 24.04, ROS 2 Jazzy)

[Lixiangqi2002]

Hi,

First of all, thank you very much for the amazing work and for open-sourcing this project!

I am using Ubuntu 24.04 with ROS 2 Jazzy, and I am particularly interested in the open-set segmentation pipeline.

• I followed all steps in the khronos README and can launch the system successfully without open-set segmentation.
• I also followed the full README in semantic_inference and verified that it works correctly in isolation.

However, when it came to open-set segmentation, I am not seeing any segmentation image displayed in RViz. Specifically, when running with open-set semantics enabled, RViz launches correctly, but no /display_image or scene graph generated as the output appears.

The open-set segmentation node appears to initialize successfully. The Khronos pipeline also starts and runs well. There are two messages printed red as errors:

[khronos_node-3] E khronos_pipeline.cpp:106 Active window is not khronos::ActiveWindow!
[khronos_node-3] E khronos_pipeline.cpp:117 Backend is not khronos::Backend!

However, the pipeline continues running after these messages.

Any guidance would be greatly appreciated. Thanks again for the excellent work!

P.S. the full logs is attached below:

ros2 launch khronos_ros uhumans2_khronos.launch.yaml use_gt_semantics:=false use_openset_semantics:=true
[INFO] [launch]: All log files can be found below /home/xiangqili/.ros/log/2026-02-09-19-03-07-772428-nmslap008884.client.kcl.ac.uk-1428940
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [dataset_player-1]: process started with pid [1428943]
[INFO] [open_set_node-2]: process started with pid [1428944]
[INFO] [khronos_node-3]: process started with pid [1428945]
[INFO] [hydra_visualizer_node-4]: process started with pid [1428947]
[INFO] [rviz2-5]: process started with pid [1428948]
[INFO] [static_transform_publisher-6]: process started with pid [1428949]
[INFO] [static_transform_publisher-7]: process started with pid [1428950]
[static_transform_publisher-7] [INFO] [1770663788.025549642] [odom_static_tf]: Spinning until stopped - publishing transform
[static_transform_publisher-7] translation: ('0.000000', '0.000000', '0.000000')
[static_transform_publisher-7] rotation: ('0.000000', '0.000000', '0.000000', '1.000000')
[static_transform_publisher-7] from 'map' to 'odom'
[static_transform_publisher-6] [INFO] [1770663788.025579885] [map_static_tf]: Spinning until stopped - publishing transform
[static_transform_publisher-6] translation: ('0.000000', '0.000000', '0.000000')
[static_transform_publisher-6] rotation: ('0.000000', '0.000000', '0.000000', '1.000000')
[static_transform_publisher-6] from 'world' to 'map'
[khronos_node-3] W20260209 19:03:08.098165 1428945 csv_reader.cpp:21] CSV file 'null' does not exist.
[khronos_node-3] I20260209 19:03:08.098446 1428945 hydra_pipeline.cpp:57] [Hydra] Initialized pipeline with:
[khronos_node-3] ================================ PipelineConfig ================================
[khronos_node-3] enable_lcd:                   false (default)
[khronos_node-3] timing_disabled:              false (default)
[khronos_node-3] disable_timer_output:         true (default)
[khronos_node-3] enable_pgmo_logging:          true (default)
[khronos_node-3] default_verbosity:            2
[khronos_node-3] default_num_threads:          -1 (default)
[khronos_node-3] store_visualization_details:  true
[khronos_node-3] label_names:                  [{label: 0, name: Unknown},
[khronos_node-3]                                {label: 1, name: Wall},
[khronos_node-3]                                {label: 2, name: Floor},
[khronos_node-3]                                {label: 3, name: Ceiling},
[khronos_node-3]                                {label: 4, name: Door},
[khronos_node-3]                                {label: 5, name: Stairs},
[khronos_node-3]                                {label: 6, name: Structure},
[khronos_node-3]                                {label: 7, name: Shelf},
[khronos_node-3]                                {label: 8, name: Plant},
[khronos_node-3]                                {label: 9, name: Bed},
[khronos_node-3]                                {label: 10, name: Storage},
[khronos_node-3]                                {label: 11, name: Table},
[khronos_node-3]                                {label: 12, name: Chair},
[khronos_node-3]                                {label: 13, name: Wall_Decoration},
[khronos_node-3]                                {label: 14, name: Couch},
[khronos_node-3]                                {label: 15, name: Light},
[khronos_node-3]                                {label: 16, name: Appliance},
[khronos_node-3]                                {label: 17, name: Thing},
[khronos_node-3]                                {label: 18, name: Deformable},
[khronos_node-3]                                {label: 19, name: Dynamic_NonHuman},
[khronos_node-3]                                {label: 20, name: Human}]
[khronos_node-3] map_window [Virtual Config: SpatialWindowChecker::Config]:
[khronos_node-3]    max_radius_m:              8 (default)
[khronos_node-3] frames [FrameConfig]:
[khronos_node-3]    robot_frame:               base_link_gt
[khronos_node-3]    odom_frame:                odom (default)
[khronos_node-3]    map_frame:                 map (default)
[khronos_node-3] graph [SharedDsgInfo::Config] (default):
[khronos_node-3]    layers:                    {AGENTS: {layer: 2, partition: 0},
[khronos_node-3]                                BUILDINGS: {layer: 5, partition: 0},
[khronos_node-3]                                MESH_PLACES: {layer: 3, partition: 1},
[khronos_node-3]                                OBJECTS: {layer: 2, partition: 0},
[khronos_node-3]                                PLACES: {layer: 3, partition: 0},
[khronos_node-3]                                ROOMS: {layer: 4, partition: 0}} (default)
[khronos_node-3] label_space [LabelSpaceConfig]:
[khronos_node-3]    total_semantic_labels:     21
[khronos_node-3]    semantic_colormap_file:    "null"
[khronos_node-3]    semantic_label_remap_filepath: "" (default)
[khronos_node-3]    dynamic_labels:            [19, 20]
[khronos_node-3]    invalid_labels:            [0]
[khronos_node-3]    object_labels:             [4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]
[khronos_node-3]    surface_places_labels:     [2, 5]
[khronos_node-3]    ground_labels:             [] (default)
[khronos_node-3]    semantic_layers:           [OBJECTS, MESH_PLACES] (default)
[khronos_node-3] mesh [MeshFieldConfgi] (default):
[khronos_node-3]    with_colors:               true (default)
[khronos_node-3]    with_first_seen_stamps:    true (default)
[khronos_node-3]    with_labels:               true (default)
[khronos_node-3] ================================================================================
[khronos_node-3] I20260209 19:03:08.183998 1428945 hydra_ros_pipeline.cpp:83] Starting Hydra-ROS with input configuration
[khronos_node-3] ============================ RosInputModule::Config ============================
[khronos_node-3] max_receiver_queue_size:      0 (default)
[khronos_node-3] clear_queue_on_fail:          true (default)
[khronos_node-3] inputs[left_cam] [InputModule::InputPair::Config] (default):
[khronos_node-3]    receiver [Virtual Config: OpenSetImageReceiver::Config]:
[khronos_node-3]       input_separation_s:     0.04
[khronos_node-3]       ns:                     "" (default)
[khronos_node-3]       queue_size:             10 (default)
[khronos_node-3]    sensor [Virtual Config: RosCamera::Config]:
[khronos_node-3]       min_range [m]:          0.1
[khronos_node-3]       max_range [m]:          5
[khronos_node-3]       ns:                     "" (default)
[khronos_node-3]       warning_timeout_s [s]:  10 (default)
[khronos_node-3]       error_timeout_s [s]:    0 (default)
[khronos_node-3]       latch_info_sub:         false (default)
[khronos_node-3]       extrinsics [Virtual Config: RosExtrinsics::Config]:
[khronos_node-3]          sensor_frame:        left_cam
[khronos_node-3]          robot_frame:         "" (default)
[khronos_node-3]          warning_timeout_s [s]: 10 (default)
[khronos_node-3]          error_timeout_s [s]: 0 (default)
[khronos_node-3]          wait_duration_s [s]: 0.1 (default)
[khronos_node-3]          verbosity:           3 (default)
[khronos_node-3] tf_lookup [TFLookup::Config] (default):
[khronos_node-3]    wait_duration_s:           0.1 (default)
[khronos_node-3]    buffer_size_s:             30 (default)
[khronos_node-3]    max_tries:                 5 (default)
[khronos_node-3]    verbosity:                 3 (default)
[khronos_node-3] ================================================================================
[rviz2-5] [INFO] [1770663788.233655449] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-5] [INFO] [1770663788.233712006] [rviz2]: OpenGl version: 4.6 (GLSL 4.6)
[rviz2-5] [INFO] [1770663788.251578332] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-5] [INFO] [1770663788.354504915] [rviz2]: Stereo is NOT SUPPORTED
[khronos_node-3] W20260209 19:03:08.360227 1428945 data_directory.cpp:88] Overwriting existing directory: '/home/xiangqili/code/ros2_ws/install/khronos_ros/share/khronos_ros/../output'.
[khronos_node-3] I20260209 19:03:08.360694 1428945 experiment_manager.cpp:140] [ExperimentManager] Setup output directory: '"/home/xiangqili/code/ros2_ws/install/khronos_ros/share/khronos_ros/../output/"'.
[rviz2-5] [INFO] [1770663788.366307079] [rviz2]: Stereo is NOT SUPPORTED
[khronos_node-3] I20260209 19:03:08.399088 1428945 experiment_manager.cpp:290] [ExperimentManager] Config: 
[khronos_node-3] =========================== KhronosPipeline::Config ============================
[khronos_node-3] enable_frontend_output:       true (default)
[khronos_node-3] enable_zmq_interface:         false (default)
[khronos_node-3] verbosity:                    3
[khronos_node-3] preprint_config:              false (default)
[khronos_node-3] active_window [Virtual Config: ActiveWindow::Config]:
[khronos_node-3]    max_input_queue_size:      0 (default)
[khronos_node-3]    validate_mesh_fields:      true (default)
[khronos_node-3]    verbosity:                 2 (default)
[khronos_node-3]    detach_object_extraction:  true (default)
[khronos_node-3]    min_output_separation [s]: 0 (default)
[khronos_node-3]    volumetric_map [VolumetricMap]:
[khronos_node-3]       voxel_size [m]:         0.1 (default)
[khronos_node-3]       voxels_per_side:        16 (default)
[khronos_node-3]       truncation_distance [m]: 0.2
[khronos_node-3]       with_semantics:         true
[khronos_node-3]       with_tracking:          true
[khronos_node-3]    map_window [Uninitialized Virtual Config]
[khronos_node-3]    projective_integrator [ProjectiveIntegrator]:
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       extra_integration_distance [m]: 0 (default)
[khronos_node-3]       extra_integration_distance_labels: [] (default)
[khronos_node-3]       skip_extra_colors_and_labels: false (default)
[khronos_node-3]       use_weight_dropoff:     true (default)
[khronos_node-3]       weight_dropoff_epsilon [vs]: -1 (default)
[khronos_node-3]       use_constant_weight:    false (default)
[khronos_node-3]       min_measurement_weight: 0.0001 (default)
[khronos_node-3]       max_weight:             100000 (default)
[khronos_node-3]       num_threads:            20
[khronos_node-3]       interpolation_method [Virtual Config: InterpolatorAdaptive::Config]:
[khronos_node-3]          max_depth_difference_m [m]: 0.2 (default)
[khronos_node-3]       semantic_integrator [Uninitialized Virtual Config]
[khronos_node-3]    tracking_integrator [TrackingIntegrator]:
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       temporal_buffer [s]:    1 (default)
[khronos_node-3]       burn_in_period [s]:     1 (default)
[khronos_node-3]       tsdf_occupancy_threshold [m]: -1.5 (default)
[khronos_node-3]       neighbor_connectivity:  18 (default)
[khronos_node-3]       temporal_window [s]:    3 (default)
[khronos_node-3]       num_threads:            20
[khronos_node-3]    motion_detector [Virtual Config: FreeSpaceMotionDetector]:
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       neighbor_connectivity:  26 (default)
[khronos_node-3]       min_cluster_size:       500
[khronos_node-3]       max_cluster_size:       1000000 (default)
[khronos_node-3]       min_separation_distance [voxels]: 2
[khronos_node-3]       max_range [m]:          5
[khronos_node-3]       min_z_coordinate [m]:   -10000 (default)
[khronos_node-3]       num_threads:            20
[khronos_node-3]    object_detector [Virtual Config: InstanceForwarding]:
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       max_range [m]:          5
[khronos_node-3]       min_cluster_size:       50
[khronos_node-3]       max_cluster_size:       -1 (default)
[khronos_node-3]       min_object_volume [m]:  0 (default)
[khronos_node-3]       max_object_volume [m]:  -1 (default)
[khronos_node-3]       max_background_score:   0.22
[khronos_node-3]       background [Virtual Config: RosEmbeddingGroup::Config]:
[khronos_node-3]          ns:                  /semantic_inference
[khronos_node-3]          silent_wait:         false (default)
[khronos_node-3]          prompts:             [structure, wall, floor]
[khronos_node-3]       metric [Virtual Config: CosineDistance::Config]:
[khronos_node-3]          tolerance:           1e-09 (default)
[khronos_node-3]    tracker [Virtual Config: MaxIoUTracker]:
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       track_by:               voxels
[khronos_node-3]       semantic_association:   assign_cluster (default)
[khronos_node-3]       min_semantic_iou:       0.25
[khronos_node-3]       min_cosine_sim:         0 (default)
[khronos_node-3]       min_cross_iou:          0.1
[khronos_node-3]       max_dynamic_distance [m]: 1 (default)
[khronos_node-3]       temporal_window [s]:    3 (default)
[khronos_node-3]       min_num_observations [frames]: 15
[khronos_node-3]       voxel_size [m]:         0.2
[khronos_node-3]    object_extractor [Virtual Config: MeshObjectExtractor::Config]:
[khronos_node-3]       verbosity:              1
[khronos_node-3]       min_object_allocation_confidence: 0.5 (default)
[khronos_node-3]       min_object_volume:      0.005
[khronos_node-3]       max_object_volume:      10
[khronos_node-3]       only_extract_reconstructed_objects: true
[khronos_node-3]       min_dynamic_displacement: 1
[khronos_node-3]       min_object_reconstruction_confidence: 0.5 (default)
[khronos_node-3]       min_object_reconstruction_observations: 0
[khronos_node-3]       object_reconstruction_resolution: -0.02 (default)
[khronos_node-3]       min_reconstruction_resolution: 0 (default)
[khronos_node-3]       visualize_classification: false (default)
[khronos_node-3]       projective_integrator [ProjectiveIntegrator]:
[khronos_node-3]          verbosity:           2 (default)
[khronos_node-3]          extra_integration_distance [m]: 0 (default)
[khronos_node-3]          extra_integration_distance_labels: [] (default)
[khronos_node-3]          skip_extra_colors_and_labels: false (default)
[khronos_node-3]          use_weight_dropoff:  true (default)
[khronos_node-3]          weight_dropoff_epsilon [vs]: -1 (default)
[khronos_node-3]          use_constant_weight: false (default)
[khronos_node-3]          min_measurement_weight: 0.0001 (default)
[khronos_node-3]          max_weight:          100000 (default)
[khronos_node-3]          num_threads:         8
[khronos_node-3]          interpolation_method [Virtual Config: InterpolatorAdaptive::Config]:
[khronos_node-3]             max_depth_difference_m [m]: 0.2 (default)
[khronos_node-3]          semantic_integrator [Uninitialized Virtual Config]
[khronos_node-3]       mesh_integrator [MeshIntegratorConfig]:
[khronos_node-3]          min_weight:          0.0001 (default)
[khronos_node-3]          integrator_threads:  20
[khronos_node-3]    extraction_worker [ObjectWorkerPool::Config] (default):
[khronos_node-3]       num_workers:            2 (default)
[khronos_node-3]       poll_time_us:           1000 (default)
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]    mesh_integrator [MeshIntegratorConfig]:
[khronos_node-3]       min_weight:             0.0001 (default)
[khronos_node-3]       integrator_threads:     20
[khronos_node-3]    frame_data_buffer [FrameDataBuffer] (default):
[khronos_node-3]       max_buffer_size:        300 (default)
[khronos_node-3]       store_every_n_frames:   1 (default)
[khronos_node-3] frontend [Virtual Config: GraphBuilder::Config]:
[khronos_node-3]    lcd_use_bow_vectors:       true (default)
[khronos_node-3]    mesh_resolution:           0.005
[khronos_node-3]    d_graph_resolution:        2.5
[khronos_node-3]    time_horizon:              15
[khronos_node-3]    enable_mesh_objects:       false
[khronos_node-3]    use_frontiers:             false (default)
[khronos_node-3]    no_packet_collation:       false (default)
[khronos_node-3]    verbosity:                 1
[khronos_node-3]    graph_connector [GraphConnector::Config] (default):
[khronos_node-3]       layers[0] [LayerConnector::Config] (default):
[khronos_node-3]          parent_layer:        PLACES (default)
[khronos_node-3]          verbosity:           0 (default)
[khronos_node-3]          child_layers[0] [ChildLayerConfig]:
[khronos_node-3]             layer:            OBJECTS
[khronos_node-3]             include_primary:  true (default)
[khronos_node-3]             include_partitions: true (default)
[khronos_node-3]    graph_updater [GraphUpdater::Config] (default):
[khronos_node-3]       layer_updates[OBJECTS] [LayerTracker::Config]:
[khronos_node-3]          prefix:              O
[khronos_node-3]          target_layer:        !<null> "" (default)
[khronos_node-3]    objects [MeshSegmenterConfig] (default):
[khronos_node-3]       layer_id:               OBJECTS (default)
[khronos_node-3]       bounding_box_type:      AABB (default)
[khronos_node-3]       timer_namespace:        frontend/objects (default)
[khronos_node-3]       clustering [ClusteringConfig] (default):
[khronos_node-3]          cluster_tolerance:   0.25 (default)
[khronos_node-3]          min_cluster_size:    40 (default)
[khronos_node-3]          max_cluster_size:    100000 (default)
[khronos_node-3]    pose_graph_tracker [Virtual Config: PoseGraphFromOdom::Config]:
[khronos_node-3]       min_pose_separation:    0.5 (default)
[khronos_node-3]       rotation_separation_weight: 0.1 (default)
[khronos_node-3]       min_time_separation_s:  0.2 (default)
[khronos_node-3]    surface_places [Uninitialized Virtual Config]
[khronos_node-3]    freespace_places [Virtual Config: GvdPlaceExtractor::Config]:
[khronos_node-3]       layer:                  PLACES (default)
[khronos_node-3]       min_component_size:     3 (default)
[khronos_node-3]       filter_places:          true (default)
[khronos_node-3]       filter_ground:          false (default)
[khronos_node-3]       robot_height:           0 (default)
[khronos_node-3]       edge_tolerance:         1 (default)
[khronos_node-3]       node_tolerance:         1 (default)
[khronos_node-3]       add_freespace_edges:    false (default)
[khronos_node-3]       gvd [GvdIntegratorConfig]:
[khronos_node-3]          max_distance_m:      4.5
[khronos_node-3]          min_distance_m:      0.1
[khronos_node-3]          min_diff_m:          0.1
[khronos_node-3]          min_weight:          1e-06 (default)
[khronos_node-3]          refine_voxel_pos:    false (default)
[khronos_node-3]          positive_distance_only: true (default)
[khronos_node-3]          voronoi_config [VoronoiCheckConfig]:
[khronos_node-3]             mode:             L1_THEN_ANGLE (default)
[khronos_node-3]             min_distance_m:   0.3
[khronos_node-3]             parent_l1_separation: 20
[khronos_node-3]             parent_cos_angle_separation: 0.2
[khronos_node-3]       graph [GraphExtractor::Config]:
[khronos_node-3]          compression_distance_m: 1.5
[khronos_node-3]          validate_graph:      false (default)
[khronos_node-3]          min_node_distance_m: 0.4
[khronos_node-3]          min_edge_distance_m: 0.25
[khronos_node-3]          merge_new_nodes:     true (default)
[khronos_node-3]          add_heuristic_edges: false (default)
[khronos_node-3]          merge_policy:        distance
[khronos_node-3]          node_merge_distance_m: 0.7
[khronos_node-3]          add_overlap_edges:   true (default)
[khronos_node-3]          add_freespace_edges: false
[khronos_node-3]          overlap_edges [OverlapEdgeConfig] (default):
[khronos_node-3]             num_neighbors_to_check: 4 (default)
[khronos_node-3]             min_clearance_m:  0.4 (default)
[khronos_node-3]       tsdf_interpolator [Virtual Config: DownsampleTsdfInterpolator::Config]:
[khronos_node-3]          ratio:               2 (default)
[khronos_node-3]          tolerance:           1e-10 (default)
[khronos_node-3]    frontier_places [Uninitialized Virtual Config]
[khronos_node-3]    view_database [ViewDatabase::Config] (default):
[khronos_node-3]       view_selection_method:  fusion (default)
[khronos_node-3]       inflation_distance:     0 (default)
[khronos_node-3]       layers:                 [PLACES, MESH_PLACES] (default)
[khronos_node-3]       verbosity:              0 (default)
[khronos_node-3] backend [Virtual Config: Backend]:
[khronos_node-3]    enable_node_merging:       true (default)
[khronos_node-3]    enable_exhaustive_merging: false (default)
[khronos_node-3]    add_places_to_deformation_graph: false
[khronos_node-3]    optimize_on_lc:            true (default)
[khronos_node-3]    verbosity:                 2
[khronos_node-3]    max_dt_merge_proposal:     3 (default)
[khronos_node-3]    optimize_on_new_merge:     true (default)
[khronos_node-3]    add_merge_factor:          true (default)
[khronos_node-3]    pose_object_covariance:    0.1
[khronos_node-3]    object_merge_covariance:   10
[khronos_node-3]    pose_object_consistency_threshold: 0.2
[khronos_node-3]    fix_input_pose_variance:   0.01 (default)
[khronos_node-3]    fix_input_poses:           true
[khronos_node-3]    run_every_n_frames:        300
[khronos_node-3]    pgmo [HydraPgmoConfig]:
[khronos_node-3]       run_mode:               FULL (default)
[khronos_node-3]       num_interp_pts:         3 (default)
[khronos_node-3]       interp_horizon:         10
[khronos_node-3]       add_initial_prior:      true (default)
[khronos_node-3]       output_prefix:          "" (default)
[khronos_node-3]       odom:                   0.01
[khronos_node-3]       loop_close:             0.05
[khronos_node-3]       prior:                  0.01
[khronos_node-3]       mesh_mesh:              0.01
[khronos_node-3]       pose_mesh:              0.01
[khronos_node-3]       place_mesh:             0.01
[khronos_node-3]       place_edge:             10
[khronos_node-3]       place_merge:            10
[khronos_node-3]       object_merge:           10
[khronos_node-3]       sg_loop_close:          0.1
[khronos_node-3]       optimizer [Virtual Config: KimeraRpgoOptimizer]:
[khronos_node-3]          solver:              LM (default)
[khronos_node-3]          verbosity:           0 (default)
[khronos_node-3]          print_summary:       true (default)
[khronos_node-3]          print_iterations:    true (default)
[khronos_node-3]          use_4dof_optim:      false (default)
[khronos_node-3]          use_gnc:             true (default)
[khronos_node-3]          barc_sq:             0
[khronos_node-3]          inlier_probability:  0.9
[khronos_node-3]          mu_step:             1.6
[khronos_node-3]          max_iterations:      100
[khronos_node-3]          fix_odom:            true
[khronos_node-3]          robust_cost:         TLS
[khronos_node-3]          use_pcm:             false (default)
[khronos_node-3]          trans_threshold:     0.1
[khronos_node-3]          rot_threshold:       0.01
[khronos_node-3]          mahalanobis_threshold: 1
[khronos_node-3]          odom_check:          false
[khronos_node-3]          odom_trans_threshold: 0.5
[khronos_node-3]          odom_rot_threshold:  0.1
[khronos_node-3]          odom_mahalanobis_threshold: 5
[khronos_node-3]          max_clique_mode:     HEURISTIC
[khronos_node-3]          metric_type:         NODE
[khronos_node-3]    external_loop_closures [ExternalLoopClosureReceiver::Config] (default):
[khronos_node-3]       layer:                  AGENTS (default)
[khronos_node-3]       max_time_difference [s]: 1 (default)
[khronos_node-3]    update_objects [UpdateKhronosObjectsFunctor::Config]:
[khronos_node-3]       merge_require_same_label: true (default)
[khronos_node-3]       merge_require_no_co_visibility: true
[khronos_node-3]       merge_min_iou:          1e-05
[khronos_node-3]       deformation_interpolator [DeformationInterpolator::Config] (default):
[khronos_node-3]          num_control_points:  4 (default)
[khronos_node-3]          control_point_tolerance_s [s]: 10 (default)
[khronos_node-3]       merge_proposer [MergeProposer::Config] (default):
[khronos_node-3]          strategy [Virtual Config: SemanticNearestNode::Config]:
[khronos_node-3]             num_merges_to_consider: 1 (default)
[khronos_node-3]    spatio_temporal_map [SpatioTemporalMap] (default):
[khronos_node-3]       verbosity:              2 (default)
[khronos_node-3]       finalize_incrementally: true (default)
[khronos_node-3]    reconciler [Reconciler]:
[khronos_node-3]       verbosity:              0
[khronos_node-3]       time_estimates_conservative: false (default)
[khronos_node-3]       allow_overestimation:   true
[khronos_node-3]       merge_object_meshes:    false (default)
[khronos_node-3]       mesh_merger [Uninitialized Virtual Config]
[khronos_node-3]    change_detection [SequentialChangeDetector] (default):
[khronos_node-3]       ray_verificator [RayVerificator]:
[khronos_node-3]          verbosity:           0
[khronos_node-3]          block_size [m]:      0.5
[khronos_node-3]          radial_tolerance [m]: 0.08
[khronos_node-3]          depth_tolerance [m]: 0.3
[khronos_node-3]          ray_policy:          Middle (default)
[khronos_node-3]          active_window_duration [s]: 3
[khronos_node-3]          prefix [RobotPrefixConfig] (default):
[khronos_node-3]             id:               0 (default)
[khronos_node-3]             key:              a (default)
[khronos_node-3]             vertex_key:       s (default)
[khronos_node-3]       ray_change_detector [RayChangeDetector]:
[khronos_node-3]          verbosity:           2 (default)
[khronos_node-3]          temporal_resolution [s]: 5
[khronos_node-3]          window_size:         5 (default)
[khronos_node-3]          use_relative_confidence: true (default)
[khronos_node-3]          absence_confidence:  0.6
[khronos_node-3]          presence_confidence: 0.5 (default)
[khronos_node-3]       objects [Virtual Config: RayObjectChangeDetector]:
[khronos_node-3]          time_filtering_threshold [s]: 3
[khronos_node-3]          query_subsampling:   100 (default)
[khronos_node-3]       background [Uninitialized Virtual Config]
[khronos_node-3] input [RosInputModule::Config] (default):
[khronos_node-3]    max_receiver_queue_size:   0 (default)
[khronos_node-3]    clear_queue_on_fail:       true (default)
[khronos_node-3]    inputs[left_cam] [InputModule::InputPair::Config] (default):
[khronos_node-3]       receiver [Virtual Config: OpenSetImageReceiver::Config]:
[khronos_node-3]          input_separation_s:  0.04
[khronos_node-3]          ns:                  "" (default)
[khronos_node-3]          queue_size:          10 (default)
[khronos_node-3]       sensor [Virtual Config: RosCamera::Config]:
[khronos_node-3]          min_range [m]:       0.1
[khronos_node-3]          max_range [m]:       5
[khronos_node-3]          ns:                  "" (default)
[khronos_node-3]          warning_timeout_s [s]: 10 (default)
[khronos_node-3]          error_timeout_s [s]: 0 (default)
[khronos_node-3]          latch_info_sub:      false (default)
[khronos_node-3]          extrinsics [Virtual Config: RosExtrinsics::Config]:
[khronos_node-3]             sensor_frame:     left_cam
[khronos_node-3]             robot_frame:      "" (default)
[khronos_node-3]             warning_timeout_s [s]: 10 (default)
[khronos_node-3]             error_timeout_s [s]: 0 (default)
[khronos_node-3]             wait_duration_s [s]: 0.1 (default)
[khronos_node-3]             verbosity:        3 (default)
[khronos_node-3]    tf_lookup [TFLookup::Config] (default):
[khronos_node-3]       wait_duration_s:        0.1 (default)
[khronos_node-3]       buffer_size_s:          30 (default)
[khronos_node-3]       max_tries:              5 (default)
[khronos_node-3]       verbosity:              3 (default)
[khronos_node-3] features [Uninitialized Virtual Config]
[khronos_node-3] status_monitor [StatusMonitor::Config] (default):
[khronos_node-3]    nickname:                  hydra (default)
[khronos_node-3]    max_time_between_spins_s [s]: 10 (default)
[khronos_node-3] ================================================================================
[khronos_node-3] I20260209 19:03:08.407584 1428945 experiment_manager.cpp:155] [ExperimentManager] Initializing pipeline...
[khronos_node-3] I20260209 19:03:08.420854 1428945 gvd_place_extractor.cpp:84] Downsampling TSDF when creating places!
[khronos_node-3] I20260209 19:03:08.433070 1428945 ros_sensors.cpp:76] Waiting for CameraInfo on /tesse/left_cam/camera_info to initialize sensor model
[dataset_player-1] [INFO] [1770663788.551819128] [bag_wrapper]: Dropping repeated tf 'base_link_gt_T_left_cam'
[dataset_player-1] [INFO] [1770663788.551889026] [bag_wrapper]: Dropping repeated tf 'base_link_gt_T_left_cam'
[dataset_player-1] stdin is not a terminal device. Keyboard handling disabled.[INFO] [1770663788.681983563] [rosbag2_player]: Set rate to 1
[dataset_player-1] [INFO] [1770663788.687833636] [rosbag2_player]: Adding keyboard callbacks.
[dataset_player-1] [INFO] [1770663788.687851372] [rosbag2_player]: Press SPACE for Pause/Resume
[dataset_player-1] [INFO] [1770663788.687854846] [rosbag2_player]: Press CURSOR_RIGHT for Play Next Message
[dataset_player-1] [INFO] [1770663788.687856807] [rosbag2_player]: Press CURSOR_UP for Increase Rate 10%
[dataset_player-1] [INFO] [1770663788.687858767] [rosbag2_player]: Press CURSOR_DOWN for Decrease Rate 10%
[dataset_player-1] [INFO] [1770663788.687863956] [rosbag2_player]: Playback until timestamp: -1
[khronos_node-3] I20260209 19:03:08.835733 1428945 ros_sensors.cpp:105] Looking for sensor extrinsics 'base_link_gt_T_left_cam' via TF
[khronos_node-3] I20260209 19:03:08.939831 1428945 ros_sensors.cpp:234] Initialized camera as 
[khronos_node-3] ============================ Virtual Config: Camera ============================
[khronos_node-3] min_range [m]:                0.1
[khronos_node-3] max_range [m]:                5
[khronos_node-3] width [px]:                   720
[khronos_node-3] height [px]:                  480
[khronos_node-3] cx [px]:                      360
[khronos_node-3] cy [px]:                      240
[khronos_node-3] fx [px]:                      415.692
[khronos_node-3] fy [px]:                      415.692
[khronos_node-3] extrinsics [Virtual Config: ParamSensorExtrinsics]:
[khronos_node-3]    body_R_sensor:             {w: -0.5, x: 0.5, y: -0.5, z: 0.5}
[khronos_node-3]    body_p_sensor:             [0, 0.05, 0]
[khronos_node-3] ================================================================================
[khronos_node-3] E20260209 19:03:08.943023 1428945 khronos_pipeline.cpp:106] Active window is not khronos::ActiveWindow!
[khronos_node-3] E20260209 19:03:08.943151 1428945 khronos_pipeline.cpp:117] Backend is not khronos::Backend!
[khronos_node-3] I20260209 19:03:08.943162 1428945 experiment_manager.cpp:157] [ExperimentManager] Starting pipeline...
[khronos_node-3] I20260209 19:03:08.955646 1428945 input_module.cpp:81] [Hydra Input] started!
[khronos_node-3] I20260209 19:03:08.955698 1428945 active_window_module.cpp:87] [Active Window] started!
[khronos_node-3] I20260209 19:03:08.955736 1428945 backend_module.cpp:136] [Hydra Backend] started!
[khronos_node-3] I20260209 19:03:08.955780 1428945 graph_builder.cpp:167] [Hydra Frontend] started!
[khronos_node-3] [INFO] [1770663788.956000214] [khronos_node]: Running...
[open_set_node-2] [INFO] [1770663789.349425695] [semantic_inference]: Initializing with OpenSetNodeConfig:
[open_set_node-2] {'model': {'clip_model': {'model_name': 'ViT-L/14', 'type': 'clip'},
[open_set_node-2]            'dense_ratio': 0.9,
[open_set_node-2]            'patches': {'crop_padding': 4,
[open_set_node-2]                        'interpolation_mode': 'bicubic',
[open_set_node-2]                        'mean': None,
[open_set_node-2]                        'min_mask_size': 0,
[open_set_node-2]                        'min_segment_area': 0,
[open_set_node-2]                        'normalize': True,
[open_set_node-2]                        'should_scale': True,
[open_set_node-2]                        'std': None},
[open_set_node-2]            'refinement': {'dilate_masks': False,
[open_set_node-2]                           'dilation_passes': 1,
[open_set_node-2]                           'kernel_size': 3,
[open_set_node-2]                           'kernel_tolerance': 0.001},
[open_set_node-2]            'segmentation': {'confidence': 0.55,
[open_set_node-2]                             'iou': 0.85,
[open_set_node-2]                             'model_name': 'FastSAM-x.pt',
[open_set_node-2]                             'output_size': 736,
[open_set_node-2]                             'type': 'fastsam'},
[open_set_node-2]            'use_dense': False,
[open_set_node-2]            'use_dense_area_interpolation': False},
[open_set_node-2]  'visualizer': {'components': [-1, -2, -3],
[open_set_node-2]                 'offset': 0.5,
[open_set_node-2]                 'scale': 0.5,
[open_set_node-2]                 'type': 'component'},
[open_set_node-2]  'worker': {'encoding': 'rgb8', 'min_separation_s': 0.0, 'queue_size': 1}}
[open_set_node-2] [INFO] [1770663797.694104933] [semantic_inference]: Finished initializing!

[IakovPodmarev]

I have just faced the same problem. Did you solve it? @Lixiangqi2002

[mr95david]

I believe I was able to partially solve the issue, as I was running into the same error.

To debug it, I reviewed two launch files involved in the open-set segmentation execution pipeline. In /khronos/khronos_ros/launch/datasets/play_uhumans.launch.yaml, the topic remapping is currently defined as follows:

- group:
    - set_remap: {from: semantic_inference/color/image_raw, to: /tesse/left_cam/rgb/image_raw}

    - include:
        if: $(and $(not $(var use_openset_semantics)) $(not $(var use_gt_semantics)))
        file: $(find-pkg-share semantic_inference_ros)/launch/closed_set.launch.yaml
        arg:
          - {name: labelspace_name, value: ade20k_indoor}

    - include:
        if: $(and $(var use_openset_semantics) $(not $(var use_gt_semantics)))
        file: $(find-pkg-share semantic_inference_ros)/launch/open_set.launch.yaml
        arg:
          - {name: config_path, value: $(var segmenter_config)}

However, after inspecting the open_set_node implementation from the semantic_inference_ros package, I noticed that the node subscribes to the relative topic:

"color/image_raw"

When checking the running node with:

ros2 node info /semantic_inference

the subscriber was listed as:

/color/image_raw: sensor_msgs/msg/Image

Therefore, the original remapping did not seem to match the actual topic used by the node. I modified the remapping to:

- set_remap: {from: /color/image_raw, to: /tesse/left_cam/rgb/image_raw}

I also made a related change in khronos/khronos_ros/launch/uhumans2_khronos.launch.yaml. In line 43, the label topic was originally set as:

- let: {name: label_topic, value: /semantic_inference/semantic/image_raw, unless: $(var use_gt_semantics)}

However, according to the publishers exposed by open_set_node, the semantic output is published as:

/semantic/image_raw

not as:

/semantic_inference/semantic/image_raw

So I changed it to:

- let: {name: label_topic, value: /semantic/image_raw, unless: $(var use_gt_semantics)}

With these changes, the semantic processing pipeline is able to continue running, and the open-set semantic description appears to be generated.

However, the remaining issue is that neither the tracking nor the dynamic semantic image visualization in RViz is being generated correctly. @IakovPodmarev @Lixiangqi2002