Where is language inputted to the model?

[rsathua]

Hi,

I would like to revisit this issue as I not sure where language instruction is actually fed into the model due to it not being one of the obs_shapes.

I see that you specifiy it as part of your batch data here. However, what I am unable to locate is where the language instruction from this batch actually gets passed to the model. From my understanding it seems the observation keys defined in file_utils.py (which does not seem to include language) are the only inputs to the observation encoders / feature concatenation in obs_nets.py. Additionally, I hadn't seen language instruction as one of the observation modalities in droid_runs_language_conditioned_rlds.py so I was hoping to get insight on where and how the language instruction is fed into the model.

Is this understanding of the code structure correct? If so where does the language instruction actually get fed into the model?

[badinkajink]

Language conditioning appears to be implemented for diffusion_policy.py, see in their batch processing function:

droid_policy_learning/robomimic/algo/diffusion_policy.py

Line 138 in 205ff6b

def process_batch_for_training(self, batch):

[szl2001]

Hi! have you figured out this problem? I also can not find where the language is fed. Actually during the ObservationGroupEncoder, the eef position, gripper position and images are processed. I notice the language is pre-processed, but for the combo net, the input shape matches the output shape of the processor, leaving no space for the language. After the combo net, output is used as obs_cond and passed to the diffusion model. Throughout the process, the language seems to remain in the obs_dict, but it is not included in the cond. If you have already figured out this problem, could you help answer it? Thanks.