Hi, thanks for releasing the paper and the code!
I have a question regarding the OGBench offline-to-online results reported in Table 2.
As far as I understand, in the original FQL paper, the OGBench offline-to-online experiments are conducted on the default task of each environment, i.e., the environments with the -singletask-v0 suffix. These correspond to one specific task among the five task variants in each environment, rather than an average over all five tasks.
For example, the FQL results reported in Table 2 seem to match the default-task results from the original FQL paper, such as:
humanoidmaze-medium-navigate-singletask-v0: 12 ± 7 → 22 ± 12antsoccer-arena-navigate-singletask-v0: 28 ± 8 → 86 ± 5cube-double-play-singletask-v0: 40 ± 11 → 92 ± 3
However, I noticed that the offline scores of "Ours" in Table 2 appear to be identical to the category-level results reported in Table 1, where each OGBench category contains 5 tasks. For example:
- humanoidmaze-medium: 62 ± 1
- antsoccer-arena: 62 ± 3
- cube-double: 3 ± 2
- scene: 60 ± 1
- puzzle-4x4: 40 ± 6
This made me wonder whether the "Ours" results in Table 2 are averaged over all five tasks in each environment, while the FQL (and possibly other baseline) results correspond only to the default task.
Could you please clarify whether the results in Table 2 are all evaluated using the same task setting? In particular, are the "Ours" results evaluated on the default task only, or averaged over all five tasks?
I may have misunderstood the evaluation protocol, so I would really appreciate any clarification. Thanks!
Hi, thanks for releasing the paper and the code!
I have a question regarding the OGBench offline-to-online results reported in Table 2.
As far as I understand, in the original FQL paper, the OGBench offline-to-online experiments are conducted on the default task of each environment, i.e., the environments with the
-singletask-v0suffix. These correspond to one specific task among the five task variants in each environment, rather than an average over all five tasks.For example, the FQL results reported in Table 2 seem to match the default-task results from the original FQL paper, such as:
humanoidmaze-medium-navigate-singletask-v0: 12 ± 7 → 22 ± 12antsoccer-arena-navigate-singletask-v0: 28 ± 8 → 86 ± 5cube-double-play-singletask-v0: 40 ± 11 → 92 ± 3However, I noticed that the offline scores of "Ours" in Table 2 appear to be identical to the category-level results reported in Table 1, where each OGBench category contains 5 tasks. For example:
This made me wonder whether the "Ours" results in Table 2 are averaged over all five tasks in each environment, while the FQL (and possibly other baseline) results correspond only to the default task.
Could you please clarify whether the results in Table 2 are all evaluated using the same task setting? In particular, are the "Ours" results evaluated on the default task only, or averaged over all five tasks?
I may have misunderstood the evaluation protocol, so I would really appreciate any clarification. Thanks!