feature: allow centralized exploration policy

[LovelyBuggies]

No description provided.

[LovelyBuggies]

Background

In CoMLRL-v1.4.0, the comparators (e.g., target policy, history checkpoints, external api) are provided in a decentralized manner: 2 comparator agents generate the responses to the same prompt as the 2 target agents. In this PR, it can be centralized: 1 agent can coordinate the subtasks of two agents and provide the responses. Specifically, if comparator_generation_mode (decentralized by default) is set to centralized, then one can select an agent comparator_centralized_agent_index as the centralized agent.

Results

In general, as with "knowledge transfer" in comparator selection, MAPL learns well with a centralized policy from a relatively intelligent model and better from the same class.

For online MADPO, when provided with a sufficiently intelligent centralized comparator (e.g., $\color{blue}{\text{14B-Instruct}}$, $\color{purple}{\text{32B-Instruct}}$, and $\color{brown}{\text{DeepSeek}}$), it can learn. But it's more likely to succeed with these same-family centralized comparators than DeepSeek. Curves drop with the 3B-level (current and checkpoints) and remain flat at the 7B-level comparators (7B-base and $\color{green}{\text{7B-instruct}}$). It's a bit strange that I'm using a 3B base model, but it can actually learn from the *-instruct models but not the base models. I guess it might be because Instruct can better "understand" how to split the task in a centralized way.

$\color{purple}{\text{32B-Instruct}}$'s Guidance

$\color{black}{\text{current (3B)}}$: Too weak to provide centralized policy.

---

Similarly, online MARLHF also needs an intelligent centralized agent (e.g., $\color{#00BBFF}{\text{14B}}$, $\color{brown}{\text{DeepSeek}}$). It's WEIRD that $\color{purple}{\text{32B}}$ (base and instruct) are even larger, but they have very little chance of working and are unstable. It's also WEIRD that this time $\color{#00BBFF}{\text{14B-base}}$ works while $\color{blue}{\text{14B-Instruct}}$ cannot 🤨. Though from a different family, $\color{brown}{\text{DeepSeek}}$ actually works well this time.

$\color{brown}{\text{DeepSeek}}$'s Guidance

$\color{purple}{\text{32B-Instruct}}$: "Not my fault!"

---

[LovelyBuggies]

Possible Extensions?

"Centralized comparator" has many variants, not necessarily to be a centralized model. Can also be other forms, like sequential pipeline, discussion, etc.

[LovelyBuggies]

Preliminary Tests on 7B Model Collab

I ran some tests on the collaboration of the 7B model using the exact same parameters and found that MADPO doesn't work under any of these conditions. However, MARLHF does work when lambda=1 (i.e., all history).

Due to the significantly different nature between 3B and 7B, we'll probably need to do more parameter tuning. That said, the advantage of using Lambda decay is that you can always find a specific Lambda value to represent various relationships.