GRPOTrainer silently no-ops (empty optimizer) when continuing from a PeftModel loaded with is_trainable=False and beta != 0

[Meredith2328]

GRPOTrainer silently no-ops (empty optimizer) when continuing from a PeftModel loaded with is_trainable=False and beta != 0

Problem summary

GRPOTrainer silently trains nothing when it is given a PeftModel that was loaded with PeftModel.from_pretrained(..., is_trainable=False) and beta != 0:

• The auto-created ref adapter (used for the KL term) inherits inference_mode=True from the default adapter's config.
• PEFT's inject_adapter then also freezes the active ("default") adapter, so after trainer init there are zero trainable parameters and the optimizer is created empty.
• During training, use_adapter() temporarily flips requires_grad, so gradients accumulate, grad_norm is non-zero, and loss/reward vary from step to step (each step samples different prompts). The run completes without any error or warning, and the saved adapter is byte-identical (md5) to the initial adapter.

We hit this in production (100-step GRPO run, complete wandb curves, everything looked healthy) and only noticed after hashing the saved weights.

Why is_trainable=True in #3031 is NOT a sufficient fix

Loading the adapter with is_trainable=True does unblock training (verified: same script, optimizer goes from 0 to 12 params and the saved adapter changes). But it does not resolve the underlying defect:

1. The trap stays silent for everyone else. The default usage (PeftModel.from_pretrained(...) with no is_trainable) still silently no-ops with healthy-looking metrics; nothing warns or errors, so any future user hits it again.
2. It cannot change the LoRA configuration. Continuing with a different r, target_modules, or a new peft_config requires merging first: TRL raises ValueError when a PeftModel and a peft_config are passed together ("Please first merge and unload the existing adapter...").
3. The ref-adapter mechanism remains fragile. Whether ref stays frozen depends on current PEFT behavior, not on an explicit TRL guarantee.
4. The default behavior is a footgun. Until the framework either raises or auto-handles trainability, users following the "continue from a fine-tuned adapter" flow keep getting burned silently.

A fix (fail fast with a clear error + explicitly freeze ref) is prepared in the follow-up PR.

Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, PeftModel
from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer

model_id = "sshleifer/tiny-gpt2"

tok = AutoTokenizer.from_pretrained(model_id)
tok.pad_token = tok.eos_token
tok.chat_template = (
    "{% for message in messages %}{{ message['role'] }}: "
    "{{ message['content'] }}{{ '\\n' if not loop.last else '' }}{% endfor %}"
)

lora = LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn", "c_proj"], task_type="CAUSAL_LM")

# 0) create a dummy "already fine-tuned" adapter to continue from
sft = get_peft_model(AutoModelForCausalLM.from_pretrained(model_id), lora)
sft.save_pretrained("./sft_adapter")

# 1) load it WITHOUT is_trainable=True (this is the default)
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(model_id), "./sft_adapter")

train = Dataset.from_list([
    {"prompt": "What is 2+2?", "answer": "4"},
    {"prompt": "What is 3+4?", "answer": "7"},
])


def reward_fn(completions, **kwargs):
    return [(len(set(c)) % 3) / 2.0 for c in completions]  # non-uniform rewards


trainer = GRPOTrainer(
    model=model,
    processing_class=tok,
    reward_funcs=[reward_fn],
    train_dataset=train,
    args=GRPOConfig(
        output_dir="./out",
        beta=0.01,  # beta != 0 triggers the buggy path
        per_device_train_batch_size=1,
        generation_batch_size=4,
        num_generations=4,
        max_completion_length=64,
        max_steps=4,
        logging_steps=1,
        save_steps=4,
        report_to="none",
        seed=42,
    ),
)

print("trainable params after trainer init:",
      sum(1 for p in trainer.model.parameters() if p.requires_grad))

trainer.train()

print("optimizer param count (captured at creation):",
      sum(len(g["params"]) for g in trainer.optimizer.param_groups))

outputs:

trainable params after trainer init: 0
optimizer param count (captured at creation): 0
{'loss': '-0.26', 'grad_norm': '0.001', 'reward': '0.625', ...}   # metrics look alive

And the saved adapter (out/checkpoint-4/adapter_model.safetensors) is byte-identical (md5) to sft_adapter/adapter_model.safetensors.

Control: loading with PeftModel.from_pretrained(..., is_trainable=True) gives optimizer param count: 12 and a saved adapter that differs from the initial one — the exact same script trains for real.

Full runnable reproduction (CPU only, ~2 min): https://www.kaggle.com/code/meredith10pi/trl-1-9-2-ref-adapter-silent-no-op-repro

System Info

trl==1.9.2
peft==0.20.0
transformers==5.14.1
torch==2.9.0+cu128 (also reproduced with torch==2.10.0+cpu)
accelerate==1.14.0
datasets==5.0.1
Python 3.10/3.12, Linux (reproduced on a single CPU process and on 2x RTX 5090 with accelerate multi-GPU)

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete