Bug +fix: padding_side not set for tokenizers with existing pad_token, causing empty responses

[accemlcc]

Bug: padding_side not set for tokenizers with existing pad_token, causing empty responses

Problem

When a tokenizer already has a pad_token defined (e.g., allenai/Olmo-3-7B-Instruct), padding_side is not set to "left", causing empty responses during batched generation.

Decoder-only models require left-padding for generation. With right-padding (the default), models see [PROMPT] [PAD] [PAD] and stop generating immediately, returning empty strings.

Impact

• Empty responses are counted as "non-refusals" in is_refusal() (no markers found → success)
• Results in falsely low refusal rates (e.g., 5/100 instead of 90/100)
• Only affects batch_size > 1 (no padding needed for single prompts)
• batch_size=1 shows correct results, larger batches show near-zero refusals

Reproduction

Test with any model that has its own pad_token (doesn't use EOS fallback):

# batch_size=1: Correct refusal count (e.g., 10/10)
# batch_size=128: Near-zero refusals, all empty responses counted as "success"

Tested with allenai/Olmo-3-7B-Instruct and squ11z1/Hypnos-i1-8B.

Current Code (model.py:50-52)

if self.tokenizer.pad_token is None:
    self.tokenizer.pad_token = self.tokenizer.eos_token
    self.tokenizer.padding_side = "left"  # ← Only set when pad_token is None!

Suggested Fix

if self.tokenizer.pad_token is None:
    self.tokenizer.pad_token = self.tokenizer.eos_token

# CRITICAL: Always use left padding for decoder-only models during generation.
# Right padding causes empty outputs because the model sees PAD tokens after
# the prompt and thinks the sequence is complete.
self.tokenizer.padding_side = "left"

Optional: Handle empty responses in evaluation

Additionally, is_refusal() in evaluator.py could treat empty/very short responses as failures rather than successes:

def is_refusal(self, response: str) -> bool:
    # Empty response = generation failure, not successful uncensoring
    if len(response.strip()) < 5:
        return True
    # ... rest of marker checking

This provides defense-in-depth against other potential generation failures.

[red40maxxer]

Yep, I noticed this last night as well. #53 will allow much more flexible refusal detection logic but I will add that check to the default implementation too

[accemlcc]

@red40maxxer Thanks! Looking forward to the plugin system (#53), that will definitely make the detection logic much more robust (e.g. penalizing empty responses).

However, regarding the tokenizer fix itself:
Since the incorrect padding side actually causes the empty generation in the first place (on decoder-only models), we should probably apply this fix globally and immediately, independent of how we evaluate the result later.

If you are touching model.py in your PR anyway, feel free to include the padding_side="left" line there! Otherwise, a quick hotfix commit to master might be safer to unblock users right now.

[p-e-w]

When a tokenizer already has a pad_token defined (e.g., allenai/Olmo-3-7B-Instruct), padding_side is not set to "left", causing empty responses during batched generation.

No, something else is going on here. Almost all tokenizers define a pad_token (random example). In fact, I only introduced this code in 0bae27f after noticing that Falcon-E-3B doesn't provide a pad token, which is very uncommon. If you just remove that if clause entirely, things will still work fine with almost all models.

With right-padding (the default), models see [PROMPT] [PAD] [PAD] and stop generating immediately, returning empty strings.

Right-padding is not the default. The default depends on the padding_side class attribute (see Transformers docs). I just spent 10 minutes in the bowels of Transformers, but I couldn't find where exactly GPT2Tokenizer (the class used by allenai/Olmo-3-7B-Instruct) sets its padding_side.

Your suggested fix is probably good, but we should make sure that we fully understand what is going on here. Empty responses should probably raise an exception, because they almost always indicate that something fundamental is going wrong. Just flagging the response as a refusal is not sufficient.

[accemlcc]

hey @p-e-w :-)

Here are the references from the HuggingFace documentation and source code to back this up:

1. Regarding the Default Value:
The padding_side defaults to 'right' in the base class PreTrainedTokenizerBase. Since GPT2Tokenizer (used by OLMo) does not explicitly override this in its __init__, it inherits 'right'.
Source Code Link (See the __init__ arguments)

2. Regarding Generation Failure:
HuggingFace explicitly warns against right-padding for decoder-only generation in their LLM tutorial:

"LLMs are decoder-only architectures, meaning they usually require padding on the left side. [...] If the inputs are padded on the right, the model will continue generating from the padding tokens, which leads to incorrect generation or gibberish."
Documentation Link

Since Heretic uses batched generation (batch_size > 1), right-padding puts [PAD] tokens after the prompt. The model attends to these pads as the "most recent context" and usually outputs EOS or empty strings immediately.

Forcing self.tokenizer.padding_side = 'left' ensures safety across all decoder models, regardless of whether they come with a pad token or not.

3. Why it sometimes appears to work (The confusion):
You might have observed cases where generation works despite padding_side='right'. This happens when:

• Batch Size = 1: No padding is ever applied.
• Uniform Lengths: All prompts in a specific batch happen to be the same length.

In these cases, no [PAD] tokens are appended, so the model doesn't see the "end of sequence" signal prematurely. The failure strictly triggers only when actual padding tokens are inserted to align lengths in a batch.

(4. Deterministic Failure:
It is important to remember that LLMs are deterministic functions regarding their stop conditions. If a model consistently outputs an empty string, it is not "refusing" in a semantic sense (deciding not to answer). It is encountering a structural stop-signal (like right-padding tokens interpreted as end-of-sequence) immediately.
Therefore, an empty response is always an indicator of a broken inference configuration, not a model behavior trait.)

[p-e-w]

Thanks for the explanation. I know how padding works and why inputs should be right-aligned in the batch frame, but I didn't see why right-padding would be used by GPT2Tokenizer until you pointed it out. The Transformers docs are quite misleading there, stating (emphasis added):

padding_side (str, optional) — The side on which the model should have padding applied. Should be selected between [‘right’, ‘left’]. Default value is picked from the class attribute of the same name.

What they mean is that the default is picked from the class attribute of the base class (unless overridden), so in practice, the default is "right" as you described.

I have implemented your suggested fixes, naming you as co-author. I changed my mind about raising an exception for empty responses, because such responses might legitimately come from a model damaged by abliteration, and this could throw off the optimization process if one of the parameter configurations happens to generate such a model.