Longer optimization runs can produce worse Pareto fronts than shorter runs

[pszemraj]

Sorry for the lack of updates wrt to support for hybrid model arch. In the process of getting to that, I realized the below was not just a one-off and should probably be dealt with first, as it impacts even cursory sanity/'what if' checks (esp. relevant for new architectures to see if something is wrong/need more trials).

---

Observed behavior

When running abliteration on rnj-1-instruct¹, an 800-trial optimization produced a strictly worse Pareto front than a 200-trial run on the same model. This had happened before and I wrote it off, but realized it was far too consistent, persists across different architectures, etc. So here we are.

Detailed trial result lists - click to expand

rnj1_abliteration_results_options.md

both done with the v1.1.0 release

200 trials

default params.

? Which trial do you want to use? (Use arrow keys)
 » [Trial  77] Refusals: 25/100, KL divergence: 0.3209
   [Trial 188] Refusals: 34/100, KL divergence: 0.2809
   [Trial  62] Refusals: 39/100, KL divergence: 0.2790
   [Trial 141] Refusals: 40/100, KL divergence: 0.2788
   [Trial  80] Refusals: 42/100, KL divergence: 0.2708
   [Trial 135] Refusals: 43/100, KL divergence: 0.2195
   [Trial 108] Refusals: 48/100, KL divergence: 0.0346
   [Trial 116] Refusals: 50/100, KL divergence: 0.0305
   [Trial 130] Refusals: 57/100, KL divergence: 0.0299
   [Trial 117] Refusals: 58/100, KL divergence: 0.0255
   [Trial  86] Refusals: 60/100, KL divergence: 0.0239
   [Trial 157] Refusals: 69/100, KL divergence: 0.0208
   [Trial 110] Refusals: 70/100, KL divergence: 0.0166
   [Trial  63] Refusals: 76/100, KL divergence: 0.0161
   [Trial 167] Refusals: 84/100, KL divergence: 0.0092
   [Trial  96] Refusals: 95/100, KL divergence: 0.0066
   [Trial  84] Refusals: 96/100, KL divergence: 0.0019
   None (exit program)

ones I kept:

? Which trial do you want to use? [Trial  77] Refusals: 25/100, KL divergence: 0.3209

Restoring model from trial 77...
* Reloading model...
Loading checkpoint shards: 100%|███████████████████████████████████| 4/4 [00:01<00:00,  2.33it/s]
* Abliterating...

? What do you want to do with the decensored model? Save the model to a local folder
? Path to the folder: /home/pszemraj/model-weights/llm/heretic/rnj-1-instruct-heretic-n200_25ref
Saving model...
Model saved to /home/pszemraj/model-weights/llm/heretic/rnj-1-instruct-heretic-n200_25ref.

? What do you want to do with the decensored model? Nothing (return to trial selection menu)

? Which trial do you want to use? [Trial 141] Refusals: 40/100, KL divergence: 0.2788

Restoring model from trial 141...
* Reloading model...
Loading checkpoint shards: 100%|███████████████████████████████████| 4/4 [00:01<00:00,  2.34it/s]
* Abliterating...

? What do you want to do with the decensored model? Save the model to a local folder
? Path to the folder: /home/pszemraj/model-weights/llm/heretic/rnj-1-instruct-heretic-n200_40ref
Saving model...
Model saved to /home/pszemraj/model-weights/llm/heretic/rnj-1-instruct-heretic-n200_40ref.

800 trials (200ish startup trials)

? Which trial do you want to use? (Use arrow keys)
 » [Trial 714] Refusals: 20/100, KL divergence: 0.3971
   [Trial 512] Refusals: 27/100, KL divergence: 0.3502
   [Trial 291] Refusals: 28/100, KL divergence: 0.3267
   [Trial 290] Refusals: 29/100, KL divergence: 0.3257
   [Trial 508] Refusals: 30/100, KL divergence: 0.3178
   [Trial 218] Refusals: 38/100, KL divergence: 0.3003
   [Trial 517] Refusals: 40/100, KL divergence: 0.2660
   [Trial 302] Refusals: 41/100, KL divergence: 0.0471
   [Trial 312] Refusals: 45/100, KL divergence: 0.0352
   [Trial 557] Refusals: 53/100, KL divergence: 0.0257
   [Trial 161] Refusals: 59/100, KL divergence: 0.0250
   [Trial 413] Refusals: 63/100, KL divergence: 0.0222
   [Trial 119] Refusals: 67/100, KL divergence: 0.0196
   [Trial 296] Refusals: 70/100, KL divergence: 0.0187
   [Trial 274] Refusals: 74/100, KL divergence: 0.0156
   [Trial 299] Refusals: 78/100, KL divergence: 0.0123
   [Trial 311] Refusals: 85/100, KL divergence: 0.0090
   [Trial 259] Refusals: 89/100, KL divergence: 0.0077
   [Trial 543] Refusals: 93/100, KL divergence: 0.0034
   [Trial 246] Refusals: 94/100, KL divergence: 0.0029
   [Trial 458] Refusals: 95/100, KL divergence: 0.0024
   [Trial 757] Refusals: 96/100, KL divergence: 0.0015
   None (exit program)

Comparison of low-refusal regime results:

Run	Best low-refusal result
200 trials	25 refusals @ 0.3209 KL
800 trials	27 refusals @ 0.3502 KL

The 200-trial result dominates the 800-trial result for both objectives. This is counterintuitive; more trials should not degrade the Pareto front. It was even more drastic with medgemma-27b-it (albeit a larger model). As you can imagine, this makes it hard to validate support/configuration for new architectures: For example, I've attempted to run longer studies when checking why <some new arch> gets a min of 30/100 refusals, and then when the run with n=1000 trials is worse than n=200, I don't have much to go on.

Both runs used v1.1.0. The n_startup_trials parameter was manually scaled proportionally (200-250 for the 800-trial run).

potential causes

Note

item 1 I checked and agree (should be easy). items 2-3 are suggested by Claude after my probing at what else could be happening beyond simple random seeds, take them with a grain of salt.

1. No seed parameter in TPESampler - Each run explores a completely different random trajectory. This alone could explain divergent outcomes, though the magnitude of the difference seems large for pure seed variance.

   • When seed=None (default): TPESampler creates a numpy.random.RandomState which seeds from /dev/urandom or system clock. Each run gets a different random state.

2. Multivariate TPE in high-dimensional space - With ~10 parameters and multivariate=True, covariance estimation from the startup samples is noisy. Different initializations can lock the optimizer into different basins that it won't escape from.

3. Disconnected Pareto-optimal regions - The data suggests two distinct regimes (note the KL cliff between ~40-48 refusals dropping from ~0.27 to ~0.03). MOTPE may struggle to traverse dominated space between disconnected basins.

Debugging limitations

On a related note, there's currently no mechanism to export/persist trial data, making it difficult to:

• Compare parameter distributions across runs
• Identify which regions of the search space each run explored
• Warm-start from prior results

As part of debugging this, can we add support for saving at least the Pareto frontier trials, if not the whole study? I can implement manually, of course, but I think this would be useful for future issues as well.

Footnotes

1. already supported, as it's Gemma3ForCausalLM arch ↩

[accemlcc]

Tested on PR #60 code (LoRA/4-bit) with default settings. Got 11/100 Refusals @ 0.06 KLD. Seems like the LoRA/Quantization approach stabilizes the optimization landscape significantly compared to standard fp16 runs. (?)
-> https://huggingface.co/arnomatic/rnj-1-instruct-heretic

However, based on my experiments, I can confirm that 800 trials are not necessarily better than 200.

[pszemraj]

@accemlcc Interesting result! One thing worth noting: the KL numbers might not be directly apples to-apples here.

The baseline for KL calculation gets captured from the model as-loaded (if I am reading the code right). So your 0.06 KL is measuring abliterated_q4 vs base_q4, while my 0.32 is measuring abliterated_bf16 vs base_bf16¹. The quantized baseline already has information loss baked in, so the KL only captures the additional deviation from the LoRA mods rather than total deviation from the true original (again, assuming I read it right).

If you wanted to do a direct comparison, you could try:

heretic --model EssentialAI/rnj-1-instruct --evaluate-model arnomatic/rnj-1-instruct-heretic

with the original loaded in bf16 (no quantization).

also:

• if the above comparisons/my interpretation is correct: you might want to account for/normalize somehow when quantization/LoRA is used to make things clearer for people, both in terms of the console/model cards
• you might still be onto something with LoRA stabilizing the optimization - the rank-1 constraint probably does limit the damage any given trial can do. Would be curious to see the numbers if you get a chance to run that eval!
• also, being able to export/visualize trials, search space would be super useful for scenarios exactly like this @p-e-w :)

Footnotes

1. in the year of our Lord 2025 (almost 2026) I really hope nobody is using fp16.. ↩

[accemlcc]

Interesting result! One thing worth noting: the KL numbers might not be directly apples to-apples here.

The baseline for KL calculation gets captured from the model as-loaded (if I am reading the code right). So your 0.06 KL is measuring abliterated_q4 vs base_q4, while my 0.32 is measuring abliterated_bf16 vs base_bf16

I'd argue it's actually a fair comparison within each format:

• q4 vs q4: Both sides share the same quantization bias → it cancels out.
• bf16 vs bf16: Both sides have the same higher precision → also a fair comparison within format.

So the KL isn't measuring "how much the quantization hurt" – it's measuring "how much did the abliteration shift the output distribution", which is consistent regardless of precision.

Ran the same optimization just yesterday on Qwen3-4B in both bf16 and bnb4bit (again, as I did before the PR) – final refusal counts and KL were only marginally different. This suggests the quantization format doesn't significantly bias the optimization or the KL metric.
-> https://huggingface.co/arnomatic/Qwen3-4B-Instruct-2507-heretic

[p-e-w]

When running abliteration on rnj-1-instruct, an 800-trial optimization produced a strictly worse Pareto front than a 200-trial run on the same model. This had happened before and I wrote it off, but realized it was far too consistent, persists across different architectures, etc.

Simply put, that's impossible in general. Unlike certain optimization techniques such as simulated annealing, TPE is oblivious to how many trials there are in total. In other words, running a 200-trial study is exactly the same thing as running an 800-trial study, and cancelling it after 200 trials. And since adding results from more trials (while keeping the existing results) can never make the Pareto front worse (by definition of Pareto optimality), what you're seeing is almost certainly an artifact of the inherent variability of the randomized TPE process.

When seed=None (default): TPESampler creates a numpy.random.RandomState which seeds from /dev/urandom or system clock. Each run gets a different random state.

Yes, and that's a feature, because it allows one to just repeat a bad run and potentially get better results. I thought about making the seed a configuration parameter, but decided against it because it would be misleading, since PyTorch operations aren't deterministic even with completely identical inputs. Changes in platform, GPU model, driver version, PyTorch version, and possibly other factors can all change the outcome, and PyTorch offers no way around that, so having the same seed wouldn't make results reproducible except on the exact same setup.

Note that TPE is based on Gaussian models of the optimization domain, and is thus inherently probabilistic. Therefore, fixing the seed will not yield a deterministic algorithm in any meaningful sense, but rather a specific slice of a randomized algorithm.

That being said, keeping a persistent log of all trials has been suggested elsewhere as well, and is a good idea. The cleanest way is probably to just write each completed trial to a JSON lines-formatted file in a /logs directory. I'll implement that in the next few days if I have time.

[p-e-w]

A general observation about consistency when running multiple studies, which might not immediately be obvious:

The optimization domain is absolutely gigantic. We have a 10-dimensional search space. One parameter is effectively a boolean (per-layer: yes/no), and if we were to allow just 3 different values for each of the other parameters, we would still be looking at 2 * 3^9 = 39366 combinations.

In other words, no practical run can ever hope to visit even a tiny fraction of the domain. Thus wildly different outcomes are completely expected, to the extent that essentially nothing can be said about optimization behavior with any certainty, unless hundreds or even thousands of studies are conducted.

I've come across this fundamental problem several times during development, and I don't think that there's much we can do about it because trials are so expensive in our case.

[pszemraj]

Interesting result! One thing worth noting: the KL numbers might not be directly apples to-apples here.
The baseline for KL calculation gets captured from the model as-loaded (if I am reading the code right). So your 0.06 KL is measuring abliterated_q4 vs base_q4, while my 0.32 is measuring abliterated_bf16 vs base_bf16

I'd argue it's actually a fair comparison within each format:

• q4 vs q4: Both sides share the same quantization bias → it cancels out.
• bf16 vs bf16: Both sides have the same higher precision → also a fair comparison within format.

So the KL isn't measuring "how much the quantization hurt" – it's measuring "how much did the abliteration shift the output distribution", which is consistent regardless of precision.

Ran the same optimization just yesterday on Qwen3-4B in both bf16 and bnb4bit (again, as I did before the PR) – final refusal counts and KL were only marginally different. This suggests the quantization format doesn't significantly bias the optimization or the KL metric. -> https://huggingface.co/arnomatic/Qwen3-4B-Instruct-2507-heretic

Right, I think we're agreeing here actually - "fair comparison within format" means you can't compare the 0.06 vs 0.32 numbers across formats to conclude one approach is better than the other. They're measuring the same thing (distribution shift from abliteration) but on different scales.

Your Qwen3-4B experiment is interesting though - if bf16 and bnb4bit produced only marginally different results on the same model, that suggests the optimization isn't dramatically better in one format vs the other. Which is kind of what I'd expect if the KL metric is consistent within each format.

[pszemraj]

I've come across this fundamental problem several times during development, and I don't think that there's much we can do about it because trials are so expensive in our case.

perhaps a separate thread, have you already explored solutions that enable "don't reload the whole model every time, just reset any changed layer(s)+ components back to original"? don't want to waste time suggesting anything if so

edit: I was testing with pypi release version. it seems the LoRA is in main, and forced on all runs. honestly.... long story but if that is truly the case, that's my red flag and I'm not sure I will be involved with the project further.

[pszemraj]

Makes sense, thanks for the clarification - I understand now that this is inherent variance from separate runs rather than something pathological.

I think the logging/persistence feature you mentioned would address the practical side of this. If results across runs could be merged (or at least compared), then running multiple shorter studies and combining Pareto fronts would be a viable strategy rather than hoping a single long run gets lucky. Would also help with the architecture validation workflow I mentioned.

[p-e-w]

Closing as invalid. The premise of this issue is impossible, as discussed.