Hi, I wanted to report that Gemma 4 output quality with the E4B model is pretty clearly subpar compared to other quants of similar size (in this case Unsloth's gemma-4-E4B-it-qat-UD-Q4_K_XL). This may or may not be due to the outdated template or something to do with quantization, but in any ase it's a bit of a shame since normally it would be one of the preferred models to use for small tool-use applications in my case. Since it's a bit hard to just showcase what I'm talking about in one go, I made a small repository that can be used to easily reproduce the issue:
https://github.com/BoneHorror/flm_g4e4b_quality_test
But the tl;dr of what one could expect is:
Scenario:
The model, with previous conversation history, is asked to find balance changes made to "Marines". It does not have earlier knowledge of Marines or other specifics, but Gemma 4 E4B can typically infer most of what's needed to complete this query.
Answers -
When FLM is generating: (G4 E4B)
<tool>Model is listing docs in: balance_changes
I've listed the files in the `balance_changes` directory. It seems to be organized by race, with files for Terran (`terran.txt`), UED (`ued.txt`), Zerg (`zerg.txt`), and Protoss (`protoss.txt`).
To find the specific balance changes made to Marines, I would need you to tell me which race group (Terran, UED, Zerg, or Protoss) Marines belong to in this documentation structure so I can read the correct file for you!
When llama-server is generating with G4 E4B Q4_K_XL:
<tool>Model is listing docs in: balance_changes
<tool>Model is reading: balance_changes/terran.txt
Reading /home/user/Repositories/discord_bot/docs/balance_changes/terran.txt
Based on the `terran.txt` documentation, here are the balance changes made to Marines:
* **Cost:** Decreased to **40** (from 50).
* **Train Time:** Decreased to **320 frames** (from 360).
* **Note:** Terran units no longer include Medics; these roles were moved to the UED faction.
This is a typical scenario, you can find more examples in the repo linked above. You can see that via FLM the model stopped short of completing the query properly, even though as seen in the other case it could absolutely find the information.
I have previously tried to report it on the project's Discord but ended up not having enough time to write it up properly. I hope someone can take a look here, as well.
Hi, I wanted to report that Gemma 4 output quality with the E4B model is pretty clearly subpar compared to other quants of similar size (in this case Unsloth's gemma-4-E4B-it-qat-UD-Q4_K_XL). This may or may not be due to the outdated template or something to do with quantization, but in any ase it's a bit of a shame since normally it would be one of the preferred models to use for small tool-use applications in my case. Since it's a bit hard to just showcase what I'm talking about in one go, I made a small repository that can be used to easily reproduce the issue:
https://github.com/BoneHorror/flm_g4e4b_quality_test
But the tl;dr of what one could expect is:
Scenario:
The model, with previous conversation history, is asked to find balance changes made to "Marines". It does not have earlier knowledge of Marines or other specifics, but Gemma 4 E4B can typically infer most of what's needed to complete this query.
Answers -
When FLM is generating: (G4 E4B)
When llama-server is generating with G4 E4B Q4_K_XL:
This is a typical scenario, you can find more examples in the repo linked above. You can see that via FLM the model stopped short of completing the query properly, even though as seen in the other case it could absolutely find the information.
I have previously tried to report it on the project's Discord but ended up not having enough time to write it up properly. I hope someone can take a look here, as well.