[FR] Prompt Enhancer - llama.cpp integration

[iwr-redmond]

This FR is partly related to #15.

Bundling a llama.cpp implementation with the GUI would allow for prompt enhancement and enrichment functions to be made available to users. This is particularly helpful for DiT models like Flux that significantly benefit from more detailed prompts.

Nexa AI builds both llama.cpp and sd.cpp together for their SDK, although the integration is new and may still be buggy. The cmake file used for Nexa is here.

[fszontagh]

To first shot, i just plan to implement an ollama / openai rest API client to achieve this feature.

For ollama, i just found an interesting model: https://ollama.com/brxce/stable-diffusion-prompt-generator

And there is some interesting results:

ollama run brxce/stable-diffusion-prompt-generator

REALISTIC

Ollama prompt:

>>> epic macro photo of a red rose, water drops, wallpaper, dynamic lights

Model's answer:

epic macro photo of a red rose, water drops, wallpaper, dynamic lights, by ilya kuvshinov and takashi kitamura and artem demura, trending on artstation, very detailed

Final prompt:

epic macro photo of a red rose, water drops, wallpaper, dynamic lights, by ilya kuvshinov and takashi kitamura and artem demura, trending on artstation, very detailed <lora:more_details:0.8>

Final negative prompt:

(worst quality, low quality:1.4, letterboxed), lowres, loli, child, bokeh, text, signature, sketch, watermark, artist name, speech bubble, blurry, pubic hair, pubes, (mole, mole under eye, mole on breast)

schedule type: Karras
steps: 30
model hash: 4fe4ab5ef0
seed: 595707153
operations: txt2img
backend: stable-diffusion.cpp (ver.: fbd42b6 using cuda)
size: 768x512
scheduler: karras
cfg scale: 7.00
app: StableDiffusionGUI 0.2.7 3aab399
parser: stable-diffusion.cpp
sampler: Automatic
model: opiate_V50

ANIME

Ollama prompt:

>>> epic macro photo of a red rose, water drops, wallpaper, dynamic lights, anime

Model's answer:

epic macro photo of a red rose, water drops, wallpaper, dynamic lights, anime, trending on artstation, by greg rutkowski and takashi takahashi and artem demura, very detailed, 8k resolution

Final prompt:

epic macro photo of a red rose, water drops, wallpaper, dynamic lights, anime, trending on artstation, by greg rutkowski and takashi takahashi and artem demura, very detailed, 8k resolution

Final negative prompt:

(worst quality, low quality:1.4, letterboxed), lowres, loli, child, bokeh, text, signature, sketch, watermark, artist name, speech bubble, blurry, pubic hair, pubes, (mole, mole under eye, mole on breast)

schedule type: Karras
steps: 30
model hash: da52bf6da0
seed: 778953140
operations: txt2img
backend: stable-diffusion.cpp (ver.: fbd42b6 using cuda)
size: 768x512
scheduler: karras
cfg scale: 7.00
app: StableDiffusionGUI 0.2.7 3aab399
parser: stable-diffusion.cpp
sampler: Automatic
model: mistoonAnime_v30

So the anime is not the best, but maybe it's just depends on the SD model's knowledge. And they are SD1.X models, so the prompts works differently from the FLUX models, but i think this feature worth it with these models too.

The plan is:

• i will add a new tab into the settings window, where user can configure the REST API settings to an ollama endpoint
• i will add a model selector to the same place. The default and recommended model will be the brxce/stable-diffusion-prompt-generator
• if the user configured it and the endpoint works, then show up a button with name "enhance prompt" near the prompts area, which sends the content of the prompt input to the language model to generate a response. The response will overwrite the original prompt

[iwr-redmond]

I have strong views against Ollama because its lack of proper packaging makes it hard to rely on and prone to error. However, you may wish to bundle vendored Ollama binaries with your own packages to work around this issue.

[fszontagh]

Hmm...
I use ollama without any issues, but mainly on Linux using the curl based installer.

Could you please explain what problems you're experiencing with its installers?

[iwr-redmond]

I also use Ollama without major issues, albeit with a customized setup and upgrade script. However, one has to be either savvy or lucky for the provided mechanisms to reliably work. It's fine for more advanced users who can troubleshoot errors and terrible to rely on for ordinary usage by the general public.

Here is just a taste from the issues list:

• Update on Linux should keep existing settings for ollama service
• Make manual updates an option
• update failure
• Warn users that install.sh may break their system
• Create Update command for ollama binary
• olama installer should ask in drive user wants to install it
• version aware linux upgrade
• Documentation for manual Linux installation is outdated/doesn't work for AMD GPU setup
• Feature request : Do not update ollama when OS is on limited data connextion
• Windows Portable Mode
• Make Ollama available on Chocolatey
• (linux) snap support for linux installation this will solve many issues
• Update command for Linux version
• Ollama Auto-Starts on Windows Without User Permission
• Introduce regular support releases

Edit: removed issue relating to ollama.com

[iwr-redmond]

I've experimented with Cortex previously, and this might be an alternative to bundling. Cortex runs as the current user and doesn't require a system service. It is also packaged for Debian, which makes it analogous to your own packages in terms of supporting Windows and Debian/Ubuntu natively.

[fszontagh]

It's sad, but it's okay. Thank You for digging into it!

I don't want to redistribute external apps with the prebuild binaries.

So i will separate the API thing into another FR, and i will go forward with the llama.cpp integration into the GUI. So it can be used not just for prompt enhancment.

I really have to create a roadmap too, there is too mutch to todo and dependencies.

[iwr-redmond]

You may wish to use the Projects feature. Some other repos that use this include:

• dgenerate, which uses a Kanban style board
• FooocusPlus, which uses multiple projects to sort issues into versioned and thematic groups
• diffusers, which publishes a roadmap for each upcoming version

[iwr-redmond]

So it can be used not just for prompt enhancment.

RuinedFooocus has a library of various prompts that you may be interested in here. There is also a humorous Support Troll:

"What did you break now?"