sd: update to master-377-2034588

[wbruna]

Github closed #1841 when I tried to rename the PR branch.

I updated to master-358-347710f, but the at-runtime LoRA loading is tangled up with the model loading from disk, so it's not that simple to call from Koboldcpp. For now, I've kept the loading as before. Edit: I adapted the LoRA code, and added a fix for loading LoRAs with sdquant.

Edit: now at master-371-5498cc0 master-377-2034588 .

[wbruna]

Updated to master-365-d5b05f7 master-366-f532972, since the changes seem low-risk enough.

[LostRuins]

sorry for the delays, been caught up with stuff and also laptop repair issues. i will get to this soon i hope

[wbruna]

And now we have master-377-2034588 😅 I'll be working on it, but let me know if it's better to move it to a follow-up PR instead...

Edit: done. I think I'll just keep moving it forward; I can move some commits to another PR and truncate this branch, if needed.

[LostRuins]

Man sd.cpp upstream is suddenly moving so fast... i can't keep up with the constant changes lol.

Okay i'm gonna try integrating this shortly, if you have any additions beyond this let's save it for the next PR.

For now we need to test and make sure this is stable and nothing broke, since I don't think I can review this much code manually. I'm gonna have to rely on runtime testing and checking.

[wbruna]

Just a suggestion: since most of these commits were either updates or fixes right after the updates, it may be better to just rebase the whole branch as-is, instead of collapsing it onto a single commit. That way, if we do find a bug, it'll be easier to bisect.

[wbruna]

One point that I'd like you to take a look anyway is the LoRA loading. Upstream code changed a lot, and I needed a different approach to make it work with Koboldcpp changes (mostly on b18aa90621ea7ec8b564c779cbc9ef8835cb9a82). The resulting diff against upstream for master-358 is: wbruna/stable-diffusion.cpp@347710f...4d822db ; I can't reliably link to the lines, but look for the apply_lora_from_file function).

I also added backend support for the LoRA loading mode to make testing easier, but I don't know of any LoRAs that need it to be manually changed on recent revisions. I could remove it from the history, if you prefer to keep koboldcpp.py unchanged through this update.

[LostRuins]

your current build isn't compiling due to lacking 2 things, however i dont have push access to your PR branch so you'll have to patch it yourself

in stable-diffusion.cpp add
#include <filesystem>

and change this lora part

  std::string lora_ident = std::filesystem::path(file_path).stem();

to this

  #ifdef _WIN32
      std::string lora_ident = std::filesystem::u8path(file_path).stem().u8string();
  #else
      std::string lora_ident = std::filesystem::path(file_path).stem().string();
  #endif

otherwise, that part is fine. so far i have tested sd1.5 and sd1.5 loras and they look ok. havent tested other archs yet

[wbruna]

Done: wbruna/stable-diffusion.cpp@872211f .

Those #ifdefs should probably be moved to a helper function. I can refactor them afterwards.

[LostRuins]

Just a suggestion: since most of these commits were either updates or fixes right after the updates, it may be better to just rebase the whole branch as-is, instead of collapsing it onto a single commit. That way, if we do find a bug, it'll be easier to bisect.

how about a hybrid approach where we break these commits into 3-4 main complete checkpoints? just squash them into feature complete chunks. dumping the whole history might be messy idk

[wbruna]

Sure, that works.

Master 366 is a small diff, it can be folded with 365. Each bunch of LoRA fixes can be folded with the previous master update (and I'd cheat a bit and fold the last fix onto 358). So we'd have: 355, 358, 366, 371, 377.

[LostRuins]

Testing Results:

• SD1.5 safetensors txt2img compress q4 and f16: looks ok
• SDXL safetensors txt2img with lora, compress q4: looks ok but image substantially different (at-runtime triggered) <- trying to load lora without at-runtime crashes, created a comment feat: support applying LoRA at runtime leejet/stable-diffusion.cpp#969 (comment)
• qwen image q4_k_s txt2img: looks ok
• qwen image edit q4_k_s with 1 ref image: looks ok, but image changed from before
• kontext with 2 reference images: works but output quality is very low. might be due to some unrelated earlier regression
• flux: looks ok
• wan2.2: looks ok
• photomaker v1: looks ok

didn't try the others yet, feel free to test too.

for now i think it's in a "mergable" state so i will merge later if no objections

[LostRuins]

@wbruna if its not too much trouble, could you do your magic again with the latest upstream commit so we can try flux2