Really late, but tested on my home setup:
2x 2690 V3s, 256gb of DDR4-2133MHz LRDIMMs, 3090 + 3080ti (sm86)
test-backend-ops: 19/19 passed on cuda0, cuda1, and cpu
Real model runs with DSV4-Flash Q4_K_M-XL from teamblobfish
-ngl 99 -cmoe -c 262144 -b 2048 -sm layer -ts 68,32 -fa off
notes: modifying ub seems to oom the cards, but not entirely clear to me why (with the above command have approx 18gb of free vram between the 3090 (12gb free) and 3080ti (6gb free). I suspect the 3080ti is holding the ub back).
pp: ~10tps, tg: ~7tps (from a few prompts of my own, can run llama-bench for more comparable numbers later, but i figure this gives a ballpark). regression to ~5tps with fa on (I suspect because of the new CSA in DSV4)
thanks for the great work, really awesome to be able to run dsv4 flash at home.
Apologies in advance for formatting errors, trying to get more into contributing but not always sure the best way to. Thanks in advance!
Really late, but tested on my home setup:
2x 2690 V3s, 256gb of DDR4-2133MHz LRDIMMs, 3090 + 3080ti (sm86)
test-backend-ops: 19/19 passed on cuda0, cuda1, and cpu
Real model runs with DSV4-Flash Q4_K_M-XL from teamblobfish
-ngl 99 -cmoe -c 262144 -b 2048 -sm layer -ts 68,32 -fa off
notes: modifying ub seems to oom the cards, but not entirely clear to me why (with the above command have approx 18gb of free vram between the 3090 (12gb free) and 3080ti (6gb free). I suspect the 3080ti is holding the ub back).
pp: ~10tps, tg: ~7tps (from a few prompts of my own, can run llama-bench for more comparable numbers later, but i figure this gives a ballpark). regression to ~5tps with fa on (I suspect because of the new CSA in DSV4)
thanks for the great work, really awesome to be able to run dsv4 flash at home.
Apologies in advance for formatting errors, trying to get more into contributing but not always sure the best way to. Thanks in advance!