70B still worth the pain

[fastquill74]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[macropoint]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[kernelghost]

I'd start with the simpler stack.

[plaingarden18]

subscribed, curious where this lands.

[phaseport36]

Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routerunner]

following this because I'm in the same boat.