first LoRA that actually helped on 7900 XTX

[batchsize]

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using llama.cpp on 7900 XTX. The headline result was better than I expected, but prefix caching still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.

[papermixer4]

Did a bit of testing and thought I'd post the result here.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.

[nanobrook]

subscribed, curious where this lands.

[batchpeak]

Interesting, I hadn't tried that. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[cacheline]

Good point. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[quiethouse]

Good point. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[thetashed24]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[cacheline]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to

I hit almost the same issue last night.

[cachelab]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.