QLoRA resource reality check

[routerunner]

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring MoE serving.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[llmtrack26]

What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[sparrowworks88]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[kernelarch]

Not sure I agree there. I've seen people blame prefill/decode split when the real issue was much more mundane.

[quiethouse]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[signalpeak]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure sho

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quiethouse]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[nanobrook]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[batchsize]

following this because I'm in the same boat.

[vramarch]

I'd start with the simpler stack.