QLoRA resource reality check with SGLang

[deepseeked]

I'm trying to sanity-check my setup.

My current setup is SGLang with A6000, and I keep running into long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

[batchsize]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

I'm happy to test one more variant if you want a second data point.

[batchpeak]

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[batchsize]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. Once you know the plain path works, then add the faster path back in piece by piece.

[benchplease]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[routearch]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[tensorfield88]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.