QLoRA resource reality check on Threadripper box

[turboyard94]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and A6000.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[cacheline]

Posting this in case it saves someone else a little time.

Did a small comparison this afternoon. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

[deltastation]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[batchsize]

worth checking the driver first.

[cacheline]

subscribed, curious where this lands.

[batchsize]

My first guess would be this: On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

If you post your exact settings, people here can probably narrow it down pretty fast.

[sanevault]

following this because I'm in the same boat.

[deltastation]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. I keep promising myself this is the last rebuild.

[nanobrook]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelghost]

Ran a quick check on my box. On A6000 with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.