Used A6000 worth it now

[embersmith]

Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring structured output.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[rocmriley]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time. I keep promising myself this is the last rebuild.

[batchember]

Maybe, but I had the opposite result. I've seen people blame prefix caching when the real issue was much more mundane.

[vrambandit]

Good point. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turboyard94]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. Once you know the plain path works, then add the faster path back in piece by piece.

[dockkite69]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmriley]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at wh

following this because I'm in the same boat.

[amberrunner]

Thanks, that helps. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vrambandit]

same here on my side.

[rocmriley]

same here on my side.

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.