Tensor cores actually helping here with LM Studio

[edgecoast]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of vLLM and B200 felt surprisingly usable once I tuned around HBM constraints.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

[pixelspark]

I hit almost the same issue last night.

[syshouse]

I hit almost the same issue last night.

[orbitbench6]

I tested this tonight. On 7900 XTX with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[audiorunner74]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[kernelmike]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

Thanks, that helps. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sparrowkite53]

Interesting, I hadn't tried that. That matches what I saw on mini PC.

[amberpatch25]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockember]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make

worth checking the driver first.

[orbitbench6]

Appreciate the detail. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.