Posts: 49
Threads: 6
Joined: Sep 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mixtral, PyTorch, GB200, with a lot of attention on KV cache pressure.
Maybe useful as a starting point for newer folks.
This is mostly for local work rather than public serving.
Posts: 42
Threads: 10
Joined: Aug 2026
Reputation:
0
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Posts: 40
Threads: 2
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Posts: 76
Threads: 8
Joined: Aug 2026
Reputation:
0
A couple questions before I guess too hard. Are you testing this on B200 or something else?
learning in public
Posts: 51
Threads: 8
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output.
learning in public
Posts: 24
Threads: 2
Joined: Aug 2026
Reputation:
0
Appreciate the detail. I ended up in about the same place using TensorRT-LLM.
Posts: 25
Threads: 4
Joined: Aug 2026
Reputation:
0
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.
small models can still be good
Posts: 36
Threads: 3
Joined: Oct 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.
still benchmarking everything
