+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: draft model choices on mini PC (/showthread.php?tid=142)
Pages:
1
2
draft model choices on mini PC - cacheline - 09-05-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and B200.
Simple and boring is a valid answer.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: draft model choices on mini PC - cacheline - 09-06-2026
Ran a quick check on my box. On 4090 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: draft model choices on mini PC - benchplease - 09-06-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: draft model choices on mini PC - cacheline - 09-06-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: draft model choices on mini PC - audiopath66 - 09-06-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: draft model choices on mini PC - cacheline - 09-09-2026
I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: draft model choices on mini PC - cacheline - 09-09-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: draft model choices on mini PC - cudaline - 09-09-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: draft model choices on mini PC - cacheline - 09-10-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.
Re: draft model choices on mini PC - phaseshed - 09-10-2026
I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.