draft model choices on mini PC

[cacheline]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and B200.

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

[cacheline]

Ran a quick check on my box. On 4090 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[benchplease]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[cacheline]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[audiopath66]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[cacheline]

I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[cacheline]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cudaline]

Need a little more detail here. What model, what runtime, and roughly what context size?

[cacheline]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.

[phaseshed]

I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.