+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: setup notes worth sharing (/showthread.php?tid=766)
Pages:
1
2
setup notes worth sharing - vectorport - 10-05-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.
Wondering if this is just where the whole space is heading now.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: setup notes worth sharing - mlxgarden - 10-05-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.
Re: setup notes worth sharing - rasterbench - 10-05-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: setup notes worth sharing - mlxgarden - 10-05-2026
Thanks, that helps. I don't think your conclusion is far off.
Re: setup notes worth sharing - cacheworks - 10-05-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: setup notes worth sharing - llmpeak - 10-05-2026
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: setup notes worth sharing - audioriver - 10-07-2026
Quote:I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Can you fill in a bit more context? Are you testing this on M3 Max or something else?
Re: setup notes worth sharing - kernelgrid - 10-07-2026
Quote:[quote]I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obviou
What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: setup notes worth sharing - audionode8 - 10-07-2026
What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: setup notes worth sharing - fastlab - 10-07-2026
Not sure I agree there. I've seen people blame HBM constraints when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.