+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: simple local serving recipe on 5090 (/showthread.php?tid=432)
Pages:
1
2
simple local serving recipe on 5090 - rocmriley - 09-23-2026
Curious what people here are seeing.
I'm mostly working with Phi in PyTorch on mini PC. The part I'm trying to understand is HBM constraints.
Not looking for the perfect answer, mostly what worked in real use.
Re: simple local serving recipe on 5090 - cachebench - 09-24-2026
I'd start with the simpler stack.
Re: simple local serving recipe on 5090 - rocmriley - 09-24-2026
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: simple local serving recipe on 5090 - deltagrid - 09-24-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple local serving recipe on 5090 - rocmhouse90 - 09-24-2026
Thanks, that helps. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: simple local serving recipe on 5090 - rocmriley - 09-26-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple local serving recipe on 5090 - kernelgrid - 09-26-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: simple local serving recipe on 5090 - vramstream - 09-27-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: simple local serving recipe on 5090 - rocmhouse90 - 09-27-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.
Re: simple local serving recipe on 5090 - mlxgarden - 09-28-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once yo
I'd start with the simpler stack.