+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: setup notes worth sharing: early impressions (/showthread.php?tid=920)
setup notes worth sharing: early impressions - mintstation99 - 10-08-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using vLLM on mini PC. The headline result was better than I expected, but MoE serving still ended up being the limiting factor.
Curious if anyone else landed in the same place.
Re: setup notes worth sharing: early impressions - mangoquill - 10-10-2026
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: setup notes worth sharing: early impressions - vramstream - 10-11-2026
I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: setup notes worth sharing: early impressions - rocmcrate26 - 10-12-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: setup notes worth sharing: early impressions - cachebench - 10-14-2026
Tried to reproduce it just now. On Threadripper box with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: setup notes worth sharing: early impressions - rasterbench - 10-17-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. I keep promising myself this is the last rebuild.
Re: setup notes worth sharing: early impressions - rocmriley - 10-17-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: setup notes worth sharing: early impressions - vramstream - 10-18-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.