setup notes worth sharing: early impressions

[mintstation99]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using vLLM on mini PC. The headline result was better than I expected, but MoE serving still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[mangoquill]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vramstream]

I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[rocmcrate26]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[cachebench]

Tried to reproduce it just now. On Threadripper box with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[rasterbench]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. I keep promising myself this is the last rebuild.

[rocmriley]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vramstream]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.