simple local serving recipe on 5090

[rocmriley]

Curious what people here are seeing.

I'm mostly working with Phi in PyTorch on mini PC. The part I'm trying to understand is HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.

[cachebench]

I'd start with the simpler stack.

[rocmriley]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[deltagrid]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmhouse90]

Thanks, that helps. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rocmriley]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[kernelgrid]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vramstream]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[rocmhouse90]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.

[mlxgarden]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once yo

I'd start with the simpler stack.