recipes for a good local stack

[modgarden]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like NVFP4 decides everything downstream.

Wondering if this is just where the whole space is heading now.

[kernelgrid]

This keeps coming up in conversation so I figured I'd start a thread.

Appreciate the detail. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[quantyard]

This keeps coming up in conversation so I figured I'd start a thread.

Ran a quick check on my box. On A6000 with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[vectorport]

This keeps coming up in conversation so I figured I'd start a thread.

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.

[modgarden]

Good point. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmcrate26]

Good point. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look

Good point. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmriley]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[relayforge]

What exact setup are you running? What model, what runtime, and roughly what context size?

[tinycontext]

What exact setup are you running? What model, what runtime, and roughly what context size?

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[mlxgarden]

same here on my side.