what truly stays local

[modgarden]

I figured I'd ask here before changing everything.

I'm mostly working with Llama in Ollama on 7900 XTX. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[rocmhouse90]

Thanks, that helps. The pattern around NVFP4 has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[llmpeak]

Interesting, I hadn't tried that. The pattern around prefill/decode split has been pretty similar for me.

[northhall88]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4.

[audioriver]

I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[mangoquill]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[modgarden]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.