Posts: 41
Threads: 7
Joined: Sep 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Kimi, Ollama, 4090, with a lot of attention on NVFP4.
Feel free to tear it apart if you see something dumb.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
small models can still be good
Posts: 54
Threads: 8
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. Are you testing this on 5090 or something else?
Posts: 73
Threads: 8
Joined: Sep 2026
Reputation:
0
Appreciate the detail. The pattern around prefill/decode split has been pretty similar for me.
Posts: 41
Threads: 7
Joined: Sep 2026
Reputation:
0
Good point. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
small models can still be good
Posts: 40
Threads: 9
Joined: Sep 2026
Reputation:
0
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Not glamorous advice, but repeatable baselines save a lot of time.
Posts: 43
Threads: 5
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. Not glamorous advice, but repeatable baselines save a lot of time.
Posts: 73
Threads: 8
Joined: Sep 2026
Reputation:
0
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Posts: 67
Threads: 5
Joined: Aug 2026
Reputation:
0
Appreciate the detail. The pattern around workstation thermals has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.
curious and caffeinated
Posts: 30
Threads: 3
Joined: Sep 2026
Reputation:
0
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
