Posts: 43
Threads: 5
Joined: Sep 2026
Reputation:
0
Did a bit of testing and thought I'd post the result here.
I tried this with FLUX using Ollama on 5090. The headline result was better than I expected, but prefix caching still ended up being the limiting factor.
Happy to post more details if anyone wants them.
Posts: 35
Threads: 3
Joined: Sep 2026
Reputation:
0
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 53
Threads: 5
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
small models can still be good
Posts: 123
Threads: 18
Joined: Sep 2026
Reputation:
0
Good point. The pattern around structured output has been pretty similar for me.
post logs or it didn't happen
Posts: 35
Threads: 8
Joined: Aug 2026
Reputation:
0
Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Posts: 123
Threads: 18
Joined: Sep 2026
Reputation:
0
Quote:Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack
A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
post logs or it didn't happen
Posts: 27
Threads: 3
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me.
mostly local
Posts: 34
Threads: 2
Joined: Sep 2026
Reputation:
0
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
small models can still be good
Posts: 39
Threads: 2
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. Are you testing this on GB200 or something else?
