Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
best local runtime for mixed models with Ollama
#1
Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once token cost becomes important.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.
Reply
#2
I'd start with the simpler stack.
Reply
#3
What exact setup are you running? Are you testing this on 4090 or something else?
Reply
#4
Quote:What exact setup are you running? Are you testing this on 4090 or something else?

subscribed, curious where this lands.
Reply
#5
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.
trying not to rebuild again
Reply
#6
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. Once you know the plain path works, then add the faster path back in piece by piece.
Reply
#7
What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Reply
#8
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Reply
#9
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.
trying not to rebuild again
Reply
#10
What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)