Ollama for power users on Threadripper box

[embershed99]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate long context until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.

[amberhall]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[edgekite]

That matches what I was seeing. That matches what I saw on B200.

[orbitbench6]

Appreciate the detail. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[edgeriver]

Not sure I agree there. I've seen people blame batch size when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[fusedhall]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[vectorcrate98]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[amberhall]

Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[edgeriver]

Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one

worth checking the driver first.