draft model choices

[cacheline]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about batch size.

I'm especially interested in mistakes worth avoiding.

[cacheline]

Posting this in case it saves someone else a little time.

Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[llmarch51]

That matches what I was seeing. I ended up in about the same place using Ollama.

[cacheline]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cobaltlab]

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

[cacheline]

What exact setup are you running? What model, what runtime, and roughly what context size?

[relaynest82]

Did a small comparison this afternoon. On M3 Max with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality. I keep promising myself this is the last rebuild.

[cacheline]

Did a small comparison this afternoon. On M3 Max with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality

same here on my side.

[torchtrack24]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.

[amberkite]

That matches what I was seeing. That matches what I saw on Threadripper box. The biggest change for me was simplifying the stack instead of adding one more optimization layer.