+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: draft model choices (/showthread.php?tid=492)
Pages:
1
2
draft model choices - cacheline - 09-26-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about batch size.
I'm especially interested in mistakes worth avoiding.
Re: draft model choices - cacheline - 09-26-2026
Quote:Posting this in case it saves someone else a little time.
Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: draft model choices - llmarch51 - 09-26-2026
That matches what I was seeing. I ended up in about the same place using Ollama.
Re: draft model choices - cacheline - 09-27-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: draft model choices - cobaltlab - 09-27-2026
A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
Re: draft model choices - cacheline - 09-27-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: draft model choices - relaynest82 - 09-27-2026
Did a small comparison this afternoon. On M3 Max with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality. I keep promising myself this is the last rebuild.
Re: draft model choices - cacheline - 09-27-2026
Quote:Did a small comparison this afternoon. On M3 Max with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality
same here on my side.
Re: draft model choices - torchtrack24 - 09-27-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.
Re: draft model choices - amberkite - 09-27-2026
That matches what I was seeing. That matches what I saw on Threadripper box. The biggest change for me was simplifying the stack instead of adding one more optimization layer.