+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: draft model choices (/showthread.php?tid=1166)
Pages:
1
2
draft model choices - cacheline - 10-13-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with quant drift.
Maybe obvious to some of you, but it clicked for me this week.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: draft model choices - promptpoint - 10-13-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: draft model choices - cacheline - 10-14-2026
that fixed it for me too.
Re: draft model choices - vramport - 10-15-2026
A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: draft model choices - cacheline - 10-15-2026
Appreciate the detail. That matches what I saw on mini PC. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: draft model choices - llm_wren - 10-15-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: draft model choices - cacheline - 10-17-2026
Tried to reproduce it just now. On 4090 with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.
Re: draft model choices - audioforge - 10-17-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: draft model choices - cacheline - 10-17-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: draft model choices - plainworks - 10-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.