+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: News & Industry (https://www.datasculpting.ai/inference/forumdisplay.php?fid=26)
+--- Thread: driver/framework news roundup with Ollama (/showthread.php?tid=1163)
Pages:
1
2
driver/framework news roundup with Ollama - deepseeked - 10-14-2026
Wanted to compare notes with people doing similar things.
I'm mostly working with Kimi in TensorRT-LLM on M3 Max. The part I'm trying to understand is speculative decoding.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: driver/framework news roundup with Ollama - deepseeked - 10-14-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: driver/framework news roundup with Ollama - bytestack - 10-14-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off.
Re: driver/framework news roundup with Ollama - deepseeked - 10-14-2026
I tested this tonight. On GB200 with SGLang, I saw basically the same behavior.
Re: driver/framework news roundup with Ollama - bytestack - 10-17-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: driver/framework news roundup with Ollama - thetashed24 - 10-17-2026
Ran a quick check on my box. On 4090 with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: driver/framework news roundup with Ollama - fastlab - 10-17-2026
Thanks, that helps. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: driver/framework news roundup with Ollama - deepseeked - 10-18-2026
Quote:Thanks, that helps. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optim
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.
Re: driver/framework news roundup with Ollama - vectorpoint - 10-18-2026
A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: driver/framework news roundup with Ollama - southspark - 10-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split.