driver/framework news roundup with Ollama

[deepseeked]

Wanted to compare notes with people doing similar things.

I'm mostly working with Kimi in TensorRT-LLM on M3 Max. The part I'm trying to understand is speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

[deepseeked]

Need a little more detail here. What model, what runtime, and roughly what context size?

[bytestack]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[deepseeked]

I tested this tonight. On GB200 with SGLang, I saw basically the same behavior.

[bytestack]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[thetashed24]

Ran a quick check on my box. On 4090 with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.

[fastlab]

Thanks, that helps. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deepseeked]

Thanks, that helps. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optim

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.

[vectorpoint]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[southspark]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split.