driver/framework news roundup

[deepseeked]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like router policy decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

[mangowire]

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me.

[vectorpoint]

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[deepseeked]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

[bytelab]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[nanogarden]

I hit almost the same issue last night.

[marbletrack]

That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mangowire]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[bytelab]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[stackpilot]

[quote]I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.[/quote

following this because I'm in the same boat.