what changed after the inference shift

[tensorfield88]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

Would rather learn from your scars than collect my own all at once.

[prompttinker44]

Posting this in case it saves someone else a little time.

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fastfield4]

Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[signalcraft14]

Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more

subscribed, curious where this lands.

[vramport]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[sparrowshed]

What exact setup are you running? Is this local-only or are you routing anything externally?

[sanevault]

What exact setup are you running? What model, what runtime, and roughly what context size?

[bytegarden73]

What exact setup are you running? What model, what runtime, and roughly what context size?

that fixed it for me too.

[southspark]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

[vramport]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.