industry move that matters for local users

[deepseeked]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 4090, keeping long context in mind from the start.

Maybe useful as a starting point for newer folks.

[amberrunner]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[southpatch]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[plainember52]

What exact setup are you running? Is this local-only or are you routing anything externally?

[bytelab]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[bytestack]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[fastlab]

That matches what I was seeing. That matches what I saw on Threadripper box. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[dockriver]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[southpatch]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads e

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Not glamorous advice, but repeatable baselines save a lot of time.

[vectorpoint]

That matches what I was seeing. That matches what I saw on 7900 XTX.