recipes for a good local stack on 4090

[mintstation99]

Did a bit of testing and thought I'd post the result here.

I tried this with Gemma using vLLM on dual 3090. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[visionforge]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[vramstream]

Interesting, I hadn't tried that. I ended up in about the same place using MLX.

[mlxgarden]

That matches what I was seeing. That matches what I saw on 7900 XTX.

[vramstream]

That matches what I was seeing. That matches what I saw on 7900 XTX.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.

[relayforge]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[mellowwire98]

I'd start with the simpler stack.