best GGUF for 24GB VRAM

[visionyard23]

I figured I'd ask here before changing everything.

I'm mostly working with Mistral in SGLang on B200. The part I'm trying to understand is INT4.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[mangotinker73]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[orbitbench6]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. No

A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[vectorport]

[quote]I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survi

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[vrambandit]

That matches what I was seeing. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[dockridge]

following this because I'm in the same boat. The logs were somehow calmer than I was.

[latentcraft]

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[vrambandit]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.

[cudascope]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.

[dockridge]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamo

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?