+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: best GGUF for 24GB VRAM (/showthread.php?tid=69)
Pages:
1
2
best GGUF for 24GB VRAM - visionyard23 - 08-29-2026
I figured I'd ask here before changing everything.
I'm mostly working with Mistral in SGLang on B200. The part I'm trying to understand is INT4.
Not looking for the perfect answer, mostly what worked in real use.
This is mostly for local work rather than public serving.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: best GGUF for 24GB VRAM - mangotinker73 - 08-30-2026
I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best GGUF for 24GB VRAM - orbitbench6 - 08-30-2026
Quote:I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. No
A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best GGUF for 24GB VRAM - vectorport - 08-30-2026
Quote:[quote]I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survi
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best GGUF for 24GB VRAM - vrambandit - 09-02-2026
That matches what I was seeing. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: best GGUF for 24GB VRAM - dockridge - 09-02-2026
following this because I'm in the same boat. The logs were somehow calmer than I was.
Re: best GGUF for 24GB VRAM - latentcraft - 09-02-2026
My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: best GGUF for 24GB VRAM - vrambandit - 09-02-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best GGUF for 24GB VRAM - cudascope - 09-02-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best GGUF for 24GB VRAM - dockridge - 09-05-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamo
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?