+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: llama.cpp / GGUF (https://www.datasculpting.ai/inference/forumdisplay.php?fid=35)
+--- Thread: best GGUF for 24GB VRAM (/showthread.php?tid=643)
best GGUF for 24GB VRAM - vrambandit - 10-01-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of Ollama and A6000 felt surprisingly usable once I tuned around router policy.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: best GGUF for 24GB VRAM - vrambandit - 10-01-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: best GGUF for 24GB VRAM - visionyard23 - 10-04-2026
Need a little more detail here. Are you testing this on dual 3090 or something else?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: best GGUF for 24GB VRAM - vectorport - 10-04-2026
Good point. The pattern around small local stacks has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: best GGUF for 24GB VRAM - macroforge - 10-04-2026
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best GGUF for 24GB VRAM - labline - 10-05-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: best GGUF for 24GB VRAM - bytesupply76 - 10-05-2026
that fixed it for me too.
Re: best GGUF for 24GB VRAM - bytelab - 10-05-2026
Quote:that fixed it for me too.
Interesting, I hadn't tried that. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: best GGUF for 24GB VRAM - vrambandit - 10-05-2026
Appreciate the detail. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.