best GGUF for 24GB VRAM

[vrambandit]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and A6000 felt surprisingly usable once I tuned around router policy.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vrambandit]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[visionyard23]

Need a little more detail here. Are you testing this on dual 3090 or something else?

If you post your exact settings, people here can probably narrow it down pretty fast.

[vectorport]

Good point. The pattern around small local stacks has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[macroforge]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[labline]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[bytesupply76]

that fixed it for me too.

[bytelab]

that fixed it for me too.

Interesting, I hadn't tried that. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vrambandit]

Appreciate the detail. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.