best GGUF for 24GB VRAM: early impressions

[pixelcrate82]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and 7900 XTX felt surprisingly usable once I tuned around prefill/decode split.

Maybe useful as a starting point for newer folks.

[northcraft90]

Thanks, that helps. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[pixelcrate82]

Thanks, that helps. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whet

My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[mellowwire98]

A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[embershed99]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmpeak]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[macroforge]

subscribed, curious where this lands.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[visionyard23]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vectorport]

Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[tensorkite]

subscribed, curious where this lands.