+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=43)
+--- Thread: QLoRA resource reality check (/showthread.php?tid=701)
Pages:
1
2
QLoRA resource reality check - cacheline - 10-03-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been small local stacks.
Interested in tradeoffs more than fan loyalty.
This is mostly for local work rather than public serving.
Re: QLoRA resource reality check - batchsize - 10-03-2026
I hit almost the same issue last night.
Re: QLoRA resource reality check - griffinloom66 - 10-03-2026
That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: QLoRA resource reality check - batchsize - 10-04-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: QLoRA resource reality check - torchtrack24 - 10-05-2026
Interesting, I hadn't tried that. The pattern around tail latency has been pretty similar for me.
Re: QLoRA resource reality check - quiethouse - 10-05-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: QLoRA resource reality check - nanobrook - 10-05-2026
Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
Thanks, that helps. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: QLoRA resource reality check - deltastation - 10-06-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: QLoRA resource reality check - vectorpoint - 10-06-2026
A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: QLoRA resource reality check - cudascope - 10-06-2026
Maybe, but I had the opposite result. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.