QLoRA resource reality check

[cacheline]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.

[batchsize]

I hit almost the same issue last night.

[griffinloom66]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchsize]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[torchtrack24]

Interesting, I hadn't tried that. The pattern around tail latency has been pretty similar for me.

[quiethouse]

What exact setup are you running? Is this local-only or are you routing anything externally?

[nanobrook]

What exact setup are you running? Is this local-only or are you routing anything externally?

Thanks, that helps. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deltastation]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[vectorpoint]

A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[cudascope]

Maybe, but I had the opposite result. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.