first LoRA that actually helped

first LoRA that actually helped - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=43)
+--- Thread: first LoRA that actually helped (/showthread.php?tid=450)

first LoRA that actually helped - cacheline - 09-24-2026

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and TensorRT-LLM for Mistral. The interesting difference for me has been long context.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

Re: first LoRA that actually helped - quiethouse - 09-25-2026

Thanks, that helps. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

Re: first LoRA that actually helped - kernelghost - 09-28-2026

What exact setup are you running? What model, what runtime, and roughly what context size?

Re: first LoRA that actually helped - benchplease - 09-28-2026

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Re: first LoRA that actually helped - rocmplane - 09-28-2026

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

Re: first LoRA that actually helped - cacheline - 09-29-2026

Quote:Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make

I'd start with the simpler stack.

Re: first LoRA that actually helped - kernelarch - 09-29-2026

that fixed it for me too.