first LoRA that actually helped

[cacheline]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and TensorRT-LLM for Mistral. The interesting difference for me has been long context.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

[quiethouse]

Thanks, that helps. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelghost]

What exact setup are you running? What model, what runtime, and roughly what context size?

[benchplease]

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[rocmplane]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make

I'd start with the simpler stack.

[kernelarch]

that fixed it for me too.