first LoRA that actually helped

[batchsize]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

[localtinker]

This keeps coming up in conversation so I figured I'd start a thread.

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[torchshed66]

Need a little more detail here. Are you testing this on Threadripper box or something else?

[kernelarch]

Good point. That matches what I saw on A6000.

[rocmrunner40]

subscribed, curious where this lands.

[benchplease]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[localtinker]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[batchfield55]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not qualit

Thanks, that helps. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cacheline]

[quote]Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[benchplease]

Good point. The pattern around speculative decoding has been pretty similar for me.