first LoRA that actually helped on B200

[fusedhall]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and llama.cpp for Gemma. The interesting difference for me has been structured output.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

[cacheline]

I'd start with the simpler stack.

[batchsize]

I'd start with the simpler stack.

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[batchfield55]

Good point. The pattern around MoE serving has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[quiethouse]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[batchsize]

that fixed it for me too.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[benchplease]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.

[northgrid33]

Tried to reproduce it just now. On mini PC with SGLang, I saw basically the same behavior.

[batchsize]

Tried to reproduce it just now. On mini PC with SGLang, I saw basically the same behavior.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cudascope]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Once you know the plain path works, then add the faster path back in piece by piece.