70B still worth the pain

[cudapoint]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

[mintstream9]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[dockbrook21]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[oxidestream]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

Thanks, that helps. That matches what I saw on 5090.

[macropoint]

I hit almost the same issue last night.

[benchplease]

I hit almost the same issue last night.

What exact setup are you running? Are you testing this on 4090 or something else?

[ggufwire22]

that fixed it for me too.

[cudapoint]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[modrunner27]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.