latest release actually worth attention: anyone testing this?

[mangowire]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on Threadripper box, keeping speculative decoding in mind from the start.

If someone wants the exact settings I can clean them up and post them.

[phaseshed]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.

[vectorpoint]

A couple things I'd check first. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.