desk setup thread

[modbench]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on dual 3090, keeping VRAM headroom in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

[routecraft]

Thanks, that helps. The pattern around continuous batching has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vrambandit]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[latentcraft]

worth checking the driver first.

If you post your exact settings, people here can probably narrow it down pretty fast.

[torchsmith25]

worth checking the driver first.

I'd start with the simpler stack.

[mangopath4]

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[fusedhall]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[signalcraft14]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[cobaltworks25]

Appreciate the detail. The pattern around continuous batching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fusedhall56]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.