+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: Off Topic (https://www.datasculpting.ai/inference/forumdisplay.php?fid=27)
+--- Thread: desk setup thread (/showthread.php?tid=820)
Pages:
1
2
desk setup thread - modbench - 10-06-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: llama.cpp on dual 3090, keeping VRAM headroom in mind from the start.
If someone wants the exact settings I can clean them up and post them.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: desk setup thread - routecraft - 10-06-2026
Thanks, that helps. The pattern around continuous batching has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: desk setup thread - vrambandit - 10-06-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: desk setup thread - latentcraft - 10-09-2026
worth checking the driver first.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: desk setup thread - torchsmith25 - 10-09-2026
Quote:worth checking the driver first.
I'd start with the simpler stack.
Re: desk setup thread - mangopath4 - 10-09-2026
Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: desk setup thread - fusedhall - 10-09-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: desk setup thread - signalcraft14 - 10-09-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: desk setup thread - cobaltworks25 - 10-11-2026
Appreciate the detail. The pattern around continuous batching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: desk setup thread - fusedhall56 - 10-12-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.