+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: disaggregated inference thoughts (/showthread.php?tid=1306)
Pages:
1
2
disaggregated inference thoughts - cachebench - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Kimi, LM Studio, 7900 XTX, with a lot of attention on continuous batching.
If someone wants the exact settings I can clean them up and post them.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: disaggregated inference thoughts - prompttinker44 - 10-16-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: disaggregated inference thoughts - tensorfield88 - 10-18-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: disaggregated inference thoughts - sparrowshed - 10-18-2026
Quote:Need a little more detail here. Is this local-only or are you routing anything externally?
following this because I'm in the same boat.
Re: disaggregated inference thoughts - rocmhouse90 - 10-18-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: disaggregated inference thoughts - turbogrid11 - 10-19-2026
Quote:I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survi
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: disaggregated inference thoughts - turboscope5 - 10-19-2026
Quote:Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimizat
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: disaggregated inference thoughts - kernelrunner - 10-19-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: disaggregated inference thoughts - rocmhouse90 - 10-19-2026
I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: disaggregated inference thoughts - localpatch78 - 10-19-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. A lot of the
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.