+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: disaggregated inference thoughts: what are you seeing? (/showthread.php?tid=1048)
Pages:
1
2
disaggregated inference thoughts: what are you seeing? - localpatch78 - 10-11-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.
Would rather learn from your scars than collect my own all at once.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: disaggregated inference thoughts: what are you seeing? - bytestack - 10-13-2026
Not sure I agree there. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere.
Re: disaggregated inference thoughts: what are you seeing? - papermixer4 - 10-13-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. At this point the real bottleneck is my patience.
Re: disaggregated inference thoughts: what are you seeing? - vramport - 10-13-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality. Turns out 'just one more test' was a lie.
Re: disaggregated inference thoughts: what are you seeing? - vrambandit - 10-13-2026
Quote:I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality. Turns
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: disaggregated inference thoughts: what are you seeing? - papermixer4 - 10-13-2026
That matches what I was seeing. The pattern around prefix caching has been pretty similar for me.
Re: disaggregated inference thoughts: what are you seeing? - edgeriver - 10-14-2026
Thanks, that helps. That matches what I saw on 7900 XTX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. At this point the real bottleneck is my patience.
Re: disaggregated inference thoughts: what are you seeing? - localpatch - 10-14-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: disaggregated inference thoughts: what are you seeing? - vramport - 10-17-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: disaggregated inference thoughts: what are you seeing? - kernelrunner - 10-17-2026
Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.