+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: disaggregated inference thoughts with llama.cpp (/showthread.php?tid=212)
Pages:
1
2
disaggregated inference thoughts with llama.cpp - quietcraft - 09-11-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of LM Studio and A6000 felt surprisingly usable once I tuned around prefix caching.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: disaggregated inference thoughts with llama.cpp - vramport - 09-11-2026
I tested this tonight. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: disaggregated inference thoughts with llama.cpp - tensorfield88 - 09-11-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: disaggregated inference thoughts with llama.cpp - rocmhouse90 - 09-12-2026
I hit almost the same issue last night.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: disaggregated inference thoughts with llama.cpp - kernelrunner - 09-13-2026
Quote:I hit almost the same issue last night.
What exact setup are you running? What model, what runtime, and roughly what context size?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: disaggregated inference thoughts with llama.cpp - vramport - 09-13-2026
I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: disaggregated inference thoughts with llama.cpp - rocmpath - 09-15-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: disaggregated inference thoughts with llama.cpp - papermixer4 - 09-15-2026
I'd start with the simpler stack.
Re: disaggregated inference thoughts with llama.cpp - localpatch - 09-15-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: disaggregated inference thoughts with llama.cpp - kernelrunner - 09-15-2026
Quote:I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing
Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.