disaggregated inference thoughts with llama.cpp

[quietcraft]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and A6000 felt surprisingly usable once I tuned around prefix caching.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[vramport]

I tested this tonight. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[tensorfield88]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[rocmhouse90]

I hit almost the same issue last night.

If you post your exact settings, people here can probably narrow it down pretty fast.

[kernelrunner]

I hit almost the same issue last night.

What exact setup are you running? What model, what runtime, and roughly what context size?

If you post your exact settings, people here can probably narrow it down pretty fast.

[vramport]

I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmpath]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[papermixer4]

I'd start with the simpler stack.

[localpatch]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[kernelrunner]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing

Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.