how much orchestration is too much

[thetanest]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[llm_wren]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints.

Did a small comparison this afternoon. On 7900 XTX with LM Studio, I saw basically the same behavior. The main thing that moved was latency, not quality.

[deepseeked]

That matches what I was seeing. That matches what I saw on 4090.

[oxidegrove40]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[llm_wren]

I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[labline]

That matches what I was seeing. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.