70B still worth the pain: what are you seeing?

[fusedriver]

I'm trying to sanity-check my setup.

I'm mostly working with Mistral in TensorRT-LLM on Threadripper box. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

[mlxgarden]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[quietstack]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.