+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 70B still worth the pain: what are you seeing? (/showthread.php?tid=842)
70B still worth the pain: what are you seeing? - fusedriver - 10-07-2026
I'm trying to sanity-check my setup.
I'm mostly working with Mistral in TensorRT-LLM on Threadripper box. The part I'm trying to understand is KV cache pressure.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: 70B still worth the pain: what are you seeing? - mlxgarden - 10-07-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: 70B still worth the pain: what are you seeing? - quietstack - 10-07-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.