10-07-2026, 05:00 AM
I'm trying to sanity-check my setup.
I'm mostly working with Mistral in TensorRT-LLM on Threadripper box. The part I'm trying to understand is KV cache pressure.
If you've tested something similar, what did you keep and what did you stop bothering with?
I'm mostly working with Mistral in TensorRT-LLM on Threadripper box. The part I'm trying to understand is KV cache pressure.
If you've tested something similar, what did you keep and what did you stop bothering with?
small models can still be good