small agent harness that stayed useful for Mixtral

[thetanest]

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using vLLM on GB200. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Happy to post more details if anyone wants them.

[ggufvault]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llm_wren]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot

What exact setup are you running? What model, what runtime, and roughly what context size?

[northgrid33]

worth checking the driver first.

[llm_wren]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.

[cacheline]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.