Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring KV cache pressure.

Happy to post more details if anyone wants them.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and A6000 felt surprisingly usable once I tuned around prefix caching.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and 5090 felt surprisingly usable once I tuned around NVFP4.

Maybe useful as a starting point for newer folks.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, TensorRT-LLM, 7900 XTX, with a lot of attention on NVFP4.

If someone wants the exact settings I can clean them up and post them.

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Happy to post more details if anyone wants them.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on A6000, keeping structured output in mind from the start.

Maybe useful as a starting point for newer folks.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with HBM constraints.

Wondering if this is just where the whole space is heading now.

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using vLLM on 7900 XTX. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.