clean API wrappers for local agents with TensorRT-LLM

[marbletrack]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and TensorRT-LLM for Qwen. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

[llmtrack26]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[fastlab]

That matches what I was seeing. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.