Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and TensorRT-LLM for Qwen. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring INT4.

Happy to post more details if anyone wants them.

I care more about stability than squeezing the last possible tokens/sec out of it.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and vLLM for DeepSeek. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate embeddings until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

I'm trying to sanity-check my setup.

My current setup is Ollama with A6000, and I keep running into token cost.

If you've tested something similar, what did you keep and what did you stop bothering with?

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

If you've tested something similar, what did you keep and what did you stop bothering with?

Curious what people here are seeing.

I'm mostly working with Phi in PyTorch on mini PC. The part I'm trying to understand is HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Curious whether that lines up with what other people are seeing.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, Ollama, 5090, with a lot of attention on INT4.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.