Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, llama.cpp, Threadripper box, with a lot of attention on router policy.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate small local stacks until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and PyTorch for Qwen. The interesting difference for me has been KV cache pressure.

What would you pick if you had to live with it for a few months?

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, A6000, with a lot of attention on tool calling.

Maybe useful as a starting point for newer folks.

Wanted to compare notes with people doing similar things.

I'm mostly working with FLUX in vLLM on A6000. The part I'm trying to understand is reranking.

Would appreciate practical advice rather than theory.

Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring speculative decoding.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: FLUX, Ollama, 7900 XTX, with a lot of attention on tool calling.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.