Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

Did a bit of testing and thought I'd post the result here.

I tried this with FLUX using PyTorch on 7900 XTX. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and TensorRT-LLM for Mistral. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring MoE serving.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and llama.cpp for Kimi. The interesting difference for me has been embeddings.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

Simple and boring is a valid answer.