Wanted to compare notes with people doing similar things.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been quant drift.

Interested in tradeoffs more than fan loyalty.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring structured output.

Happy to post more details if anyone wants them.

Did a bit of testing and thought I'd post the result here.

After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around embeddings.

Happy to post more details if anyone wants them.

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring reranking.

Happy to post more details if anyone wants them. At this point the real bottleneck is my patience.

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring prefill/decode split.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and TensorRT-LLM for Mistral. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring workstation thermals.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.