when a plain Python stack is enough

[routerunner]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and TensorRT-LLM for Gemma. The interesting difference for me has been batch size.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.

[modrunner27]

Appreciate the detail. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[localpatch78]

worth checking the driver first.

[labpath]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[mangofield33]

What exact setup are you running? Are you testing this on 7900 XTX or something else?

[mintstream9]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[pixelcrate82]

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. I got better results once I stopped chasing

Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sparrowworks88]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.

[visionyard23]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

If you post your exact settings, people here can probably narrow it down pretty fast.

[cobalttinker]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.