+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: when a plain Python stack is enough (/showthread.php?tid=996)
Pages:
1
2
when a plain Python stack is enough - routerunner - 10-11-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between vLLM and TensorRT-LLM for Gemma. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.
Re: when a plain Python stack is enough - modrunner27 - 10-11-2026
Appreciate the detail. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: when a plain Python stack is enough - localpatch78 - 10-12-2026
worth checking the driver first.
Re: when a plain Python stack is enough - labpath - 10-15-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off.
Re: when a plain Python stack is enough - mangofield33 - 10-16-2026
What exact setup are you running? Are you testing this on 7900 XTX or something else?
Re: when a plain Python stack is enough - mintstream9 - 10-18-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: when a plain Python stack is enough - pixelcrate82 - 10-18-2026
Quote:Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. I got better results once I stopped chasing
Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: when a plain Python stack is enough - sparrowworks88 - 10-19-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.
Re: when a plain Python stack is enough - visionyard23 - 10-19-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: when a plain Python stack is enough - cobalttinker - 10-19-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.