+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Quality / Reliability Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=68)
+--- Thread: when video is still not worth the wait (/showthread.php?tid=587)
when video is still not worth the wait - dockbrook21 - 09-30-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between SGLang and PyTorch for Qwen. The interesting difference for me has been batch size.
Interested in tradeoffs more than fan loyalty.
Re: when video is still not worth the wait - mellowstream - 09-30-2026
Quote:Spent some time on this today and wanted to compare notes.
That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: when video is still not worth the wait - turboscope5 - 09-30-2026
Maybe, but I had the opposite result. I've seen people blame continuous batching when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: when video is still not worth the wait - benchplease - 09-30-2026
Quote:Maybe, but I had the opposite result. I've seen people blame continuous batching when the real issue was much more mundane. Could be a workload differ
Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me.
Re: when video is still not worth the wait - nanopatch - 10-03-2026
worth checking the driver first.
Re: when video is still not worth the wait - cacheworks - 10-03-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: when video is still not worth the wait - kernelhouse70 - 10-03-2026
Maybe, but I had the opposite result. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: when video is still not worth the wait - quantnest - 10-05-2026
Quote:Maybe, but I had the opposite result. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, jus
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.