when video is still not worth the wait

[dockbrook21]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and PyTorch for Qwen. The interesting difference for me has been batch size.

Interested in tradeoffs more than fan loyalty.

[mellowstream]

Spent some time on this today and wanted to compare notes.

That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turboscope5]

Maybe, but I had the opposite result. I've seen people blame continuous batching when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[benchplease]

Maybe, but I had the opposite result. I've seen people blame continuous batching when the real issue was much more mundane. Could be a workload differ

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me.

[nanopatch]

worth checking the driver first.

[cacheworks]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[kernelhouse70]

Maybe, but I had the opposite result. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[quantnest]

Maybe, but I had the opposite result. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, jus

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.