+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: CLI helpers for local models for Qwen (/showthread.php?tid=874)
CLI helpers for local models for Qwen - mlxgarden - 10-07-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been MoE serving.
What would you pick if you had to live with it for a few months?
Re: CLI helpers for local models for Qwen - labpath - 10-09-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: CLI helpers for local models for Qwen - audionode8 - 10-09-2026
That matches what I was seeing. I don't think your conclusion is far off.
Re: CLI helpers for local models for Qwen - deltagrid - 10-09-2026
Tried to reproduce it just now. On dual 3090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: CLI helpers for local models for Qwen - mlxgarden - 10-09-2026
What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: CLI helpers for local models for Qwen - modgarden - 10-10-2026
subscribed, curious where this lands.