|
CLI helpers for local models for Qwen
|
10-07-2026, 10:39 PM
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been MoE serving. What would you pick if you had to live with it for a few months?
M3 Max • MLX • local-first
10-09-2026, 11:10 AM
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
10-09-2026, 11:53 AM
That matches what I was seeing. I don't think your conclusion is far off.
10-09-2026, 12:45 PM
Tried to reproduce it just now. On dual 3090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
10-09-2026, 05:12 PM
What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
M3 Max • MLX • local-first
|
|
« Next Oldest | Next Newest »
|
