Posts: 93
Threads: 10
Joined: Oct 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tool calling.
Wondering if this is just where the whole space is heading now.
This is mostly for local work rather than public serving.
Posts: 36
Threads: 8
Joined: Aug 2026
Reputation:
0
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Posts: 72
Threads: 7
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Posts: 39
Threads: 1
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
learning in public
Posts: 55
Threads: 7
Joined: Aug 2026
Reputation:
0
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.
still benchmarking everything
Posts: 72
Threads: 7
Joined: Sep 2026
Reputation:
0
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Posts: 40
Threads: 2
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Posts: 76
Threads: 12
Joined: Aug 2026
Reputation:
0
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Posts: 40
Threads: 2
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. On 4090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.
