Posts: 34
Threads: 6
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.
Wondering if this is just where the whole space is heading now.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.
small models can still be good
Posts: 54
Threads: 7
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Quote:I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once
Appreciate the detail. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
throughput matters, but tail latency matters too
Posts: 50
Threads: 4
Joined: Sep 2026
Reputation:
0
Good point. The pattern around INT4 has been pretty similar for me.
Posts: 23
Threads: 3
Joined: Aug 2026
Reputation:
0
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
VRAM first, aesthetics later
