10-04-2026, 10:11 PM
Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
throughput matters, but tail latency matters too