Posts: 18
Threads: 3
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tokens/sec.
Curious whether that lines up with what other people are seeing.
trying not to rebuild again
Posts: 72
Threads: 7
Joined: Sep 2026
Reputation:
0
subscribed, curious where this lands.
Posts: 61
Threads: 6
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Posts: 33
Threads: 5
Joined: Aug 2026
Reputation:
0
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. On A6000 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
small models can still be useful
Posts: 57
Threads: 8
Joined: Oct 2026
Reputation:
0
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Posts: 142
Threads: 20
Joined: Sep 2026
Reputation:
0
Thanks, that helps. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
small models can still be useful
Posts: 93
Threads: 10
Joined: Oct 2026
Reputation:
0
worth checking the driver first.
