+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: new here, what's your stack (/showthread.php?tid=700)
new here, what's your stack - edgekite - 10-03-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tokens/sec.
Curious whether that lines up with what other people are seeing.
Re: new here, what's your stack - ggufwire22 - 10-03-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
worth checking the driver first.
Re: new here, what's your stack - vectorshed43 - 10-03-2026
subscribed, curious where this lands.
Re: new here, what's your stack - marbletrack - 10-05-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: new here, what's your stack - syshouse - 10-05-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: new here, what's your stack - quiettoken - 10-05-2026
Did a small comparison this afternoon. On A6000 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: new here, what's your stack - rocmrunner40 - 10-05-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: new here, what's your stack - quiettoken - 10-05-2026
Thanks, that helps. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: new here, what's your stack - ggufwire22 - 10-05-2026
worth checking the driver first.