+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=1465)
tokens/sec tuning ideas - kernelmike - 10-18-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like structured output decides everything downstream.
Wondering if this is just where the whole space is heading now.
Re: tokens/sec tuning ideas - northgarden37 - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas - sysstation69 - 10-18-2026
Appreciate the detail. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: tokens/sec tuning ideas - northquill - 10-18-2026
I tested this tonight. On Threadripper box with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.
Re: tokens/sec tuning ideas - cacheworks - 10-19-2026
Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: tokens/sec tuning ideas - rocmplane - 10-19-2026
Quote:Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock th
I'd start with the simpler stack.
Re: tokens/sec tuning ideas - cachebench - 10-19-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: tokens/sec tuning ideas - amberpatch25 - 10-19-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas - vramstream - 10-19-2026
Quote:I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lo
same here on my side.