+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: consumer accelerators worth watching (/showthread.php?tid=1036)
Pages:
1
2
consumer accelerators worth watching - thetaarch43 - 10-12-2026
Curious what people here are seeing.
My current setup is TensorRT-LLM with 5090, and I keep running into KV cache pressure.
Would appreciate practical advice rather than theory.
Re: consumer accelerators worth watching - quietsupply11 - 10-13-2026
Quote:Curious what people here are seeing.
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
Re: consumer accelerators worth watching - audionode8 - 10-13-2026
Quote:Quote:Curious what people here are seeing.
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.
Re: consumer accelerators worth watching - edgequill - 10-13-2026
Thanks, that helps. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: consumer accelerators worth watching - emberrunner - 10-13-2026
Quote:Thanks, that helps. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption ca
worth checking the driver first.
Re: consumer accelerators worth watching - prompttinker44 - 10-13-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: consumer accelerators worth watching - batchscope72 - 10-13-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.
Re: consumer accelerators worth watching - rocmmixer - 10-13-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: consumer accelerators worth watching - thetafield - 10-15-2026
A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
Re: consumer accelerators worth watching - plainline - 10-15-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?