+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: little utility you keep installed: early impressions (/showthread.php?tid=1300)
Pages:
1
2
little utility you keep installed: early impressions - relayforge - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: PyTorch on dual 3090, keeping speculative decoding in mind from the start.
Maybe useful as a starting point for newer folks.
Re: little utility you keep installed: early impressions - embersmith - 10-15-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: little utility you keep installed: early impressions - paperrunner - 10-16-2026
Quote:Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
I hit almost the same issue last night.
Re: little utility you keep installed: early impressions - thetashed24 - 10-16-2026
My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: little utility you keep installed: early impressions - deltagrid - 10-16-2026
Quote:My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you kn
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: little utility you keep installed: early impressions - rocmriley - 10-17-2026
worth checking the driver first.
Re: little utility you keep installed: early impressions - northquill - 10-17-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.
Re: little utility you keep installed: early impressions - mlxgarden - 10-17-2026
Appreciate the detail. I ended up in about the same place using vLLM.
Re: little utility you keep installed: early impressions - rasterbench - 10-17-2026
Good point. I don't think your conclusion is far off.
Re: little utility you keep installed: early impressions - llmpeak - 10-18-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.