+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: AMD / ROCm (https://www.datasculpting.ai/inference/forumdisplay.php?fid=32)
+--- Thread: Instinct pricing thoughts with TensorRT-LLM (/showthread.php?tid=505)
Pages:
1
2
Instinct pricing thoughts with TensorRT-LLM - fastquill74 - 09-26-2026
Has anyone else run into this?
I'm mostly working with Kimi in Ollama on 4090. The part I'm trying to understand is speculative decoding.
If you've tested something similar, what did you keep and what did you stop bothering with?
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: Instinct pricing thoughts with TensorRT-LLM - signalpeak - 09-28-2026
Quote:Has anyone else run into this?
I hit almost the same issue last night.
Re: Instinct pricing thoughts with TensorRT-LLM - northhall88 - 10-01-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Re: Instinct pricing thoughts with TensorRT-LLM - tensorworks - 10-02-2026
What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Instinct pricing thoughts with TensorRT-LLM - macropoint - 10-02-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.
Re: Instinct pricing thoughts with TensorRT-LLM - northhall88 - 10-05-2026
I'd start with the simpler stack.
Re: Instinct pricing thoughts with TensorRT-LLM - routeline23 - 10-05-2026
Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Instinct pricing thoughts with TensorRT-LLM - phasetrack - 10-05-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: Instinct pricing thoughts with TensorRT-LLM - macropoint - 10-06-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: Instinct pricing thoughts with TensorRT-LLM - rocmriley - 10-07-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.