Instinct pricing thoughts with TensorRT-LLM

[fastquill74]

Has anyone else run into this?

I'm mostly working with Kimi in Ollama on 4090. The part I'm trying to understand is speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[signalpeak]

Has anyone else run into this?

I hit almost the same issue last night.

[northhall88]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.

[tensorworks]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[macropoint]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

[northhall88]

I'd start with the simpler stack.

[routeline23]

Appreciate the detail. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[phasetrack]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[macropoint]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmriley]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.