little utility you keep installed: early impressions

[relayforge]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on dual 3090, keeping speculative decoding in mind from the start.

Maybe useful as a starting point for newer folks.

[embersmith]

Thought I'd post a small recipe because a lot of threads ask similar things.

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[paperrunner]

Thought I'd post a small recipe because a lot of threads ask similar things.

I hit almost the same issue last night.

[thetashed24]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[deltagrid]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you kn

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[rocmriley]

worth checking the driver first.

[northquill]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.

[mlxgarden]

Appreciate the detail. I ended up in about the same place using vLLM.

[rasterbench]

Good point. I don't think your conclusion is far off.

[llmpeak]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.