consumer accelerators worth watching

[thetaarch43]

Curious what people here are seeing.

My current setup is TensorRT-LLM with 5090, and I keep running into KV cache pressure.

Would appreciate practical advice rather than theory.

[quietsupply11]

Curious what people here are seeing.

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.

[audionode8]

Curious what people here are seeing.

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

[edgequill]

Thanks, that helps. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[emberrunner]

Thanks, that helps. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption ca

worth checking the driver first.

[prompttinker44]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[batchscope72]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[rocmmixer]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[thetafield]

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

[plainline]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?