what made local AI click for you

[quiettoken]

Can you fill in a bit more context? Are you testing this on A6000 or something else?

[ggufwire22]

following this because I'm in the same boat.

[macroforge]

following this because I'm in the same boat.

[amberrunner]

subscribed, curious where this lands.

[tensorkite]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

[vectorshed43]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.