when video is still not worth the wait

[bytehall]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and Threadripper box.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

[plainember52]

What worked for me was pretty simple. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[fusedriver]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[phaseport36]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[sparrowshed]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

[labline]

That matches what I was seeing. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[audiodesk]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.