adapter stacking for DeepSeek

[cacheline]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once continuous batching becomes important.

Interested in tradeoffs more than fan loyalty.

[cudascope]

worth checking the driver first.

[batchsize]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelghost]

I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. Turns out 'just one more test' was a lie.

[benchplease]

I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong,

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[batchsize]

[quote]I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cacheline]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[kernelarch]

Thanks, that helps. I ended up in about the same place using LM Studio.

[quiethouse]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchsize]

same here on my side.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.