small win thread for DeepSeek

[ggufwire22]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: SGLang on dual 3090, keeping KV cache pressure in mind from the start.

Maybe useful as a starting point for newer folks.

[plaingarden18]

Thanks, that helps. That matches what I saw on dual 3090.

[rocmrunner40]

Good point. That matches what I saw on 4090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchstation]

Good point. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[tensorkite]

That matches what I was seeing. I ended up in about the same place using SGLang.

[cacheplane98]

I tested this tonight. On Threadripper box with vLLM, I saw basically the same behavior.

[plaingarden18]

Need a little more detail here. What model, what runtime, and roughly what context size?

[marbletrack]

What exact setup are you running? What model, what runtime, and roughly what context size?

[ggufwire22]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[marbletrack]

subscribed, curious where this lands.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps. Turns out 'just one more test' was a lie.