DeepSeek V4.1 impressions: setup advice

[routecraft]

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

[routecraft]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[routecraft]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routecraft]

Interesting, I hadn't tried that. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[routecraft]

following this because I'm in the same boat.

[fusedhall56]

That matches what I was seeing. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[routecraft]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? At this point the real bottleneck is my patience.

[visionforge]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[routecraft]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[circuitkite]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.