flux pipeline local tips on B200

[cudaworks]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[routearch]

This keeps coming up in conversation so I figured I'd start a thread.

Ran a quick check on my box. On 7900 XTX with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[southspark]

That matches what I was seeing. The pattern around continuous batching has been pretty similar for me.

[rocmplane]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[llmarch83]

Appreciate the detail. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[promptpoint]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[visionforge]

I tested this tonight. On M3 Max with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[cacheplane98]

What exact setup are you running? Are you testing this on Threadripper box or something else?

[llmtrack26]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[macropoint]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.