long context that stayed useful for Llama

[vectorpoint]

Interesting, I hadn't tried that. That matches what I saw on dual 3090.

[plainember52]

Did a small comparison this afternoon. On 4090 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[tinycontext]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[llm_wren]

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[sanescope]

Can you fill in a bit more context? Are you testing this on B200 or something else?