what are you all running this week on dual 3090

[rocmrunner40]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

That matches what I was seeing. The pattern around prefix caching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vectorshed43]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[plaingarden18]

That matches what I was seeing. I don't think your conclusion is far off.

[tensorkite]

That matches what I was seeing. I don't think your conclusion is far off.

Appreciate the detail. I ended up in about the same place using SGLang.

[thetagrid15]

Appreciate the detail. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[quiettoken]

Good point. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.