simple router policy ideas with PyTorch

[kernelmike]

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[edgekite]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[cudacrate]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[rocmcrate26]

Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cobaltspark]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast. At this point the real bottleneck is my patience.

[sparrowshed]

I don't think that's the whole story. I've seen people blame continuous batching when the real issue was much more mundane.

[macropilot]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[orbitgarden]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[routeline23]

That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[promptspark58]

Can you fill in a bit more context? Are you testing this on dual 3090 or something else?