+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: simple router policy ideas with PyTorch (/showthread.php?tid=911)
Pages:
1
2
simple router policy ideas with PyTorch - kernelmike - 10-08-2026
Has anyone else run into this?
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.
Would appreciate practical advice rather than theory.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: simple router policy ideas with PyTorch - edgekite - 10-09-2026
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: simple router policy ideas with PyTorch - cudacrate - 10-09-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: simple router policy ideas with PyTorch - rocmcrate26 - 10-09-2026
Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: simple router policy ideas with PyTorch - cobaltspark - 10-12-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
If you post your exact settings, people here can probably narrow it down pretty fast. At this point the real bottleneck is my patience.
Re: simple router policy ideas with PyTorch - sparrowshed - 10-12-2026
I don't think that's the whole story. I've seen people blame continuous batching when the real issue was much more mundane.
Re: simple router policy ideas with PyTorch - macropilot - 10-12-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: simple router policy ideas with PyTorch - orbitgarden - 10-13-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: simple router policy ideas with PyTorch - routeline23 - 10-13-2026
That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: simple router policy ideas with PyTorch - promptspark58 - 10-13-2026
Can you fill in a bit more context? Are you testing this on dual 3090 or something else?