simple router policy ideas

[kernelgrid]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like INT4 decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

[sparrowcraft]

A couple questions before I guess too hard. Are you testing this on mini PC or something else?

[quantghost]

That matches what I was seeing. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

I'm happy to test one more variant if you want a second data point.

[cacheline]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Once you know the plain path works, then add the faster path back in piece by piece.

[modgarden]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[torchkite]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[oxidestream]

that fixed it for me too.

[fusedhall56]

that fixed it for me too.

What exact setup are you running? Is this local-only or are you routing anything externally?

[kernelrunner]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

If you post your exact settings, people here can probably narrow it down pretty fast.

[embershed99]

Need a little more detail here. What model, what runtime, and roughly what context size?