routing between local and hosted

[sanevault]

I'm trying to sanity-check my setup.

I'm mostly working with Phi in PyTorch on 4090. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

[tinycontext]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[kernelghost]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.