routing between local and hosted: what are you seeing?

[oxidegrove40]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 7900 XTX.

I'm especially interested in mistakes worth avoiding. The logs were somehow calmer than I was.

[deepseeked]

I tested this tonight. On mini PC with TensorRT-LLM, I saw basically the same behavior.

[tensorworks]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[visionforge]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mintstation99]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[signalcraft14]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.

[batchsize]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once

I hit almost the same issue last night.

I'm happy to test one more variant if you want a second data point.

[mangoquill]

same here on my side.

[localpatch]

What exact setup are you running? Are you testing this on mini PC or something else?