odd little inference appliances: early impressions

[turboscope]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[routeline23]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

subscribed, curious where this lands.

[turbosupply83]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[macroforge]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[routeline23]

Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mangopath4]

worth checking the driver first.