SBC inference experiments: early impressions

[vramghost]

What exact setup are you running? What model, what runtime, and roughly what context size?

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.