Tensor cores actually helping here

[sparrowkite53]

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.

[audiopath66]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[kernelmike]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[phaseport36]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.

[edgecoast]

Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetapath]

Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead o

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[cobalttinker]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[thetapath]

worth checking the driver first.

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[audioriver]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[kernelmike]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?