+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: Tensor cores actually helping here (/showthread.php?tid=1266)
Pages:
1
2
Tensor cores actually helping here - sparrowkite53 - 10-16-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.
Curious if anyone else landed in the same place.
Re: Tensor cores actually helping here - audiopath66 - 10-16-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: Tensor cores actually helping here - kernelmike - 10-16-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: Tensor cores actually helping here - phaseport36 - 10-16-2026
Quote:Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.
Re: Tensor cores actually helping here - edgecoast - 10-16-2026
Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Tensor cores actually helping here - thetapath - 10-17-2026
Quote:Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead o
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: Tensor cores actually helping here - cobalttinker - 10-18-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: Tensor cores actually helping here - thetapath - 10-18-2026
Quote:worth checking the driver first.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: Tensor cores actually helping here - audioriver - 10-18-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: Tensor cores actually helping here - kernelmike - 10-18-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?