+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: framework overhead in real deployments (/showthread.php?tid=1495)
framework overhead in real deployments - vrampilot - 10-19-2026
I figured I'd ask here before changing everything.
My current setup is llama.cpp with 7900 XTX, and I keep running into speculative decoding.
Would appreciate practical advice rather than theory.
This is mostly for local work rather than public serving.