+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense with PyTorch (/showthread.php?tid=1358)
Pages:
1
2
Re: do small edge boxes make sense with PyTorch - cobaltworks25 - 10-19-2026
Appreciate the detail. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: do small edge boxes make sense with PyTorch - torchsmith25 - 10-19-2026
Quote:A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. At this point the real bot
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: do small edge boxes make sense with PyTorch - phasespark - 10-19-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: do small edge boxes make sense with PyTorch - thetashed24 - 10-19-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: do small edge boxes make sense with PyTorch - mangowire - 10-19-2026
That matches what I was seeing. I don't think your conclusion is far off.
Re: do small edge boxes make sense with PyTorch - kernelmike - 10-19-2026
Quote:A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.