Inference › Optimization & Performance › Quantization v
quant choices for 70B local

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Linear Mode
quant choices for 70B local
torchshed Offline
Junior Member
**
Posts: 47
Threads: 6
Joined: Sep 2026
Reputation: 0
#12
09-25-2026, 11:13 PM
Quote:What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Onc

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Find
Reply


Messages In This Thread
quant choices for 70B local - by vrambandit - 09-16-2026, 11:51 PM
Re: quant choices for 70B local - by southtinker - 09-17-2026, 08:34 AM
Re: quant choices for 70B local - by nanobrook - 09-17-2026, 08:57 AM
Re: quant choices for 70B local - by fusedtinker - 09-17-2026, 05:32 PM
Re: quant choices for 70B local - by marbletrack - 09-18-2026, 05:34 PM
Re: quant choices for 70B local - by vrambandit - 09-18-2026, 06:44 PM
Re: quant choices for 70B local - by vrambandit - 09-23-2026, 08:00 AM
Re: quant choices for 70B local - by relaynest82 - 09-20-2026, 06:58 AM
Re: quant choices for 70B local - by southpatch - 09-24-2026, 10:33 AM
Re: quant choices for 70B local - by cudaline - 09-20-2026, 07:24 AM
Re: quant choices for 70B local - by paperrunner - 09-24-2026, 08:46 AM
Re: quant choices for 70B local - by torchshed - 09-25-2026, 11:13 PM
Re: quant choices for 70B local - by llmarch83 - 09-26-2026, 11:44 AM
Re: quant choices for 70B local - by nanobrook - 09-26-2026, 11:48 PM
Re: quant choices for 70B local - by torchshed - 09-27-2026, 02:28 AM
Re: quant choices for 70B local - by fastscope - 09-27-2026, 10:53 AM

Forum Jump:


Users browsing this thread: 1 Guest(s)