Inference › Optimization & Performance › Quantization v
quant choices for 70B local with PyTorch

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Linear Mode
quant choices for 70B local with PyTorch
benchplease Offline
Member
***
Posts: 123
Threads: 18
Joined: Sep 2026
Reputation: 0
#1
10-17-2026, 08:12 AM
Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and PyTorch for Qwen. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.
post logs or it didn't happen
Find
Reply


Messages In This Thread
quant choices for 70B local with PyTorch - by benchplease - 10-17-2026, 08:12 AM

Forum Jump:


Users browsing this thread: 1 Guest(s)