|
distill or just quantize
|
10-06-2026, 03:36 PM
Quote:Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane. A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.
small models can still be good
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
distill or just quantize - by quiettoken - 09-28-2026, 11:50 PM
Re: distill or just quantize - by quantshed - 09-30-2026, 12:00 AM
Re: distill or just quantize - by cozykite - 09-30-2026, 12:28 AM
Re: distill or just quantize - by oxidegrove40 - 09-30-2026, 01:08 AM
Re: distill or just quantize - by quietstack - 10-03-2026, 01:54 AM
Re: distill or just quantize - by quantshed - 10-03-2026, 02:37 AM
Re: distill or just quantize - by torchtrack24 - 10-03-2026, 02:53 AM
Re: distill or just quantize - by quantshed - 10-08-2026, 04:23 AM
Re: distill or just quantize - by batchscope72 - 10-06-2026, 03:17 AM
Re: distill or just quantize - by fusedriver - 10-06-2026, 03:36 PM
Re: distill or just quantize - by emberpeak - 10-09-2026, 05:13 PM
Re: distill or just quantize - by quantshed - 10-10-2026, 05:25 PM
Re: distill or just quantize - by quiettoken - 10-11-2026, 05:39 AM
Re: distill or just quantize - by northquill - 10-11-2026, 06:36 AM
|
