09-23-2026, 12:56 PM
Did a bit of testing and thought I'd post the result here.
I tried this with DeepSeek using Ollama on 4090. The headline result was better than I expected, but INT4 still ended up being the limiting factor.
Happy to post more details if anyone wants them.
This is mostly for local work rather than public serving.
I tried this with DeepSeek using Ollama on 4090. The headline result was better than I expected, but INT4 still ended up being the limiting factor.
Happy to post more details if anyone wants them.
This is mostly for local work rather than public serving.
Mostly CUDA, sometimes wrong, usually measured