Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CLI helpers for local models with MLX
#11
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.
learning in public
Reply
#12
Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
context windows are not a product strategy
Reply
#13
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)