70B still worth the pain with llama.cpp

[stackpilot]

[quote][quote]I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Not

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time. At this point the real bottleneck is my patience.