good hosted AI app that still respects privacy on A6000

[torchsmith25]

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using LM Studio on A6000. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.

[deepseeked]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[audiodesk]

Thanks, that helps. The pattern around structured output has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[mintstation99]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[emberpeak]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box. I keep promising myself this is the last rebuild.

[cachebench]

Can you fill in a bit more context? Are you testing this on 5090 or something else?

[latentgarden]

Can you fill in a bit more context? Are you testing this on 5090 or something else?

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[batchshed]

That matches what I was seeing. That matches what I saw on 5090.

[quantghost]

Good point. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[amberplane]

Good point. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.