good hosted AI app that still respects privacy for Mistral

[dockworks23]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, MLX, B200, with a lot of attention on token cost.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. Turns out 'just one more test' was a lie.

[tensorworks]

Thought I'd post a small recipe because a lot of threads ask similar things.

Thanks, that helps. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cozyarch]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[deepseeked]

Good point. The pattern around reranking has been pretty similar for me.

[vramarch]

worth checking the driver first.

[latentgarden]

Need a little more detail here. Is this local-only or are you routing anything externally?

[southpatch]

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[dockworks23]

subscribed, curious where this lands.

[thetarunner]

That matches what I was seeing. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelpeak]

subscribed, curious where this lands.