good hosted AI app that still respects privacy on 4090

[macrolab]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once reranking becomes important.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

[batchshed]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

[torchsmith25]

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[amberplane]

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

What worked for me was pretty simple. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[batchshed]

[quote]My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.[/q

I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were somehow calmer than I was.

[macronest]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[embershed99]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

[vramarch]

Tried to reproduce it just now. On A6000 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.

[phaseshed]

following this because I'm in the same boat.

[llmpeak]

I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane.