good hosted AI app that still respects privacy for Mistral

[mangopath4]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

[cudahouse]

This keeps coming up in conversation so I figured I'd start a thread.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[macrolab]

What exact setup are you running? Are you testing this on B200 or something else?

If you post your exact settings, people here can probably narrow it down pretty fast.

[sparrowcraft]

I hit almost the same issue last night.

[paperrunner]

I hit almost the same issue last night.

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[bytelab]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[deepseeked]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[amberplane]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

I don't think that's the whole story. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.

[macronest]

Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sparrowshed]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?