dedicated AI gadgets you actually use

[flintriver50]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

[flintriver50]

Posting this in case it saves someone else a little time.

Need a little more detail here. Is this local-only or are you routing anything externally?

[audiorunner74]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

I'm happy to test one more variant if you want a second data point.

[quantghost]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[latentscope35]

Tried to reproduce it just now. On 4090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

[flintriver50]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[turbosupply83]

Ran a quick check on my box. On mini PC with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[routecrate]

I think the bottleneck is somewhere else. I've seen people blame MoE serving when the real issue was much more mundane.

[mintquill]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Not glamorous advice, but repeatable baselines save a lot of time.

[audiopoint18]

Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.