dedicated AI gadgets you actually use

[audiorunner74]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[plainline]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[rasterbench]

Thanks, that helps. That matches what I saw on mini PC.

[latentscope35]

Interesting, I hadn't tried that. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[bytepilot]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[turboscope]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[rasterbench]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.

[routeforge64]

Need a little more detail here. Are you testing this on 4090 or something else?

[routeline23]

That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rasterbench]

following this because I'm in the same boat.