few-shot when you actually need it: setup advice

[mintstation99]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using MLX on 4090. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

[northgarden37]

Did a bit of testing and thought I'd post the result here.

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[mlxgarden]

same here on my side.

[audiopoint18]

Appreciate the detail. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. At this point the real bottleneck is my patience.

[quiettoken]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mlxgarden]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[northgarden37]

following this because I'm in the same boat.

[mlxgarden]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[marbleyard32]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[quiettoken]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.