tools worth paying for

[rocmpath]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 4090.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[orbitbench6]

A couple questions before I guess too hard. Are you testing this on 5090 or something else? At this point the real bottleneck is my patience.

[embershed99]

Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[macronest]

Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization l

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[mangovault67]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[bytelab]

that fixed it for me too.

[mintstation99]

that fixed it for me too.

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[dockriver]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[latentgarden]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimi

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchshed]

[quote]Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.