Best workstation layout for local models

[stackpilot]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[labnest]

Appreciate the detail. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[torchkite]

I hit almost the same issue last night.

I'm happy to test one more variant if you want a second data point.

[homelabhenry]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[bytegarden73]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[rasterpoint]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. O

I hit almost the same issue last night.

[batchwire]

[quote]I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obv

Thanks, that helps. The pattern around reranking has been pretty similar for me.

[sanevault]

My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[batchwire]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.