Threadripper build for inference on A6000

[vectorember]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

Would rather learn from your scars than collect my own all at once.

[labnest]

I hit almost the same issue last night.

[stackarch]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[tinycontext]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[Data Sculpting]

What exact setup are you running? What model, what runtime, and roughly what context size?

[sanevault]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[latentpeak22]

worth checking the driver first.

[tensorworks]

following this because I'm in the same boat.

[homelabhenry]

I hit almost the same issue last night.

[kernelghost]

I hit almost the same issue last night.

I hit almost the same issue last night.