where open models feel surprisingly good with TensorRT-LLM

[localtinker]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like long context decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[vrampilot]

This keeps coming up in conversation so I figured I'd start a thread.

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmriley]

This keeps coming up in conversation so I figured I'd start a thread.

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.