system prompt habits that helped on M3 Max

[quantyard]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[cozywire]

Appreciate the detail. The pattern around prefix caching has been pretty similar for me.

[promptspark58]

following this because I'm in the same boat.

[quiettoken]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetafield]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetarunner]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.