prompt templates worth keeping for Llama

[quiettoken]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.

[sysstation69]

Maybe, but I had the opposite result. I've seen people blame tokens/sec when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[thetafield]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.