prompt injection in local agent setups

[pixelcrate82]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, Ollama, 7900 XTX, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vectorhouse71]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[cudacrate]

Maybe, but I had the opposite result. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[routearch]

I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[deepseeked]

Appreciate the detail. I ended up in about the same place using MLX.

[mlxgarden]

same here on my side.

[cobaltspark]

that fixed it for me too.

I'm happy to test one more variant if you want a second data point.

[thetanest]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. At this point the real bottleneck is my patience.

[kernelrunner]

I don't think that's the whole story. I've seen people blame reranking when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[batchsize]

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.