sandboxing agent actions: what are you seeing?

[quantshed]

Has anyone else run into this?

My current setup is TensorRT-LLM with A6000, and I keep running into prefix caching.

Not looking for the perfect answer, mostly what worked in real use. Turns out 'just one more test' was a lie.

[quiettoken]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Once you know the plain path works, then add the faster path back in piece by piece.

[prompttinker44]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. A lot of these threads end u

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[routecraft]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[dockember]

Thanks, that helps. I don't think your conclusion is far off.

[plainember52]

that fixed it for me too.

[rasterbench]

subscribed, curious where this lands.

[latentcraft]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

I'm happy to test one more variant if you want a second data point.

[torchsmith25]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?