prefix caching on repeated workflows with TensorRT-LLM

[cacheline]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cacheline]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.

[turbogrid11]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you k

Not sure I agree there. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere.

[cacheline]

[quote]A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Onc

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelghost]

Need a little more detail here. What model, what runtime, and roughly what context size?

[cacheline]

That matches what I was seeing. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[kernelghost]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[sparrowhall]

I hit almost the same issue last night.

[cacheline]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

[cacheline]

What exact setup are you running? Is this local-only or are you routing anything externally?