Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
prefix caching on repeated workflows with TensorRT-LLM
#7
Quote:Interesting, I hadn't tried that. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack

That matches what I was seeing. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
measure twice, benchmark three times
Reply


Messages In This Thread
Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-10-2026, 12:10 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)