prefix caching on repeated workflows with TensorRT-LLM

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Linear Mode

prefix caching on repeated workflows with TensorRT-LLM

cacheline Offline

Senior Member

Posts: 341
Threads: 51
Joined: Sep 2026
Reputation: 0

#12

10-15-2026, 03:24 PM

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

measure twice, benchmark three times

Find

« Next Oldest | Next Newest »

Messages In This Thread

prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-08-2026, 11:40 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-09-2026, 12:29 AM

Re: prefix caching on repeated workflows with TensorRT-LLM - by quiethouse - 10-09-2026, 09:05 AM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-09-2026, 04:29 AM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-10-2026, 09:10 AM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-10-2026, 12:10 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-14-2026, 03:20 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by thetafield - 10-10-2026, 11:57 AM

Re: prefix caching on repeated workflows with TensorRT-LLM - by homelabhenry - 10-10-2026, 02:01 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by emberquill - 10-11-2026, 02:19 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by mangohouse - 10-11-2026, 02:53 PM

Re: prefix caching on repeated workflows with TensorRT-LLM - by cacheline - 10-15-2026, 03:24 PM

View a Printable Version

Forum Jump:

Users browsing this thread: 1 Guest(s)

Login
Username:
Password:	Lost Password?
	Remember me

Sign in with X.com