09-27-2026, 01:55 PM
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Mostly CUDA, sometimes wrong, usually measured
|
KV cache compression worth it
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
KV cache compression worth it - by cachelab - 09-22-2026, 12:02 AM
Re: KV cache compression worth it - by mangovault67 - 09-22-2026, 01:02 AM
Re: KV cache compression worth it - by circuitwire58 - 09-22-2026, 01:43 AM
Re: KV cache compression worth it - by turboscope5 - 09-22-2026, 02:16 AM
Re: KV cache compression worth it - by fastquill74 - 09-22-2026, 02:50 AM
Re: KV cache compression worth it - by quantshed - 09-22-2026, 03:44 AM
Re: KV cache compression worth it - by kernelmike - 09-27-2026, 01:55 PM
Re: KV cache compression worth it - by circuitwire58 - 09-23-2026, 04:19 AM
Re: KV cache compression worth it - by audiorunner74 - 09-23-2026, 05:36 AM
Re: KV cache compression worth it - by edgequill - 09-23-2026, 01:46 PM
Re: KV cache compression worth it - by llm_wren - 09-24-2026, 01:52 PM
Re: KV cache compression worth it - by vramstream - 09-28-2026, 02:35 PM
|
Users browsing this thread: 1 Guest(s)