memory layers vs plain long context: setup advice

[emberquill]

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[edgequill]

Appreciate the detail. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchkite]

Appreciate the detail. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can ma

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.