context stuffing mistakes: what are you seeing?

[flintriver50]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate reranking until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

[torchstation]

This keeps coming up in conversation so I figured I'd start a thread.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[llmarch83]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[tensorriver]

that fixed it for me too.

[ggufvault]

worth checking the driver first.

[torchkite]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vrampilot]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time. At this point the real bottleneck is my patience.

[tensorriver]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[circuitkite]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[tinycontext]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?