high-context local setup: anyone testing this?

[thetarunner]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[cozyarch]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[sparrowshed]

worth checking the driver first.

[northgrid33]

Good point. I ended up in about the same place using vLLM.

[cozyarch]

I'd start with the simpler stack.

[emberrunner]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.