context stuffing mistakes for Phi

[vrampilot]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and PyTorch for Kimi. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.

[fusedhall56]

Interesting, I hadn't tried that. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[bytesupply76]

Good point. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.