student models for local use

[dockember]

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Happy to post more details if anyone wants them.

[rocmmixer]

that fixed it for me too.

[sysstation69]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[plainworks]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[quiettoken]

I hit almost the same issue last night.

[cobaltlab]

worth checking the driver first.

[llmtrack26]

Ran a quick check on my box. On dual 3090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[emberpeak]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[thetagrid15]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

I hit almost the same issue last night.

[emberpeak]

subscribed, curious where this lands.