student models for local use

[cacheline]

subscribed, curious where this lands.

[llmtrack26]

I tested this tonight. On mini PC with SGLang, I saw basically the same behavior. I expected the model choice to matter more than it did.

[rocmmixer]

I hit almost the same issue last night.

[amberpatch25]

What exact setup are you running? Are you testing this on dual 3090 or something else?

[quantshed]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[amberpatch25]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.