system prompt habits that helped: worth it?

[vramforge]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[cozywire]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[batchember]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[paperport64]

Tried to reproduce it just now. On 7900 XTX with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[cobaltworks25]

I hit almost the same issue last night.

If you post your exact settings, people here can probably narrow it down pretty fast.

[fusedhall56]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cozywire]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[mlxgarden]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[torchshed66]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[mlxgarden]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.