Forum structure feedback thread on 4090

[sparrowworks88]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and M3 Max felt surprisingly usable once I tuned around continuous batching.

If someone wants the exact settings I can clean them up and post them.

[audionode8]

subscribed, curious where this lands.

[sparrowhall]

I tested this tonight. On A6000 with TensorRT-LLM, I saw basically the same behavior.

[tinycontext]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[sanevault]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[rocmrunner40]

[quote]Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure sho

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[northcraft90]

that fixed it for me too.

[rocmplane]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[relayforge]

Good point. That matches what I saw on dual 3090.

[cudapoint]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.