synthetic data for support bots: anyone testing this?

[routeline23]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.

[promptpoint]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[phaseport36]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[deepseeked]

What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[plainworks]

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[phaseport36]

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

worth checking the driver first.

[audioforge]

worth checking the driver first.

[quantsupply]

worth checking the driver first.

That matches what I was seeing. That matches what I saw on M3 Max.

[forgemixer]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[plainline]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one mor

I don't think that's the whole story. I've seen people blame consumer GPUs when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.