synthetic data for support bots: early impressions

[routecraft]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and LM Studio for Gemma. The interesting difference for me has been router policy.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[quantsupply]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[routecraft]

That matches what I was seeing. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchember]

Good point. The pattern around token cost has been pretty similar for me.

[macrostack69]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[rocmplane]

I hit almost the same issue last night.

[phaseport36]

I hit almost the same issue last night.

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[prompttinker44]

I hit almost the same issue last night.

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[routecraft]

I hit almost the same issue last night.

Maybe, but I had the opposite result. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[phaseport36]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.