+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Routers (https://www.datasculpting.ai/inference/forumdisplay.php?fid=45)
+--- Thread: synthetic data for support bots: early impressions (/showthread.php?tid=101)
synthetic data for support bots: early impressions - routecraft - 09-02-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between vLLM and LM Studio for Gemma. The interesting difference for me has been router policy.
Interested in tradeoffs more than fan loyalty.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: synthetic data for support bots: early impressions - quantsupply - 09-02-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: synthetic data for support bots: early impressions - routecraft - 09-02-2026
That matches what I was seeing. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: synthetic data for support bots: early impressions - batchember - 09-03-2026
Good point. The pattern around token cost has been pretty similar for me.
Re: synthetic data for support bots: early impressions - macrostack69 - 09-06-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: synthetic data for support bots: early impressions - rocmplane - 09-06-2026
I hit almost the same issue last night.
Re: synthetic data for support bots: early impressions - phaseport36 - 09-06-2026
Quote:I hit almost the same issue last night.
My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: synthetic data for support bots: early impressions - prompttinker44 - 09-06-2026
Quote:Quote:I hit almost the same issue last night.
What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: synthetic data for support bots: early impressions - routecraft - 09-06-2026
Quote:Quote:Quote:I hit almost the same issue last night.
Maybe, but I had the opposite result. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: synthetic data for support bots: early impressions - phaseport36 - 09-08-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Not glamorous advice, but repeatable baselines save a lot of time.
If you post your exact settings, people here can probably narrow it down pretty fast.