https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1476 Sun, 18 Oct 2026 15:26:25 +0000 routerunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1476
Short version: PyTorch behaved well for me, but only after I stopped ignoring MoE serving.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Short version: PyTorch behaved well for me, but only after I stopped ignoring MoE serving.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1488 Sun, 18 Oct 2026 06:24:28 +0000 northgrid33]]> https://www.datasculpting.ai/inference/showthread.php?tid=1488
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1409 Sat, 17 Oct 2026 19:11:53 +0000 sparrowbrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=1409
Short version: SGLang behaved well for me, but only after I stopped ignoring prefix caching.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Short version: SGLang behaved well for me, but only after I stopped ignoring prefix caching.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1340 Fri, 16 Oct 2026 20:55:52 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1340
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1110 Mon, 12 Oct 2026 19:42:23 +0000 quietcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1110
My current setup is LM Studio with GB200, and I keep running into token cost.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
My current setup is LM Studio with GB200, and I keep running into token cost.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1059 Mon, 12 Oct 2026 03:16:56 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1059
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and A6000.

Simple and boring is a valid answer.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and A6000.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=918 Thu, 08 Oct 2026 23:58:31 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=918
I've gone back and forth between llama.cpp and llama.cpp for Gemma. The interesting difference for me has been structured output.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I've gone back and forth between llama.cpp and llama.cpp for Gemma. The interesting difference for me has been structured output.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=850 Thu, 08 Oct 2026 02:25:08 +0000 cacheline]]> https://www.datasculpting.ai/inference/showthread.php?tid=850
Two paths both seem viable, but they break down differently once continuous batching becomes important.

Interested in tradeoffs more than fan loyalty.]]>
Two paths both seem viable, but they break down differently once continuous batching becomes important.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=853 Wed, 07 Oct 2026 17:44:42 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=853
I tried this with Kimi using llama.cpp on 7900 XTX. The headline result was better than I expected, but prefix caching still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I tried this with Kimi using llama.cpp on 7900 XTX. The headline result was better than I expected, but prefix caching still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=807 Tue, 06 Oct 2026 21:10:26 +0000 sparrowworks88]]> https://www.datasculpting.ai/inference/showthread.php?tid=807
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and 5090.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and 5090.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=806 Tue, 06 Oct 2026 18:00:32 +0000 deltastation]]> https://www.datasculpting.ai/inference/showthread.php?tid=806
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=837 Tue, 06 Oct 2026 11:41:38 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=837
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=742 Mon, 05 Oct 2026 02:19:46 +0000 kernelpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=742
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=701 Sat, 03 Oct 2026 16:04:14 +0000 cacheline]]> https://www.datasculpting.ai/inference/showthread.php?tid=701
I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]>
I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=493 Sat, 26 Sep 2026 00:12:40 +0000 deltastation]]> https://www.datasculpting.ai/inference/showthread.php?tid=493
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling small local stacks.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling small local stacks.

Would appreciate practical advice rather than theory.]]>