https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:48 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1473 Sun, 18 Oct 2026 20:16:00 +0000 rocmmixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1473 https://www.datasculpting.ai/inference/showthread.php?tid=1344 Sat, 17 Oct 2026 03:25:30 +0000 cobaltworks25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1344
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around continuous batching.

Happy to post more details if anyone wants them.]]>
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around continuous batching.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1336 Fri, 16 Oct 2026 22:20:32 +0000 latentgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1336
Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]>
Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1375 Fri, 16 Oct 2026 07:56:29 +0000 mintforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1375
My current setup is TensorRT-LLM with B200, and I keep running into prefix caching.

Would appreciate practical advice rather than theory.]]>
My current setup is TensorRT-LLM with B200, and I keep running into prefix caching.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1312 Fri, 16 Oct 2026 05:58:22 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1312
Just sharing a setup that ended up working reasonably well: Phi, LM Studio, 5090, with a lot of attention on long context.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Phi, LM Studio, 5090, with a lot of attention on long context.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1218 Thu, 15 Oct 2026 05:23:18 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1218
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

Simple and boring is a valid answer.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1174 Tue, 13 Oct 2026 16:29:48 +0000 torchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1174
My current setup is TensorRT-LLM with dual 3090, and I keep running into router policy.

Not looking for the perfect answer, mostly what worked in real use.]]>
My current setup is TensorRT-LLM with dual 3090, and I keep running into router policy.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1150 Tue, 13 Oct 2026 13:02:54 +0000 amberhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1150
One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1112 Tue, 13 Oct 2026 03:31:19 +0000 thetaember38]]> https://www.datasculpting.ai/inference/showthread.php?tid=1112
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1077 Sun, 11 Oct 2026 15:06:37 +0000 rocmmixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1077
I'm mostly working with Mistral in MLX on mini PC. The part I'm trying to understand is tokens/sec.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm mostly working with Mistral in MLX on mini PC. The part I'm trying to understand is tokens/sec.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1081 Sun, 11 Oct 2026 10:54:11 +0000 rocmriley]]> https://www.datasculpting.ai/inference/showthread.php?tid=1081
I've gone back and forth between PyTorch and SGLang for Phi. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I've gone back and forth between PyTorch and SGLang for Phi. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1007 Sat, 10 Oct 2026 22:54:39 +0000 southden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1007
I'm mostly working with DeepSeek in vLLM on B200. The part I'm trying to understand is MoE serving.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm mostly working with DeepSeek in vLLM on B200. The part I'm trying to understand is MoE serving.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=947 Sat, 10 Oct 2026 03:36:09 +0000 southtinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=947
The more I work with local setups, the more it seems like quant drift decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]>
The more I work with local setups, the more it seems like quant drift decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=905 Thu, 08 Oct 2026 19:28:35 +0000 thetaember38]]> https://www.datasculpting.ai/inference/showthread.php?tid=905
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tokens/sec.

Happy to post more details if anyone wants them.]]>
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tokens/sec.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=784 Tue, 06 Oct 2026 03:31:12 +0000 orbitgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=784
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Wondering if this is just where the whole space is heading now.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Wondering if this is just where the whole space is heading now.]]>