https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:30 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1245 Thu, 15 Oct 2026 04:52:39 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1245
This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around long context.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around long context.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1061 Mon, 12 Oct 2026 02:26:40 +0000 mangohouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=1061
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1042 Sun, 11 Oct 2026 23:55:10 +0000 modrunner27]]> https://www.datasculpting.ai/inference/showthread.php?tid=1042
Just sharing a setup that ended up working reasonably well: DeepSeek, vLLM, A6000, with a lot of attention on reranking.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
Just sharing a setup that ended up working reasonably well: DeepSeek, vLLM, A6000, with a lot of attention on reranking.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1067 Sun, 11 Oct 2026 20:40:47 +0000 latentpeak22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1067
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=944 Sat, 10 Oct 2026 00:56:17 +0000 rocmriley]]> https://www.datasculpting.ai/inference/showthread.php?tid=944
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=870 Wed, 07 Oct 2026 10:11:59 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=870
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=789 Mon, 05 Oct 2026 22:41:50 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=789
Short version: MLX behaved well for me, but only after I stopped ignoring reranking.

Happy to post more details if anyone wants them. At this point the real bottleneck is my patience.]]>
Short version: MLX behaved well for me, but only after I stopped ignoring reranking.

Happy to post more details if anyone wants them. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=780 Mon, 05 Oct 2026 09:54:16 +0000 audiopath66]]> https://www.datasculpting.ai/inference/showthread.php?tid=780
Maybe useful as a starting point: llama.cpp on A6000, keeping prefix caching in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Maybe useful as a starting point: llama.cpp on A6000, keeping prefix caching in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=653 Thu, 01 Oct 2026 22:20:24 +0000 paperrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=653
My current setup is MLX with Threadripper box, and I keep running into MoE serving.

Would appreciate practical advice rather than theory.]]>
My current setup is MLX with Threadripper box, and I keep running into MoE serving.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=638 Thu, 01 Oct 2026 07:15:34 +0000 phasespark]]> https://www.datasculpting.ai/inference/showthread.php?tid=638
Short version: llama.cpp behaved well for me, but only after I stopped ignoring tokens/sec.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Short version: llama.cpp behaved well for me, but only after I stopped ignoring tokens/sec.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=627 Wed, 30 Sep 2026 21:49:27 +0000 cobaltworks25]]> https://www.datasculpting.ai/inference/showthread.php?tid=627
I'm mostly working with FLUX in LM Studio on A6000. The part I'm trying to understand is tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I'm mostly working with FLUX in LM Studio on A6000. The part I'm trying to understand is tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=628 Wed, 30 Sep 2026 11:30:17 +0000 marbletrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=628
My current setup is PyTorch with mini PC, and I keep running into tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
My current setup is PyTorch with mini PC, and I keep running into tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=549 Tue, 29 Sep 2026 04:58:12 +0000 latentpeak22]]> https://www.datasculpting.ai/inference/showthread.php?tid=549
My current setup is SGLang with Threadripper box, and I keep running into KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]>
My current setup is SGLang with Threadripper box, and I keep running into KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=552 Mon, 28 Sep 2026 14:52:31 +0000 mangohouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=552
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and GB200.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and GB200.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=454 Thu, 24 Sep 2026 15:16:52 +0000 audiopath66]]> https://www.datasculpting.ai/inference/showthread.php?tid=454
My current setup is SGLang with 4090, and I keep running into small local stacks.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]>
My current setup is SGLang with 4090, and I keep running into small local stacks.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]>