https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:43:48 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1493 Mon, 19 Oct 2026 02:33:39 +0000 nanobrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=1493
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1452 Sun, 18 Oct 2026 16:32:53 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1452
I tried this with FLUX using PyTorch on 7900 XTX. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with FLUX using PyTorch on 7900 XTX. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1412 Sat, 17 Oct 2026 17:34:48 +0000 nanobrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=1412
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1162 Tue, 13 Oct 2026 23:03:31 +0000 labnest]]> https://www.datasculpting.ai/inference/showthread.php?tid=1162
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and M3 Max.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and M3 Max.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1128 Mon, 12 Oct 2026 17:21:55 +0000 latentpeak22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1128
I've gone back and forth between MLX and Ollama for Llama. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between MLX and Ollama for Llama. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1000 Sat, 10 Oct 2026 20:58:33 +0000 tensorworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=1000
Short version: vLLM behaved well for me, but only after I stopped ignoring token cost.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Short version: vLLM behaved well for me, but only after I stopped ignoring token cost.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1028 Sat, 10 Oct 2026 11:51:16 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1028
I'm mostly working with FLUX in MLX on 5090. The part I'm trying to understand is workstation thermals.

Would appreciate practical advice rather than theory.]]>
I'm mostly working with FLUX in MLX on 5090. The part I'm trying to understand is workstation thermals.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=957 Sat, 10 Oct 2026 00:54:35 +0000 batchyard]]> https://www.datasculpting.ai/inference/showthread.php?tid=957
My current setup is SGLang with dual 3090, and I keep running into workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
My current setup is SGLang with dual 3090, and I keep running into workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=974 Fri, 09 Oct 2026 09:48:04 +0000 cozygrid12]]> https://www.datasculpting.ai/inference/showthread.php?tid=974
Two paths both seem viable, but they break down differently once tokens/sec becomes important.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Two paths both seem viable, but they break down differently once tokens/sec becomes important.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=889 Wed, 07 Oct 2026 13:05:39 +0000 plainscope50]]> https://www.datasculpting.ai/inference/showthread.php?tid=889
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=843 Tue, 06 Oct 2026 09:31:31 +0000 vectorcrate98]]> https://www.datasculpting.ai/inference/showthread.php?tid=843
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=760 Mon, 05 Oct 2026 05:24:35 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=760
I tried this with Qwen using LM Studio on B200. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.]]>
I tried this with Qwen using LM Studio on B200. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=663 Sat, 03 Oct 2026 00:29:02 +0000 nanobrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=663
My current setup is MLX with GB200, and I keep running into embeddings.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
My current setup is MLX with GB200, and I keep running into embeddings.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=666 Fri, 02 Oct 2026 21:52:32 +0000 stackpilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=666
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=662 Fri, 02 Oct 2026 12:24:58 +0000 vectorember]]> https://www.datasculpting.ai/inference/showthread.php?tid=662
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

Would rather learn from your scars than collect my own all at once.]]>