https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:25:20 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1334 Fri, 16 Oct 2026 19:49:34 +0000 llmarch83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1334
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

Simple and boring is a valid answer.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1159 Wed, 14 Oct 2026 00:10:11 +0000 vectorfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=1159
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1184 Tue, 13 Oct 2026 18:46:23 +0000 batchpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1184
I've gone back and forth between LM Studio and MLX for DeepSeek. The interesting difference for me has been reranking.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I've gone back and forth between LM Studio and MLX for DeepSeek. The interesting difference for me has been reranking.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1049 Mon, 12 Oct 2026 01:51:05 +0000 sparrowhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1049
Just sharing a setup that ended up working reasonably well: Phi, SGLang, Threadripper box, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Phi, SGLang, Threadripper box, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1006 Sat, 10 Oct 2026 17:33:49 +0000 cudapoint]]> https://www.datasculpting.ai/inference/showthread.php?tid=1006
Maybe useful as a starting point: MLX on dual 3090, keeping NVFP4 in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: MLX on dual 3090, keeping NVFP4 in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=972 Sat, 10 Oct 2026 04:28:45 +0000 signalpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=972
I tried this with Mistral using LM Studio on 7900 XTX. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I tried this with Mistral using LM Studio on 7900 XTX. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=946 Fri, 09 Oct 2026 16:46:01 +0000 signalcraft14]]> https://www.datasculpting.ai/inference/showthread.php?tid=946
The more I work with local setups, the more it seems like router policy decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like router policy decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=904 Thu, 08 Oct 2026 16:12:46 +0000 batchpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=904
After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Nothing scientific here, just practical notes from a real setup.]]>
After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=827 Tue, 06 Oct 2026 23:52:23 +0000 vramport]]> https://www.datasculpting.ai/inference/showthread.php?tid=827
One thing I keep noticing is that people underestimate HBM constraints until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate HBM constraints until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=821 Tue, 06 Oct 2026 07:56:43 +0000 signalcraft14]]> https://www.datasculpting.ai/inference/showthread.php?tid=821
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and Threadripper box.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and Threadripper box.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=695 Sat, 03 Oct 2026 23:25:57 +0000 signalcraft14]]> https://www.datasculpting.ai/inference/showthread.php?tid=695
I tried this with Mixtral using SGLang on GB200. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]>
I tried this with Mixtral using SGLang on GB200. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=551 Mon, 28 Sep 2026 21:16:32 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=551
I've gone back and forth between SGLang and MLX for Mixtral. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I've gone back and forth between SGLang and MLX for Mixtral. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=568 Mon, 28 Sep 2026 18:47:13 +0000 cacheworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=568
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and A6000.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and A6000.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=525 Mon, 28 Sep 2026 01:53:13 +0000 ggufvault]]> https://www.datasculpting.ai/inference/showthread.php?tid=525
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=474 Thu, 24 Sep 2026 07:47:38 +0000 amberkite]]> https://www.datasculpting.ai/inference/showthread.php?tid=474
One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.]]>
One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.]]>