https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:47 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1380 Sat, 17 Oct 2026 16:09:28 +0000 circuitkite]]> https://www.datasculpting.ai/inference/showthread.php?tid=1380
I've gone back and forth between llama.cpp and llama.cpp for DeepSeek. The interesting difference for me has been quant drift.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between llama.cpp and llama.cpp for DeepSeek. The interesting difference for me has been quant drift.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1420 Sat, 17 Oct 2026 11:05:37 +0000 deltastation]]> https://www.datasculpting.ai/inference/showthread.php?tid=1420
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about NVFP4.

Simple and boring is a valid answer.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about NVFP4.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1328 Sat, 17 Oct 2026 04:42:25 +0000 tensorriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=1328
I've gone back and forth between PyTorch and LM Studio for Llama. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between PyTorch and LM Studio for Llama. The interesting difference for me has been speculative decoding.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1191 Tue, 13 Oct 2026 19:11:43 +0000 promptfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=1191
I've gone back and forth between SGLang and MLX for Llama. The interesting difference for me has been embeddings.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I've gone back and forth between SGLang and MLX for Llama. The interesting difference for me has been embeddings.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1142 Tue, 13 Oct 2026 10:11:22 +0000 nanobrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=1142
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=937 Sat, 10 Oct 2026 03:11:52 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=937
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and A6000.

Simple and boring is a valid answer.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and A6000.

Simple and boring is a valid answer.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=955 Fri, 09 Oct 2026 16:24:33 +0000 torchstation]]> https://www.datasculpting.ai/inference/showthread.php?tid=955
Just sharing a setup that ended up working reasonably well: Kimi, llama.cpp, A6000, with a lot of attention on router policy.

Feel free to tear it apart if you see something dumb.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
Just sharing a setup that ended up working reasonably well: Kimi, llama.cpp, A6000, with a lot of attention on router policy.

Feel free to tear it apart if you see something dumb.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=767 Tue, 06 Oct 2026 05:48:22 +0000 sparrowghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=767
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=737 Mon, 05 Oct 2026 05:02:34 +0000 vrampilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=737
I've gone back and forth between LM Studio and PyTorch for Kimi. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]>
I've gone back and forth between LM Studio and PyTorch for Kimi. The interesting difference for me has been small local stacks.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=583 Tue, 29 Sep 2026 18:56:39 +0000 turboscope5]]> https://www.datasculpting.ai/inference/showthread.php?tid=583
I've gone back and forth between SGLang and LM Studio for Mixtral. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I've gone back and forth between SGLang and LM Studio for Mixtral. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=571 Mon, 28 Sep 2026 17:17:35 +0000 plainember52]]> https://www.datasculpting.ai/inference/showthread.php?tid=571
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=566 Mon, 28 Sep 2026 09:07:32 +0000 edgequill]]> https://www.datasculpting.ai/inference/showthread.php?tid=566
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling workstation thermals.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling workstation thermals.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=489 Fri, 25 Sep 2026 20:47:58 +0000 ggufvault]]> https://www.datasculpting.ai/inference/showthread.php?tid=489
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=401 Mon, 21 Sep 2026 23:01:07 +0000 mellowsupply]]> https://www.datasculpting.ai/inference/showthread.php?tid=401
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=326 Thu, 17 Sep 2026 22:07:45 +0000 emberquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=326
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.]]>