https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:39 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1431 Sat, 17 Oct 2026 22:14:56 +0000 latentgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1431
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1348 Sat, 17 Oct 2026 04:09:18 +0000 fastbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1348
I've gone back and forth between llama.cpp and MLX for Qwen. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I've gone back and forth between llama.cpp and MLX for Qwen. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1304 Fri, 16 Oct 2026 04:00:21 +0000 latentgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1304
The more I work with local setups, the more it seems like tail latency decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]>
The more I work with local setups, the more it seems like tail latency decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1239 Thu, 15 Oct 2026 02:36:23 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1239
Just sharing a setup that ended up working reasonably well: Mixtral, Ollama, 4090, with a lot of attention on small local stacks.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Mixtral, Ollama, 4090, with a lot of attention on small local stacks.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1173 Tue, 13 Oct 2026 15:02:49 +0000 turboarch95]]> https://www.datasculpting.ai/inference/showthread.php?tid=1173
The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]>
The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1030 Sun, 11 Oct 2026 05:48:53 +0000 batchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1030
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=945 Sat, 10 Oct 2026 03:32:49 +0000 thetarunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=945
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about HBM constraints.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about HBM constraints.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=602 Thu, 01 Oct 2026 05:19:25 +0000 thetashed24]]> https://www.datasculpting.ai/inference/showthread.php?tid=602
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=579 Wed, 30 Sep 2026 04:36:21 +0000 cozyarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=579
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Wondering if this is just where the whole space is heading now.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=540 Sun, 27 Sep 2026 18:51:34 +0000 deepseeked]]> https://www.datasculpting.ai/inference/showthread.php?tid=540
This isn't revolutionary, but the combination of SGLang and A6000 felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of SGLang and A6000 felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=471 Thu, 24 Sep 2026 23:32:06 +0000 vrambandit]]> https://www.datasculpting.ai/inference/showthread.php?tid=471
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=335 Fri, 18 Sep 2026 01:01:43 +0000 thetashed24]]> https://www.datasculpting.ai/inference/showthread.php?tid=335
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=310 Thu, 17 Sep 2026 01:06:51 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=310
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.]]>
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=292 Wed, 16 Sep 2026 05:35:16 +0000 promptfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=292
I tried this with Llama using SGLang on 7900 XTX. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]>
I tried this with Llama using SGLang on 7900 XTX. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=291 Tue, 15 Sep 2026 22:37:41 +0000 batchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=291
I've gone back and forth between LM Studio and MLX for Qwen. The interesting difference for me has been workstation thermals.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between LM Studio and MLX for Qwen. The interesting difference for me has been workstation thermals.

Interested in tradeoffs more than fan loyalty.]]>