https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1378 Sun, 18 Oct 2026 03:38:18 +0000 mellowstream]]> https://www.datasculpting.ai/inference/showthread.php?tid=1378
Just sharing a setup that ended up working reasonably well: Llama, LM Studio, M3 Max, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Just sharing a setup that ended up working reasonably well: Llama, LM Studio, M3 Max, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1389 Sat, 17 Oct 2026 08:24:35 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1389
Short version: SGLang behaved well for me, but only after I stopped ignoring VRAM headroom.

Curious if anyone else landed in the same place.]]>
Short version: SGLang behaved well for me, but only after I stopped ignoring VRAM headroom.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1329 Fri, 16 Oct 2026 09:30:16 +0000 vramarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1329
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling router policy.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling router policy.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1236 Wed, 14 Oct 2026 13:53:18 +0000 modrunner27]]> https://www.datasculpting.ai/inference/showthread.php?tid=1236
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1043 Mon, 12 Oct 2026 03:06:40 +0000 mangopath4]]> https://www.datasculpting.ai/inference/showthread.php?tid=1043
After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.]]>
After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1064 Sun, 11 Oct 2026 19:05:42 +0000 mangopeak72]]> https://www.datasculpting.ai/inference/showthread.php?tid=1064
Just sharing a setup that ended up working reasonably well: Llama, PyTorch, M3 Max, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Llama, PyTorch, M3 Max, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1063 Sun, 11 Oct 2026 17:20:18 +0000 emberrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1063
I'm mostly working with Mistral in MLX on A6000. The part I'm trying to understand is router policy.

Would appreciate practical advice rather than theory.]]>
I'm mostly working with Mistral in MLX on A6000. The part I'm trying to understand is router policy.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1037 Sun, 11 Oct 2026 14:25:14 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1037
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 4090.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 4090.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=949 Fri, 09 Oct 2026 15:47:25 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=949
I tried this with Mistral using LM Studio on A6000. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with Mistral using LM Studio on A6000. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=926 Thu, 08 Oct 2026 14:34:39 +0000 modrunner27]]> https://www.datasculpting.ai/inference/showthread.php?tid=926
I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been INT4.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between llama.cpp and MLX for FLUX. The interesting difference for me has been INT4.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=860 Thu, 08 Oct 2026 04:02:43 +0000 macrolab]]> https://www.datasculpting.ai/inference/showthread.php?tid=860
Two paths both seem viable, but they break down differently once reranking becomes important.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Two paths both seem viable, but they break down differently once reranking becomes important.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=725 Sat, 03 Oct 2026 13:46:56 +0000 mangopath4]]> https://www.datasculpting.ai/inference/showthread.php?tid=725
One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=729 Sat, 03 Oct 2026 09:13:31 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=729
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]>
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=633 Thu, 01 Oct 2026 23:40:31 +0000 batchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=633
Just sharing a setup that ended up working reasonably well: DeepSeek, SGLang, 5090, with a lot of attention on structured output.

Maybe useful as a starting point for newer folks. At this point the real bottleneck is my patience.]]>
Just sharing a setup that ended up working reasonably well: DeepSeek, SGLang, 5090, with a lot of attention on structured output.

Maybe useful as a starting point for newer folks. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=452 Wed, 23 Sep 2026 23:51:24 +0000 macronest]]> https://www.datasculpting.ai/inference/showthread.php?tid=452
I've gone back and forth between LM Studio and vLLM for DeepSeek. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.]]>
I've gone back and forth between LM Studio and vLLM for DeepSeek. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.]]>