https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:30 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1467 Sun, 18 Oct 2026 16:56:00 +0000 bytelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1467 https://www.datasculpting.ai/inference/showthread.php?tid=1463 Sun, 18 Oct 2026 10:58:27 +0000 nanogarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1463
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, M3 Max, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.]]>
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, M3 Max, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1433 Sun, 18 Oct 2026 05:56:45 +0000 mangowire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1433
One thing I keep noticing is that people underestimate structured output until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.]]>
One thing I keep noticing is that people underestimate structured output until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1418 Sun, 18 Oct 2026 02:55:33 +0000 thetashed24]]> https://www.datasculpting.ai/inference/showthread.php?tid=1418
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.]]>
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1394 Sun, 18 Oct 2026 02:28:25 +0000 northcraft90]]> https://www.datasculpting.ai/inference/showthread.php?tid=1394
I'm mostly working with Qwen in Ollama on A6000. The part I'm trying to understand is token cost.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm mostly working with Qwen in Ollama on A6000. The part I'm trying to understand is token cost.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1387 Sun, 18 Oct 2026 00:56:47 +0000 deepseeked]]> https://www.datasculpting.ai/inference/showthread.php?tid=1387
I've gone back and forth between vLLM and SGLang for Mixtral. The interesting difference for me has been tail latency.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between vLLM and SGLang for Mixtral. The interesting difference for me has been tail latency.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1384 Sat, 17 Oct 2026 23:11:07 +0000 southpatch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1384
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1397 Sat, 17 Oct 2026 23:06:10 +0000 audiopoint18]]> https://www.datasculpting.ai/inference/showthread.php?tid=1397
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1395 Sat, 17 Oct 2026 15:38:43 +0000 fastlab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1395
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1414 Sat, 17 Oct 2026 08:57:33 +0000 sparrowworks88]]> https://www.datasculpting.ai/inference/showthread.php?tid=1414
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1332 Sat, 17 Oct 2026 05:40:50 +0000 modbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1332
I tried this with Mixtral using Ollama on M3 Max. The headline result was better than I expected, but prefill/decode split still ended up being the limiting factor.

Happy to post more details if anyone wants them.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I tried this with Mixtral using Ollama on M3 Max. The headline result was better than I expected, but prefill/decode split still ended up being the limiting factor.

Happy to post more details if anyone wants them.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1371 Sat, 17 Oct 2026 04:36:47 +0000 deepseeked]]> https://www.datasculpting.ai/inference/showthread.php?tid=1371
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1370 Fri, 16 Oct 2026 21:52:38 +0000 deepseeked]]> https://www.datasculpting.ai/inference/showthread.php?tid=1370
Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, dual 3090, with a lot of attention on HBM constraints.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, dual 3090, with a lot of attention on HBM constraints.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1305 Thu, 15 Oct 2026 23:49:52 +0000 modbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1305
Two paths both seem viable, but they break down differently once quant drift becomes important.

Trying to avoid rebuilding the whole thing twice.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
Two paths both seem viable, but they break down differently once quant drift becomes important.

Trying to avoid rebuilding the whole thing twice.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1274 Thu, 15 Oct 2026 22:49:10 +0000 mangovault67]]> https://www.datasculpting.ai/inference/showthread.php?tid=1274
Maybe useful as a starting point: TensorRT-LLM on 7900 XTX, keeping consumer GPUs in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: TensorRT-LLM on 7900 XTX, keeping consumer GPUs in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>