https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:43:48 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1494 Sun, 18 Oct 2026 22:01:00 +0000 dockember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1494
Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, A6000, with a lot of attention on batch size.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, A6000, with a lot of attention on batch size.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1489 Sun, 18 Oct 2026 19:02:00 +0000 thetapath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1489 https://www.datasculpting.ai/inference/showthread.php?tid=1441 Sun, 18 Oct 2026 18:50:00 +0000 oxidemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1441 https://www.datasculpting.ai/inference/showthread.php?tid=1446 Sun, 18 Oct 2026 15:13:53 +0000 edgecoast]]> https://www.datasculpting.ai/inference/showthread.php?tid=1446
I've gone back and forth between llama.cpp and llama.cpp for Kimi. The interesting difference for me has been embeddings.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I've gone back and forth between llama.cpp and llama.cpp for Kimi. The interesting difference for me has been embeddings.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1381 Sun, 18 Oct 2026 05:15:55 +0000 cobaltspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1381
I'm mostly working with Kimi in PyTorch on B200. The part I'm trying to understand is INT4.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm mostly working with Kimi in PyTorch on B200. The part I'm trying to understand is INT4.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1419 Sat, 17 Oct 2026 22:39:54 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1419
Maybe useful as a starting point: TensorRT-LLM on Threadripper box, keeping HBM constraints in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: TensorRT-LLM on Threadripper box, keeping HBM constraints in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1401 Sat, 17 Oct 2026 19:10:26 +0000 sparrowkite53]]> https://www.datasculpting.ai/inference/showthread.php?tid=1401
Maybe useful as a starting point: MLX on 5090, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: MLX on 5090, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1365 Sat, 17 Oct 2026 01:30:18 +0000 audiopath66]]> https://www.datasculpting.ai/inference/showthread.php?tid=1365
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about KV cache pressure.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about KV cache pressure.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1287 Fri, 16 Oct 2026 04:05:31 +0000 edgecoast]]> https://www.datasculpting.ai/inference/showthread.php?tid=1287
I tried this with Gemma using vLLM on M3 Max. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I tried this with Gemma using vLLM on M3 Max. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1266 Fri, 16 Oct 2026 01:27:17 +0000 sparrowkite53]]> https://www.datasculpting.ai/inference/showthread.php?tid=1266
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.]]>
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1221 Thu, 15 Oct 2026 02:19:51 +0000 southpatch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1221
This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefill/decode split.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefill/decode split.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1231 Wed, 14 Oct 2026 14:38:13 +0000 edgecoast]]> https://www.datasculpting.ai/inference/showthread.php?tid=1231
I'm mostly working with Mistral in Ollama on 4090. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm mostly working with Mistral in Ollama on 4090. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1147 Tue, 13 Oct 2026 14:11:33 +0000 edgecoast]]> https://www.datasculpting.ai/inference/showthread.php?tid=1147
This isn't revolutionary, but the combination of vLLM and B200 felt surprisingly usable once I tuned around HBM constraints.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
This isn't revolutionary, but the combination of vLLM and B200 felt surprisingly usable once I tuned around HBM constraints.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1074 Sun, 11 Oct 2026 15:31:44 +0000 amberpatch25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1074
My current setup is SGLang with B200, and I keep running into long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]>
My current setup is SGLang with B200, and I keep running into long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=943 Sat, 10 Oct 2026 03:52:34 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=943
I tried this with Gemma using PyTorch on Threadripper box. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I tried this with Gemma using PyTorch on Threadripper box. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>