https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:40 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1382 Sun, 18 Oct 2026 03:28:37 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1382
I've gone back and forth between SGLang and llama.cpp for Mixtral. The interesting difference for me has been long context.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between SGLang and llama.cpp for Mixtral. The interesting difference for me has been long context.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1423 Sat, 17 Oct 2026 23:39:14 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1423
My current setup is Ollama with M3 Max, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
My current setup is Ollama with M3 Max, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1425 Sat, 17 Oct 2026 13:24:34 +0000 mintquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=1425
This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefix caching.

Feel free to tear it apart if you see something dumb.]]>
This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefix caching.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1354 Fri, 16 Oct 2026 22:10:41 +0000 batchcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1354
Maybe useful as a starting point: PyTorch on Threadripper box, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.]]>
Maybe useful as a starting point: PyTorch on Threadripper box, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1285 Fri, 16 Oct 2026 05:35:56 +0000 pixelcrate82]]> https://www.datasculpting.ai/inference/showthread.php?tid=1285
I'm mostly working with Phi in TensorRT-LLM on Threadripper box. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm mostly working with Phi in TensorRT-LLM on Threadripper box. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1302 Fri, 16 Oct 2026 02:34:41 +0000 batchcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1302
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1267 Thu, 15 Oct 2026 09:19:34 +0000 turboscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1267
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1242 Thu, 15 Oct 2026 04:36:04 +0000 torchtrack24]]> https://www.datasculpting.ai/inference/showthread.php?tid=1242
I've gone back and forth between Ollama and vLLM for FLUX. The interesting difference for me has been MoE serving.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between Ollama and vLLM for FLUX. The interesting difference for me has been MoE serving.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1169 Wed, 14 Oct 2026 03:51:13 +0000 kernelgrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1169
I tried this with Kimi using SGLang on Threadripper box. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I tried this with Kimi using SGLang on Threadripper box. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1172 Wed, 14 Oct 2026 03:17:31 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1172
This isn't revolutionary, but the combination of Ollama and mini PC felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
This isn't revolutionary, but the combination of Ollama and mini PC felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1145 Tue, 13 Oct 2026 21:31:28 +0000 rasterbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1145
Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1139 Mon, 12 Oct 2026 18:47:27 +0000 syshouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=1139
One thing I keep noticing is that people underestimate router policy until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]>
One thing I keep noticing is that people underestimate router policy until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1057 Mon, 12 Oct 2026 03:34:29 +0000 macrotinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=1057
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and B200.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and B200.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1023 Sun, 11 Oct 2026 00:09:36 +0000 kernelgrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1023
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling quant drift.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling quant drift.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=948 Fri, 09 Oct 2026 19:41:48 +0000 torchtrack24]]> https://www.datasculpting.ai/inference/showthread.php?tid=948
My current setup is Ollama with B200, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.]]>
My current setup is Ollama with B200, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.]]>