https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:42 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1438 Sun, 18 Oct 2026 16:09:56 +0000 batchcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1438
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1367 Sat, 17 Oct 2026 01:08:34 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1367
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1335 Fri, 16 Oct 2026 22:39:10 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1335
I'm comparing two approaches here: keep it simple on 7900 XTX, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm comparing two approaches here: keep it simple on 7900 XTX, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1294 Thu, 15 Oct 2026 12:12:45 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1294
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with embeddings.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with embeddings.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1213 Thu, 15 Oct 2026 04:26:11 +0000 mintstation99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1213
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around batch size.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around batch size.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1232 Wed, 14 Oct 2026 23:41:38 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1232
Maybe useful as a starting point: TensorRT-LLM on 7900 XTX, keeping structured output in mind from the start.

Feel free to tear it apart if you see something dumb.]]>
Maybe useful as a starting point: TensorRT-LLM on 7900 XTX, keeping structured output in mind from the start.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1220 Wed, 14 Oct 2026 13:43:33 +0000 mintstation99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1220
I tried this with Mistral using Ollama on 7900 XTX. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I tried this with Mistral using Ollama on 7900 XTX. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1254 Wed, 14 Oct 2026 06:25:42 +0000 rocmriley]]> https://www.datasculpting.ai/inference/showthread.php?tid=1254
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1182 Wed, 14 Oct 2026 02:40:30 +0000 mintstation99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1182
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1143 Tue, 13 Oct 2026 15:02:21 +0000 deltagrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1143
Just sharing a setup that ended up working reasonably well: Mistral, TensorRT-LLM, GB200, with a lot of attention on INT4.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.]]>
Just sharing a setup that ended up working reasonably well: Mistral, TensorRT-LLM, GB200, with a lot of attention on INT4.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1153 Tue, 13 Oct 2026 11:39:42 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1153
I tried this with Llama using SGLang on Threadripper box. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with Llama using SGLang on Threadripper box. The headline result was better than I expected, but tokens/sec still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1113 Tue, 13 Oct 2026 02:08:24 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1113
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1025 Sat, 10 Oct 2026 19:12:10 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1025
I've gone back and forth between Ollama and SGLang for Llama. The interesting difference for me has been tokens/sec.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]>
I've gone back and forth between Ollama and SGLang for Llama. The interesting difference for me has been tokens/sec.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1002 Sat, 10 Oct 2026 13:47:35 +0000 vectorport]]> https://www.datasculpting.ai/inference/showthread.php?tid=1002
I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=987 Sat, 10 Oct 2026 06:14:40 +0000 relayforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=987
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.]]>