https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:44 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1477 Sun, 18 Oct 2026 17:15:00 +0000 quiethouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=1477 https://www.datasculpting.ai/inference/showthread.php?tid=1486 Sun, 18 Oct 2026 16:47:00 +0000 cudascope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1486 https://www.datasculpting.ai/inference/showthread.php?tid=1385 Sat, 17 Oct 2026 11:59:29 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1385
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1244 Thu, 15 Oct 2026 01:49:04 +0000 phasetrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=1244
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1195 Wed, 14 Oct 2026 02:48:26 +0000 amberpatch25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1195
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1164 Tue, 13 Oct 2026 12:01:54 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1164
Two paths both seem viable, but they break down differently once tail latency becomes important.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Two paths both seem viable, but they break down differently once tail latency becomes important.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1039 Mon, 12 Oct 2026 00:58:42 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1039
One thing I keep noticing is that people underestimate MoE serving until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
One thing I keep noticing is that people underestimate MoE serving until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1009 Sun, 11 Oct 2026 00:06:21 +0000 kernelpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1009
I tried this with DeepSeek using Ollama on M3 Max. The headline result was better than I expected, but long context still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with DeepSeek using Ollama on M3 Max. The headline result was better than I expected, but long context still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=979 Fri, 09 Oct 2026 12:10:39 +0000 visionforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=979
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=928 Fri, 09 Oct 2026 01:40:33 +0000 cobaltworks25]]> https://www.datasculpting.ai/inference/showthread.php?tid=928
The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=809 Tue, 06 Oct 2026 23:24:50 +0000 bytegarden73]]> https://www.datasculpting.ai/inference/showthread.php?tid=809
The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=831 Tue, 06 Oct 2026 08:44:59 +0000 llmtrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=831
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=794 Mon, 05 Oct 2026 15:38:00 +0000 vectorfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=794
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=694 Fri, 02 Oct 2026 08:04:01 +0000 vectorfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=694
My current setup is TensorRT-LLM with B200, and I keep running into prefill/decode split.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
My current setup is TensorRT-LLM with B200, and I keep running into prefill/decode split.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=565 Tue, 29 Sep 2026 04:25:23 +0000 dockember]]> https://www.datasculpting.ai/inference/showthread.php?tid=565
Two paths both seem viable, but they break down differently once KV cache pressure becomes important.

Interested in tradeoffs more than fan loyalty.]]>
Two paths both seem viable, but they break down differently once KV cache pressure becomes important.

Interested in tradeoffs more than fan loyalty.]]>