<![CDATA[Inference - Workflows / Automation]]> https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:39 +0000 MyBB <![CDATA[DeepSeek V4.1 impressions for Llama]]> https://www.datasculpting.ai/inference/showthread.php?tid=1464 Sun, 18 Oct 2026 18:25:00 +0000 rocmhouse90]]> https://www.datasculpting.ai/inference/showthread.php?tid=1464 Quote:
Quote:that fixed it for me too.

Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.]]> Quote:
Quote:that fixed it for me too.

Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.]]> <![CDATA[which hosted model is actually worth paying for]]> https://www.datasculpting.ai/inference/showthread.php?tid=1316 Fri, 16 Oct 2026 08:18:24 +0000 circuitkite]]> https://www.datasculpting.ai/inference/showthread.php?tid=1316
Short version: PyTorch behaved well for me, but only after I stopped ignoring NVFP4.

Happy to post more details if anyone wants them.]]>
Short version: PyTorch behaved well for me, but only after I stopped ignoring NVFP4.

Happy to post more details if anyone wants them.]]> <![CDATA[which hosted model is actually worth paying for]]> https://www.datasculpting.ai/inference/showthread.php?tid=1327 Fri, 16 Oct 2026 08:06:05 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1327
I tried this with Phi using LM Studio on B200. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I tried this with Phi using LM Studio on B200. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> <![CDATA[release notes that matter: worth it?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1270 Fri, 16 Oct 2026 05:57:37 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1270
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> <![CDATA[which hosted model is actually worth paying for for DeepSeek]]> https://www.datasculpting.ai/inference/showthread.php?tid=1288 Fri, 16 Oct 2026 02:05:31 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1288
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> <![CDATA[release notes that matter]]> https://www.datasculpting.ai/inference/showthread.php?tid=1228 Thu, 15 Oct 2026 05:24:07 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1228
Two paths both seem viable, but they break down differently once batch size becomes important.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Two paths both seem viable, but they break down differently once batch size becomes important.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> <![CDATA[DeepSeek V4.1 impressions]]> https://www.datasculpting.ai/inference/showthread.php?tid=1207 Wed, 14 Oct 2026 09:25:27 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1207
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> <![CDATA[DeepSeek V4.1 impressions]]> https://www.datasculpting.ai/inference/showthread.php?tid=1175 Tue, 13 Oct 2026 09:49:59 +0000 llmarch83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1175
I tried this with Phi using llama.cpp on mini PC. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]>
I tried this with Phi using llama.cpp on mini PC. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]> <![CDATA[which hosted model is actually worth paying for]]> https://www.datasculpting.ai/inference/showthread.php?tid=994 Sat, 10 Oct 2026 06:00:29 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=994
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]> <![CDATA[frontier API value lately]]> https://www.datasculpting.ai/inference/showthread.php?tid=785 Mon, 05 Oct 2026 17:54:45 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=785
After a few runs on 4090, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on 4090, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> <![CDATA[frontier API value lately]]> https://www.datasculpting.ai/inference/showthread.php?tid=797 Mon, 05 Oct 2026 16:15:18 +0000 edgekite]]> https://www.datasculpting.ai/inference/showthread.php?tid=797
Short version: PyTorch behaved well for me, but only after I stopped ignoring speculative decoding.

Happy to post more details if anyone wants them.]]>
Short version: PyTorch behaved well for me, but only after I stopped ignoring speculative decoding.

Happy to post more details if anyone wants them.]]> <![CDATA[DeepSeek V4.1 impressions]]> https://www.datasculpting.ai/inference/showthread.php?tid=801 Mon, 05 Oct 2026 07:48:48 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=801
Short version: SGLang behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.]]>
Short version: SGLang behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.]]> <![CDATA[DeepSeek V4.1 impressions]]> https://www.datasculpting.ai/inference/showthread.php?tid=745 Mon, 05 Oct 2026 05:43:00 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=745
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> <![CDATA[frontier API value lately: what are you seeing?]]> https://www.datasculpting.ai/inference/showthread.php?tid=756 Sun, 04 Oct 2026 20:50:47 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=756
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> <![CDATA[release notes that matter]]> https://www.datasculpting.ai/inference/showthread.php?tid=677 Sat, 03 Oct 2026 04:07:00 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=677
The more I work with local setups, the more it seems like reranking decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like reranking decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>