https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:41 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1471 Sun, 18 Oct 2026 17:43:00 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1471 https://www.datasculpting.ai/inference/showthread.php?tid=1474 Sun, 18 Oct 2026 17:32:00 +0000 amberpatch25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1474 https://www.datasculpting.ai/inference/showthread.php?tid=1451 Sun, 18 Oct 2026 17:26:00 +0000 sanepeak48]]> https://www.datasculpting.ai/inference/showthread.php?tid=1451 https://www.datasculpting.ai/inference/showthread.php?tid=1417 Sun, 18 Oct 2026 01:09:01 +0000 sparrowworks88]]> https://www.datasculpting.ai/inference/showthread.php?tid=1417
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1212 Thu, 15 Oct 2026 02:14:33 +0000 dockember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1212
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1246 Wed, 14 Oct 2026 23:03:28 +0000 vectorhouse71]]> https://www.datasculpting.ai/inference/showthread.php?tid=1246
Short version: PyTorch behaved well for me, but only after I stopped ignoring NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Short version: PyTorch behaved well for me, but only after I stopped ignoring NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1034 Sat, 10 Oct 2026 18:45:50 +0000 deltagrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1034
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]>
One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1027 Sat, 10 Oct 2026 10:32:11 +0000 macropoint]]> https://www.datasculpting.ai/inference/showthread.php?tid=1027
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and B200.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and B200.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=956 Sat, 10 Oct 2026 01:19:40 +0000 torchsmith25]]> https://www.datasculpting.ai/inference/showthread.php?tid=956
Short version: vLLM behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.]]>
Short version: vLLM behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=925 Fri, 09 Oct 2026 03:35:58 +0000 vectorfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=925
One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]>
One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=909 Thu, 08 Oct 2026 22:45:45 +0000 cozygrid12]]> https://www.datasculpting.ai/inference/showthread.php?tid=909
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and GB200.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and GB200.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=724 Sat, 03 Oct 2026 15:23:41 +0000 quantyard]]> https://www.datasculpting.ai/inference/showthread.php?tid=724
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about structured output.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about structured output.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=476 Fri, 25 Sep 2026 06:48:50 +0000 vrampilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=476
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=402 Mon, 21 Sep 2026 23:37:32 +0000 sanescope]]> https://www.datasculpting.ai/inference/showthread.php?tid=402
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

I'm especially interested in mistakes worth avoiding.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=395 Mon, 21 Sep 2026 21:35:04 +0000 nanogarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=395
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.]]>