https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1483 Sun, 18 Oct 2026 16:41:00 +0000 deltagrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1483 https://www.datasculpting.ai/inference/showthread.php?tid=1408 Sat, 17 Oct 2026 17:34:17 +0000 cudapeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1408
I'm mostly working with DeepSeek in LM Studio on 5090. The part I'm trying to understand is NVFP4.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with DeepSeek in LM Studio on 5090. The part I'm trying to understand is NVFP4.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1349 Fri, 16 Oct 2026 19:37:17 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1349
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1342 Fri, 16 Oct 2026 14:35:10 +0000 kernelrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1342
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1306 Thu, 15 Oct 2026 23:22:22 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1306
Just sharing a setup that ended up working reasonably well: Kimi, LM Studio, 7900 XTX, with a lot of attention on continuous batching.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Just sharing a setup that ended up working reasonably well: Kimi, LM Studio, 7900 XTX, with a lot of attention on continuous batching.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1192 Tue, 13 Oct 2026 22:48:08 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1192
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.]]>
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1109 Tue, 13 Oct 2026 04:49:12 +0000 cudapeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1109
After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around router policy.

Curious if anyone else landed in the same place.]]>
After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around router policy.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1130 Mon, 12 Oct 2026 11:38:45 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1130
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1140 Mon, 12 Oct 2026 09:11:05 +0000 macropilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=1140
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1048 Sun, 11 Oct 2026 14:41:57 +0000 localpatch78]]> https://www.datasculpting.ai/inference/showthread.php?tid=1048
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1031 Sun, 11 Oct 2026 03:34:51 +0000 localpatch78]]> https://www.datasculpting.ai/inference/showthread.php?tid=1031
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling router policy.

Not looking for the perfect answer, mostly what worked in real use.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling router policy.

Not looking for the perfect answer, mostly what worked in real use.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1017 Sun, 11 Oct 2026 01:40:50 +0000 cudapeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1017
I'm mostly working with FLUX in MLX on 7900 XTX. The part I'm trying to understand is quant drift.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm mostly working with FLUX in MLX on 7900 XTX. The part I'm trying to understand is quant drift.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1029 Sat, 10 Oct 2026 13:54:14 +0000 sparrowshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1029
Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.]]>
Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=953 Sat, 10 Oct 2026 01:05:15 +0000 kernelrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=953
This isn't revolutionary, but the combination of LM Studio and GB200 felt surprisingly usable once I tuned around continuous batching.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of LM Studio and GB200 felt surprisingly usable once I tuned around continuous batching.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=878 Thu, 08 Oct 2026 04:04:27 +0000 papermixer4]]> https://www.datasculpting.ai/inference/showthread.php?tid=878
Maybe useful as a starting point: LM Studio on 5090, keeping prefix caching in mind from the start.

Feel free to tear it apart if you see something dumb.]]>
Maybe useful as a starting point: LM Studio on 5090, keeping prefix caching in mind from the start.

Feel free to tear it apart if you see something dumb.]]>