https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:32:15 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1498 Sun, 18 Oct 2026 18:13:10 +0000 plainworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=1498
This isn't revolutionary, but the combination of LM Studio and 5090 felt surprisingly usable once I tuned around MoE serving.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of LM Studio and 5090 felt surprisingly usable once I tuned around MoE serving.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1436 Sat, 17 Oct 2026 15:39:06 +0000 turboscope5]]> https://www.datasculpting.ai/inference/showthread.php?tid=1436
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tail latency.

Curious if anyone else landed in the same place.]]>
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tail latency.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1359 Sat, 17 Oct 2026 00:37:26 +0000 latentscope35]]> https://www.datasculpting.ai/inference/showthread.php?tid=1359
Maybe useful as a starting point: llama.cpp on 4090, keeping VRAM headroom in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: llama.cpp on 4090, keeping VRAM headroom in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1356 Fri, 16 Oct 2026 12:14:44 +0000 fastbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1356
The more I work with local setups, the more it seems like structured output decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like structured output decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1204 Wed, 14 Oct 2026 15:03:32 +0000 cobaltworks25]]> https://www.datasculpting.ai/inference/showthread.php?tid=1204
My current setup is TensorRT-LLM with Threadripper box, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
My current setup is TensorRT-LLM with Threadripper box, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1137 Mon, 12 Oct 2026 22:28:23 +0000 thetaarch43]]> https://www.datasculpting.ai/inference/showthread.php?tid=1137
Maybe useful as a starting point: LM Studio on 7900 XTX, keeping consumer GPUs in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: LM Studio on 7900 XTX, keeping consumer GPUs in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1134 Mon, 12 Oct 2026 15:15:10 +0000 sanevault]]> https://www.datasculpting.ai/inference/showthread.php?tid=1134
I'm mostly working with Phi in PyTorch on 4090. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with Phi in PyTorch on 4090. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1089 Mon, 12 Oct 2026 13:29:24 +0000 pixelspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1089
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1033 Sat, 10 Oct 2026 23:42:56 +0000 dockkite69]]> https://www.datasculpting.ai/inference/showthread.php?tid=1033
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=961 Sat, 10 Oct 2026 01:10:49 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=961
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=968 Fri, 09 Oct 2026 11:52:47 +0000 dockember]]> https://www.datasculpting.ai/inference/showthread.php?tid=968
The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=895 Fri, 09 Oct 2026 05:54:30 +0000 kernelrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=895
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=911 Thu, 08 Oct 2026 23:07:52 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=911
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=841 Wed, 07 Oct 2026 00:34:05 +0000 vectorcrate98]]> https://www.datasculpting.ai/inference/showthread.php?tid=841
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Maybe obvious to some of you, but it clicked for me this week.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=707 Sun, 04 Oct 2026 02:08:03 +0000 fusedhall56]]> https://www.datasculpting.ai/inference/showthread.php?tid=707
Maybe useful as a starting point: MLX on Threadripper box, keeping quant drift in mind from the start.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Maybe useful as a starting point: MLX on Threadripper box, keeping quant drift in mind from the start.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>