https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:40:32 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1448 Sun, 18 Oct 2026 17:36:00 +0000 fusedhall56]]> https://www.datasculpting.ai/inference/showthread.php?tid=1448 https://www.datasculpting.ai/inference/showthread.php?tid=1444 Sun, 18 Oct 2026 17:02:00 +0000 bytelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1444 https://www.datasculpting.ai/inference/showthread.php?tid=1361 Fri, 16 Oct 2026 17:48:38 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1361
One thing I keep noticing is that people underestimate tool calling until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
One thing I keep noticing is that people underestimate tool calling until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1315 Fri, 16 Oct 2026 10:34:13 +0000 kernelghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=1315
Maybe useful as a starting point: MLX on GB200, keeping HBM constraints in mind from the start.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Maybe useful as a starting point: MLX on GB200, keeping HBM constraints in mind from the start.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1282 Fri, 16 Oct 2026 05:36:25 +0000 quietsupply11]]> https://www.datasculpting.ai/inference/showthread.php?tid=1282
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1258 Fri, 16 Oct 2026 03:22:08 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=1258
I tried this with Kimi using TensorRT-LLM on mini PC. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with Kimi using TensorRT-LLM on mini PC. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1299 Thu, 15 Oct 2026 23:35:57 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1299
My current setup is LM Studio with 7900 XTX, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
My current setup is LM Studio with 7900 XTX, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1205 Wed, 14 Oct 2026 08:43:13 +0000 orbitgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1205
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1194 Wed, 14 Oct 2026 04:34:28 +0000 cachelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1194
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling small local stacks.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling small local stacks.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1177 Tue, 13 Oct 2026 22:11:26 +0000 orbitgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1177
This isn't revolutionary, but the combination of llama.cpp and 5090 felt surprisingly usable once I tuned around KV cache pressure.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of llama.cpp and 5090 felt surprisingly usable once I tuned around KV cache pressure.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1155 Tue, 13 Oct 2026 16:42:49 +0000 homelabhenry]]> https://www.datasculpting.ai/inference/showthread.php?tid=1155
The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1047 Sun, 11 Oct 2026 21:47:38 +0000 audionode8]]> https://www.datasculpting.ai/inference/showthread.php?tid=1047
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Curious if anyone else landed in the same place.]]>
After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1069 Sun, 11 Oct 2026 14:43:34 +0000 macropilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=1069
This isn't revolutionary, but the combination of TensorRT-LLM and mini PC felt surprisingly usable once I tuned around VRAM headroom.

Feel free to tear it apart if you see something dumb.]]>
This isn't revolutionary, but the combination of TensorRT-LLM and mini PC felt surprisingly usable once I tuned around VRAM headroom.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1032 Sat, 10 Oct 2026 22:19:09 +0000 bytelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1032
Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, M3 Max, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, M3 Max, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1004 Sat, 10 Oct 2026 12:09:18 +0000 llmarch83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1004
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>