https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:25:23 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1429 Sun, 18 Oct 2026 01:33:57 +0000 amberplane]]> https://www.datasculpting.ai/inference/showthread.php?tid=1429
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.]]>
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1376 Sat, 17 Oct 2026 15:58:54 +0000 fastlab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1376
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1338 Fri, 16 Oct 2026 20:30:39 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1338
Two paths both seem viable, but they break down differently once tail latency becomes important.

Interested in tradeoffs more than fan loyalty.]]>
Two paths both seem viable, but they break down differently once tail latency becomes important.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1196 Tue, 13 Oct 2026 15:22:13 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1196
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and B200.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and B200.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1095 Mon, 12 Oct 2026 22:15:00 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1095
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1071 Sun, 11 Oct 2026 23:26:44 +0000 forgemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1071
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Nothing scientific here, just practical notes from a real setup.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Nothing scientific here, just practical notes from a real setup.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=993 Sat, 10 Oct 2026 23:50:17 +0000 circuitkite]]> https://www.datasculpting.ai/inference/showthread.php?tid=993
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around prefix caching.

Curious if anyone else landed in the same place.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around prefix caching.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=883 Wed, 07 Oct 2026 22:27:10 +0000 mellowstream]]> https://www.datasculpting.ai/inference/showthread.php?tid=883
My current setup is llama.cpp with A6000, and I keep running into tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
My current setup is llama.cpp with A6000, and I keep running into tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=803 Tue, 06 Oct 2026 00:47:09 +0000 marbletrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=803
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=743 Sun, 04 Oct 2026 12:19:02 +0000 torchstation]]> https://www.datasculpting.ai/inference/showthread.php?tid=743
This isn't revolutionary, but the combination of SGLang and B200 felt surprisingly usable once I tuned around NVFP4.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of SGLang and B200 felt surprisingly usable once I tuned around NVFP4.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=577 Tue, 29 Sep 2026 12:23:11 +0000 phasenode]]> https://www.datasculpting.ai/inference/showthread.php?tid=577
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=464 Thu, 24 Sep 2026 21:26:43 +0000 torchstation]]> https://www.datasculpting.ai/inference/showthread.php?tid=464
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=439 Thu, 24 Sep 2026 03:02:16 +0000 emberpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=439
The more I work with local setups, the more it seems like embeddings decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like embeddings decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=406 Mon, 21 Sep 2026 16:32:23 +0000 audiopoint18]]> https://www.datasculpting.ai/inference/showthread.php?tid=406
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=267 Sun, 13 Sep 2026 18:57:32 +0000 cachelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=267
This isn't revolutionary, but the combination of MLX and A6000 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of MLX and A6000 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]>