https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1318 Fri, 16 Oct 2026 16:14:05 +0000 macronest]]> https://www.datasculpting.ai/inference/showthread.php?tid=1318
I'm mostly working with Gemma in LM Studio on 5090. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]>
I'm mostly working with Gemma in LM Studio on 5090. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1260 Fri, 16 Oct 2026 01:26:53 +0000 southspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1260
Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Happy to post more details if anyone wants them.]]>
Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1277 Thu, 15 Oct 2026 15:19:50 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1277
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1090 Tue, 13 Oct 2026 03:28:41 +0000 relayforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1090
I tried this with Phi using vLLM on B200. The headline result was better than I expected, but workstation thermals still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with Phi using vLLM on B200. The headline result was better than I expected, but workstation thermals still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1091 Tue, 13 Oct 2026 02:36:17 +0000 macronest]]> https://www.datasculpting.ai/inference/showthread.php?tid=1091
I'm mostly working with DeepSeek in MLX on GB200. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm mostly working with DeepSeek in MLX on GB200. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1096 Mon, 12 Oct 2026 11:31:26 +0000 turboyard94]]> https://www.datasculpting.ai/inference/showthread.php?tid=1096
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=898 Thu, 08 Oct 2026 23:48:15 +0000 amberplane]]> https://www.datasculpting.ai/inference/showthread.php?tid=898
I've gone back and forth between LM Studio and llama.cpp for Llama. The interesting difference for me has been prefill/decode split.

What would you pick if you had to live with it for a few months? I keep promising myself this is the last rebuild.]]>
I've gone back and forth between LM Studio and llama.cpp for Llama. The interesting difference for me has been prefill/decode split.

What would you pick if you had to live with it for a few months? I keep promising myself this is the last rebuild.]]> https://www.datasculpting.ai/inference/showthread.php?tid=819 Tue, 06 Oct 2026 11:49:05 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=819
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=574 Tue, 29 Sep 2026 21:48:46 +0000 southspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=574
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=514 Sun, 27 Sep 2026 01:15:45 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=514
My current setup is llama.cpp with 4090, and I keep running into consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
My current setup is llama.cpp with 4090, and I keep running into consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=475 Fri, 25 Sep 2026 10:14:06 +0000 macronest]]> https://www.datasculpting.ai/inference/showthread.php?tid=475
Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, A6000, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, A6000, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=427 Tue, 22 Sep 2026 23:09:40 +0000 vectorember]]> https://www.datasculpting.ai/inference/showthread.php?tid=427
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around quant drift.

Happy to post more details if anyone wants them.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around quant drift.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=343 Fri, 18 Sep 2026 12:21:24 +0000 orbitbench6]]> https://www.datasculpting.ai/inference/showthread.php?tid=343
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=296 Tue, 15 Sep 2026 22:53:33 +0000 macronest]]> https://www.datasculpting.ai/inference/showthread.php?tid=296
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=276 Tue, 15 Sep 2026 01:14:52 +0000 macroforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=276
I tried this with Gemma using PyTorch on 5090. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I tried this with Gemma using PyTorch on 5090. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.]]>