+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: what made local AI click for you (/showthread.php?tid=675)
Pages:
1
2
what made local AI click for you - tensorkite - 10-02-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once embeddings becomes important.
Trying to avoid rebuilding the whole thing twice.
Re: what made local AI click for you - marbletrack - 10-03-2026
Quote:Spent some time on this today and wanted to compare notes.
Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: what made local AI click for you - llm_wren - 10-04-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: what made local AI click for you - tensorkite - 10-06-2026
Can you fill in a bit more context? Are you testing this on M3 Max or something else?
Re: what made local AI click for you - plaingarden18 - 10-06-2026
Good point. I don't think your conclusion is far off.
Re: what made local AI click for you - vectorshed43 - 10-06-2026
Appreciate the detail. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: what made local AI click for you - mintforge - 10-06-2026
I tested this tonight. On dual 3090 with LM Studio, I saw basically the same behavior.
Re: what made local AI click for you - ggufwire22 - 10-06-2026
Quote:I tested this tonight. On dual 3090 with LM Studio, I saw basically the same behavior.
That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: what made local AI click for you - quiettoken - 10-06-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: what made local AI click for you - plainscope50 - 10-06-2026
Quote:Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow f
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
I'm happy to test one more variant if you want a second data point.