what made local AI click for you

[tensorkite]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once embeddings becomes important.

Trying to avoid rebuilding the whole thing twice.

[marbletrack]

Spent some time on this today and wanted to compare notes.

Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

I'm happy to test one more variant if you want a second data point.

[llm_wren]

Need a little more detail here. Is this local-only or are you routing anything externally?

[tensorkite]

Can you fill in a bit more context? Are you testing this on M3 Max or something else?

[plaingarden18]

Good point. I don't think your conclusion is far off.

[vectorshed43]

Appreciate the detail. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mintforge]

I tested this tonight. On dual 3090 with LM Studio, I saw basically the same behavior.

[ggufwire22]

I tested this tonight. On dual 3090 with LM Studio, I saw basically the same behavior.

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[quiettoken]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[plainscope50]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow f

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

I'm happy to test one more variant if you want a second data point.