vision-language models on consumer gear: worth it?

[marbletrack]

Wanted to compare notes with people doing similar things.

My current setup is PyTorch with mini PC, and I keep running into tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.