on-device inference worth the effort with llama.cpp

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Linear Mode

on-device inference worth the effort with llama.cpp

sparrowghost Offline

Junior Member

Posts: 24
Threads: 2
Joined: Sep 2026
Reputation: 0

10-18-2026, 04:34 AM

Good point. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

Find

« Next Oldest | Next Newest »

Messages In This Thread

on-device inference worth the effort with llama.cpp - by marbleyard32 - 10-17-2026, 05:22 PM

Re: on-device inference worth the effort with llama.cpp - by marbleyard32 - 10-17-2026, 06:55 PM

Re: on-device inference worth the effort with llama.cpp - by cozysmith - 10-17-2026, 08:15 PM

Re: on-device inference worth the effort with llama.cpp - by sparrowghost - 10-18-2026, 04:34 AM

Re: on-device inference worth the effort with llama.cpp - by batchpeak - 10-19-2026, 03:16 AM

Re: on-device inference worth the effort with llama.cpp - by llmpeak - 10-18-2026, 06:55 AM

Re: on-device inference worth the effort with llama.cpp - by turbosupply83 - 10-18-2026, 10:09 PM

Re: on-device inference worth the effort with llama.cpp - by sparrowshed - 10-18-2026, 10:50 PM

Re: on-device inference worth the effort with llama.cpp - by cozywire - 10-18-2026, 11:04 PM

Re: on-device inference worth the effort with llama.cpp - by mangopath4 - 10-18-2026, 11:12 PM

View a Printable Version

Forum Jump:

Users browsing this thread: 1 Guest(s)

Login
Username:
Password:	Lost Password?
	Remember me

Sign in with X.com