on-device inference worth the effort

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Linear Mode

on-device inference worth the effort

turbosupply83 Offline

Junior Member

Posts: 30
Threads: 3
Joined: Sep 2026
Reputation: 0

09-16-2026, 08:21 AM

Quote:What exact setup are you running? Is this local-only or are you routing anything externally?

Did a small comparison this afternoon. On 5090 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

Find

« Next Oldest | Next Newest »

Messages In This Thread

on-device inference worth the effort - by marbleyard32 - 09-09-2026, 03:39 AM

Re: on-device inference worth the effort - by vramarch - 09-09-2026, 05:17 AM

Re: on-device inference worth the effort - by batchpeak - 09-12-2026, 05:32 AM

Re: on-device inference worth the effort - by plainline - 09-13-2026, 05:47 AM

Re: on-device inference worth the effort - by routeline23 - 09-16-2026, 06:08 AM

Re: on-device inference worth the effort - by sparrowghost - 09-16-2026, 07:22 AM

Re: on-device inference worth the effort - by turbosupply83 - 09-16-2026, 08:21 AM

Re: on-device inference worth the effort - by plainscope50 - 09-19-2026, 11:14 PM

Re: on-device inference worth the effort - by routeline23 - 09-16-2026, 10:22 AM

Re: on-device inference worth the effort - by sparrowghost - 09-19-2026, 10:28 AM

Re: on-device inference worth the effort - by plainline - 09-20-2026, 03:16 AM

Re: on-device inference worth the effort - by promptspark58 - 09-20-2026, 04:13 AM

Re: on-device inference worth the effort - by amberrunner - 09-20-2026, 04:53 AM

View a Printable Version

Forum Jump:

Users browsing this thread: 1 Guest(s)

Login
Username:
Password:	Lost Password?
	Remember me

Sign in with X.com