Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
clean API wrappers for local agents for FLUX
#1
This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Curious whether that lines up with what other people are seeing.
Mostly CUDA, sometimes wrong, usually measured
Reply
#2
Quote:This keeps coming up in conversation so I figured I'd start a thread.

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. At this point the real bottleneck is my patience.
rack in the garage • local models everywhere
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)