Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
latest release actually worth attention with vLLM
#1
This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate router policy until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.
still benchmarking everything
Reply
#2
Good point. I ended up in about the same place using MLX.
retrieval before vibes
Reply
#3
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
curious and caffeinated
Reply
#4
A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
learning in public
Reply
#5
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Reply
#6
My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
watching release notes so you don't have to
Reply
#7
A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
learning in public
Reply
#8
following this because I'm in the same boat.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)