| Welcome, Guest |
You have to register before you can post on our site.
|
| Online Users |
There are currently 208 online users.
» 0 Member(s) | 205 Guest(s)
Applebot, Bing, Google
|
| Latest Threads |
FC 27 Season 2 Coins Guid...
Forum: Off Topic
Last Post: salisy
Yesterday, 01:50 AM
» Replies: 0
» Views: 20
|
Aion 2 October 21 Update ...
Forum: Off Topic
Last Post: salisy
Yesterday, 01:33 AM
» Replies: 0
» Views: 26
|
CoD: Modern Warfare 4 Is ...
Forum: Off Topic
Last Post: salisy
Yesterday, 01:18 AM
» Replies: 0
» Views: 22
|
consumer accelerators wor...
Forum: CPUs / NPUs / Accelerators
Last Post: rocmcrate26
10-19-2026, 05:59 AM
» Replies: 11
» Views: 291
|
structured output failure...
Forum: Structured Output
Last Post: latentpeak22
10-19-2026, 05:59 AM
» Replies: 13
» Views: 410
|
CUDA update broke my thro...
Forum: NVIDIA / CUDA
Last Post: nanobrook
10-19-2026, 05:58 AM
» Replies: 13
» Views: 377
|
70B still worth the pain
Forum: Proprietary / Frontier
Last Post: routerunner
10-19-2026, 05:58 AM
» Replies: 15
» Views: 408
|
context size vs speed in ...
Forum: llama.cpp / GGUF
Last Post: macroforge
10-19-2026, 05:58 AM
» Replies: 15
» Views: 428
|
do small edge boxes make ...
Forum: CPUs / NPUs / Accelerators
Last Post: kernelmike
10-19-2026, 05:58 AM
» Replies: 15
» Views: 436
|
8B vs 14B sweet spot: any...
Forum: Proprietary / Frontier
Last Post: forgemixer
10-19-2026, 05:57 AM
» Replies: 8
» Views: 263
|
|
|
| Ollama for power users |
|
Posted by: dockridge - 10-08-2026, 04:14 AM - Forum: Ollama / LM Studio / Desktop Runtimes
- Replies (6)
|
 |
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of MLX and 7900 XTX felt surprisingly usable once I tuned around long context.
Feel free to tear it apart if you see something dumb.
|
|
|
| good hosted AI app that still respects privacy on 4090 |
|
Posted by: macrolab - 10-08-2026, 04:02 AM - Forum: Online / Hosted
- Replies (13)
|
 |
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once reranking becomes important.
Interested in tradeoffs more than fan loyalty.
I care more about stability than squeezing the last possible tokens/sec out of it.
|
|
|
|
| prefix caching in llama.cpp for Kimi |
|
Posted by: macroforge - 10-08-2026, 03:58 AM - Forum: llama.cpp / GGUF
- Replies (6)
|
 |
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: LM Studio on A6000, keeping consumer GPUs in mind from the start.
Feel free to tear it apart if you see something dumb.
|
|
|
|
| best local open-weight model this week with vLLM |
|
Posted by: cozykite - 10-08-2026, 03:33 AM - Forum: Proprietary / Frontier
- Replies (2)
|
 |
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mixtral, llama.cpp, mini PC, with a lot of attention on workstation thermals.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it.
|
|
|
|
| adapter stacking for DeepSeek |
|
Posted by: cacheline - 10-08-2026, 02:25 AM - Forum: Benchmarks
- Replies (14)
|
 |
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once continuous batching becomes important.
Interested in tradeoffs more than fan loyalty.
|
|
|
|
| distill or just quantize with Ollama |
|
Posted by: thetaplane - 10-08-2026, 02:23 AM - Forum: Agents
- Replies (12)
|
 |
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: vLLM on mini PC, keeping HBM constraints in mind from the start.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
|
|
|
|
| non-AI thing you built recently for DeepSeek |
|
Posted by: promptfield - 10-08-2026, 02:06 AM - Forum: Off Topic
- Replies (6)
|
|
Has anyone else run into this?
I'm mostly working with DeepSeek in vLLM on GB200. The part I'm trying to understand is speculative decoding.
Would appreciate practical advice rather than theory.
I care more about stability than squeezing the last possible tokens/sec out of it.
|
|
|
|
| best mini PC for local use with vLLM |
|
Posted by: vramghost - 10-07-2026, 11:41 PM - Forum: SBCs / Mini PCs
- Replies (15)
|
 |
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: LM Studio on A6000, keeping HBM constraints in mind from the start.
Feel free to tear it apart if you see something dumb.
|
|
|
|
| KV cache compression worth it for Mixtral |
|
Posted by: deltagrid - 10-07-2026, 11:37 PM - Forum: Other Engines
- No Replies
|
 |
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.
Trying to avoid rebuilding the whole thing twice.
|
|
|
|
|
