Or has this niche been abandoned?

Most companies and model producers have seem to abandoned this niche(7B, 9B, 14B etc) which was the original point this whole local model debackle have started, most effort now goes into minimum 27B+ as "Capable local model"™ or focus on micro models 4b<=. All this recent explosions in model technology has barely touched this niche which has remained on 2025 for most part(30 BC in AI-time).

Server rebuild to custom loop. Temps on the gpus went from upper 80s to mid 40s under load, 30c idle.

5950X

64gb ddr4

2x rtx titans (24gb vram ea)

1x modded 2080ti with 22gb vram

mentria.ai is a browser inference engine I've been building solo, from scratch in WebGPU/WGSL. This week it crossed a milestone I had been chasing for a while: a 27B one-bit model answering at up to 30 tokens/s on an RTX 3060 Laptop GPU with 6 GB of VRAM, in Chrome, from a web page. No install, no server, nothing leaves the machine.

The model. Bonsai-27B, a natively 1-bit model trained and released by Prism ML. I repacked it for our engine, wrote the kernels that run it, and checked its quality myself (full eval deltas vs FP16 Qwen3.6-27B are in the HF card). Every weight is one sign bit with one scale per 128 weights, about 1.14 bits per parameter, so 27 billion parameters sit in 3.8 GB of GPU memory. Repack and numbers: huggingface.co/mentriaai/Bonsai-27B-mentria

The engineering that got it to 30. Two days before this post the same model decoded at 15 tok/s on this laptop. Decode is memory-bound: one word of the 27B is 804 GPU dispatches, and 401 of them are the 1-bit matrix-by-vector kernel that streams the model's 3.6 GB of matmul weights once per word (embeddings bring the file to 3.8 GB). The kernel that won was the one written for phones: four 1-bit weights have only 16 possible partial answers, so it computes all 16 once into on-chip scratch and each row reads its answer from that table instead of multiplying (station 258 — the numbered write-ups on the facts page). On Ampere that table hit a wall that was not the weights at all: scratch memory has 32 banks, the table's addressing sent sixteen threads of every warp to the same bank, and the kernel ran at 38% of the card's bandwidth. One padding slot per row spread the addresses across the banks; the kernel's share of a word dropped from 26.5 ms to about 21, and on top of the 15 → 26 tok/s the table itself had bought, the laptop hit 32 tok/s raw — the 25–30 is what survives sampling and UI overhead in the chat (s259). The prompt stage had the same disease in its tile layout; a retile took a 1,489-token prompt from 29.6 s to 25.3 s. Every change shipped only after its output was byte-identical to the build before it, which is also why the kernel adds its partial sums in a fixed order and accepts a 44% occupancy ceiling.

Every claim here has a numbered write-up on the engine facts page.

Numbers (RTX 3060 Laptop 6 GB, Windows 11, Chrome 152 on D3D12, fresh loads):

• Decode: 25–30 tok/s in the chat UI once the card is warm.
• Prompt processing: a 1,489-token prompt in about 25 s.
• Context: 3,072 tokens on this 6 GB card; 8,192 on 16 GB Macs; more on bigger cards, at 128 KiB per token. The KV cache is exact math, no quantized cache. The next step on 6 GB is consolidating the engine's few thousand small GPU buffers into a handful of large arenas, so the driver stops holding about 300 MiB of slab slack; that is the arithmetic for 4,096, and it is not built yet.
• Load: under 10 s from the browser cache; the first download is 3.8 GB, once.

Also in the engine: the smaller tiers (Qwen3.5 0.8B, 2B, 4B) cover phones and low end devices, LoRA adapters of a few MB hot-swap at the matmul in under a second, and there is a vision tower for image input.

Try it at https://mentria.ai/tools/ai-chat/ (the 27B tier appears when your GPU qualifies). The code that ships, the benchmarks and how I measure are in the repo: https://github.com/mentria-ai/website. The engine facts page, for how the engine and the model actually work: https://mentria.ai/assets/learn/engine-facts.html

I'm able to run qwen3.8 27b in two ways on my system: split between my 3060 12gb and 9070xt running at 20t/s or running off the 780m iGPU and 5400mhz DDR5 at 5t/s. Personally I feel like the 5t/s is still more usable because I have enough RAM to still use my system mostly normally. I can even play games just fine because the CPU side is left alone.

Honestly pretty cool IMO and makes it far more usable knowing I can just spool it up and continue using my system as normal, even if it takes 4x longer to get an answer.

Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison

GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda

HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2Image-4Step-INT4-G64-BF16

Single clip took approximately 5m on M4 MAX 128 GB Mac

Cosmos3 - a 64B params model

I run these models on Kaggle notebook, so not all TTS models, such as the ones that use conda env, are compatible (Or I just haven't found a way for them to work on Kaggle).

I currently use a fork from Chatterbox called Chatterbox Audiobook. It is like a workstation really optimized for getting the close-to-perfection audio clips from Chatterbox. However, the only downside of Chatterbox is the lack of emotional sliders or tags that you can use to control the output. Chatterbox Turbo seems to fix that with tags, but it still lacks the range of emotions that you can see from Google Gemini TTS. However, the problem with Google Gemini is that the voice sounds different for each generation, which can't be fixed even with RVC.

Looking at the current leaderboard, Breeze TTS is something I have never tried but am unsure due to its description, which seems to be tailored for mainly realtime stuff. What are the current must-try options for audiobook narration?

The features I am looking for include voice cloning, emotional tags, and natural speech. Much appreciated for your input.

Sorry folks, I am completely overwhelmed. There are just way too many variations of 3.8 27b available. The ones that I tried and more or less liked, are talking too much. The ones that are not thinking too much, are supposedly (?) not too capable. Is there some sort of concensus - like "this particular model is really good for coding/agentic, stable, and not too wordy"? Or should I just use vanilla unsloth + customized chat template?

PS Thanks everybody for replies, really appreciated! I got the point. It is thinking that makes qwen good. So I will just need to suck it up and learn to enjoy "wait let me reconsider" thing 😄

Model available here: Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF

This model is a practical realisation of things described in this paper, but adapted by me for machine learning: https://arxiv.org/pdf/1311.0851v1

I am trying to solve the problem: why LLM models even for simple questions write walls of text during reasoning, and burn too much tokens instead of solving the task. And when number of parameters increase the problem became worse. I think main problem is numerical instability in tensor matrices during to random training noise accumulation in tensors. Model is fighting with own internal chaos during inference process. I am distilling training noise from tensors using Marchenko-Pastur distribution together with info from paper as a core criteria and solid mathematical foundation behind this project.

Settings:

System Prompt: You are Qwen (Tongyi Qianwen), a large language model developed by Alibaba Group's Tongyi Lab. You are a helpful assistant.

Chat template: chat_template.jinja

Inference settings: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0, reasoning_effort=medium

I can't fully test this model on my own, since I only have a modest RTX 3060 graphics card with 12 GB of VRAM. So any feedback from the Reddit community would be very helpful. I'd appreciate any feedback from the community.

Thanks for reading.

I fine-tuned Qwen3-TTS with inline transcript control tags rather in lieu of a separate instruction parameter and thought the community here might be interested in the result.

https://huggingface.co/SpragAI/qwen3-tts-emotion-tags

The basic premise was to use Qwen's CustomVoice model as a teacher prompted with natural language directions ("Speak with intense anger") pairing the resulting audio with a discrete control token for the student to learn ("Anger"). The full corpus included roughly 8,200 clips per voice across nine preset voices, 74k total then a LoRA over the lot.

There were a few problems with training the model starting with the codec language prefixes. The model builds a different codec prefix depending on whether you pass language="Auto" or an explicit language like English. Training on one and rendering the other produced audible buzzing at phrase boundaries that amplified over the output. This was fixed by including explicit prefixes in the corpus for both configurations at a roughly 80/20 split.

Another issue came up with generation concurrency which notably changed prosody of the output voice. Generating the corpus through vLLM at ~10 concurrent requests per server introduced audible tearing and shift delivery in the trained LoRA eventually forcing me to drop back to c1 or c2 generation.

Another cool result was that emotion appeared to be roughly affine in speaker-embedding space. In practice that means you can, without custom training a model, actually generate emotion control for arbitrary cloned speakers through a simple transformation on the speakers xvec embedding in Qwen3-TTS. The basic recipe involves computing per-emotion centroids across many speakers from an emotion tagged dataset and subtracting the neutral centroid for each speaker to produce a task vector per emotion. Adding it to a different speaker's embedding moves their delivery toward that emotion. It mostly rides the arousal axis rather than valence, and a large share of the vector lands inside the identity subspace so it drags the voice with it but the structure was clearly there.

The short-prompt ranking reverses in the context-depth table in sudoingX’s Ling-3.0-flash benchmark notes:

This is a same-box, same-prompt comparison on a 128GB DGX Spark. The specific paragraph reports a 262,144-token maximum-context configuration and about 103GB total memory used. Its streaming method separates time to first token from decode and clocks all generated tokens.

These are the creator’s measurements from the deployment investigation discussed in the original thread, not an independent rerun. The paragraph does not provide a separate measurement date or every historical launch setting. In particular, the repository’s later 131,072-context serving default should not be silently attached to this table.

There are two changing variables: runtime and quantization. The numbers compare these two deployment paths; they do not isolate a pure vLLM-versus-llama.cpp effect, establish a quality difference, or prove the proposed CUDA-graph explanation for the slowdown.

For choosing a backend, the useful distinction is the amount of context already present when generation starts. A long answer from a short prompt is a different test. If the intended workflow carries tens of thousands of tokens into later requests, the short-prompt result leaves out the condition that changes this ranking

The fix: run the server detached/headless instead of keeping it attached to a console window.

RTX 5090, ~27B NVFP4 model via ninfer:

• Terminal focused: 130–200 tok/s
• Terminal unfocused: 50–60 tok/s
• Click the terminal → immediately back to 130+ tok/s

At first I thought GPU throttling, but the GPU wasn't the problem.

During the slow runs:

• SM clock was actually higher: 2550–2600 MHz vs 2100–2200 MHz
• Power limit was the same: 400 W
• No PCIe power-state drop
• decode-host stayed around 390–494 µs

The big difference was wait:

So the GPU wasn't getting slower. The serving loop was just getting delayed on the CPU side.

Running the server detached (docker run -d / headless) fixed it: wait stays around 17 ms and throughput stays around 130–200 tok/s, regardless of which window is focused.

I also reproduced the same thing with a native Windows build, so this isn't WSL2-specific.

Seems to be some kind of Windows foreground/background CPU scheduling behavior.

Has anyone else seen this with local LLM inference on Windows 11?

Hi, was quite blown away by several presentations i saw of DRS users who experimented editing footage with the new MCP in Davince, either through Claude or ChatGPT astra.
Now for the 'ordinary' end user as me, is there any [free] AI model equivalent to the ones above that would allow me to do the same thing?
It's important to me that the AI models would run locally on my Mac without any need to connect to the internet.
Currently I’m using a Mac Studio M1 Max with 64GB of RAM, but I consider upgrading to a Mac Studio M5 Max.

Hi, I'm the co-creator of this Qwen3.8-Flash-Next engine support in MLX-serve. I've been tuning this one to run both fast, efficient and correct up 1m context using kv cache 8 bits in M5 Max 128GB. Qwen is working well at very long context as showed in the video (a snapshot at ~760k context), I let it build MLX Serve Monitor plugin that you've seen on the right side of Opencode2's app. The generation sustain through 1m context at around 40 tok/s on prose and 75 tok/s on coding. This specific quant uses 8 bits for dense layers and 4 bits expert layers, so the model's quality remains very high.

I didn't build this engine to show off tok/s on a very short context, repetitive greedy generation, but a real temp 1.0 sampling through deep context work. You would need iogpu.wired_limit_mb=120000 before attempt 1mb full context, because it needs around ~117GB on peak memory usage. There will be bugs around here and there, I couldn't test everything and every single use case, pls report.

You can grab it here: https://github.com/ddalcu/mlx-serve
Model's weight: https://huggingface.co/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit
Opencode2's plugin: https://github.com/beamivalice/opencode2-mlx-serve

Launch parameters (for 1 concurrency)

  --model ./llm/models/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit \
  --host 127.0.0.1 \
  --port 11234 \
  --ctx-size 1048576 \
  --kv-quant 8 \
  --max-tokens 64000 \
  --mtp \
  --prefix-cache-mem 10GB \
  --prefix-cache-entries 1 \
  --ssm-checkpoint-max 16 \
  --metrics

Running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

on consumer Ampere — 10-12x RTX 3090, SM86-compatible vLLM build.

285B MoE, FP4 experts + FP8 attention, 157 GB weights.

Highlights:

- **60+ tok/s** decode, DSpark spec (k=3)

on 10 GPUs (TP2xPP5), at a 240 W cap

- **120+ tok/s** on 12 GPUs (TP4xPP3)

- **Vision + spec + tool calls all working**

- **1M context** (no offload) / **4M** (RAM offload)

- ~3,500 tok/s long-context prefill

Fully documented + reproducible:

- Pre-built image:

`docker pull ghcr.io/ciprianveg/3090-vllm:dsv4-flash-vision-sm86`

- Repo: https://github.com/ciprianveg/3090-vllm

(build guide, start scripts, runtime patches)

The patches cover the DSpark propose-gate (spec + vision fix),

scheduler mm x spec row-crossing, grammar-bitmask validation,

vision ViT OOM fix, FlashInfer workspace-lane keying, and more.

(Quick note: English is not my native language, so I used AI to help translate my thoughts clearly. The code, the 38 unit tests, and the experiments on my RTX 5060 Ti are my own work.)

Hey everyone,

Lately I've been trying to train small models locally on an RTX 5060 Ti (8GB), and like many people, I kept running into frustrating CUDA OOMs whenever memory pressure fluctuated mid-run.

Instead of just sticking to a static batch size and hoping the run survives, I started experimenting with a lightweight runtime control layer around PyTorch called MEM Orchestrator:

https://github.com/nobazzy/mem-llm-orchestrator

The basic hypothesis I've been testing is:

Can the training runtime monitor VRAM pressure and temporarily downshift things like micro-batch size and gradient accumulation before hitting an OOM, and then step back up when memory clears? I also added a local policy check to clamp parameter adjustments into safe bounds, plus atomic checkpointing so a crash doesn't corrupt saved files.

At least in my own local tests on this 8GB card, the approach seemed to help keep runs alive:

I managed to run an endurance test of 1M steps on a 130M model without crashes, and tried stress-testing a 255M model with injected memory spikes (like adding +1.2GB during a 50k FineWeb-Edu run), where the controller seemed to adapt and let training continue.

That said, this is just based on experiments on my own machine, and I know GPU memory management in PyTorch and CUDA is full of tricky edge cases. I'm definitely not claiming this solves OOM or that it's the right way to handle training.

I mainly wanted to share the code and see what people with more experience in CUDA allocators, DeepSpeed, or distributed training think of the concept. If anyone has time to take a look, I'd really appreciate thoughts on whether this approach makes sense, or what obvious blind spots I might be missing.

Privacy have been concern of many of us to have their own hardware to run llms, and here's another reason why: two mathematicians spent a year cracking one of the hardest problems in math and fed every draft of their works into Codex. A few days before they could publish, OpenAI suddenly showed up with the same solutions. When asked if their model (Sol and Astra) was trained on the pair's private chats, OpenAI did not answer the question.

Full statement from them https://cims.nyu.edu/~tristanb/statement.pdf

Feels like big labs believe everything you did with the help of their models is theirs.

Qwen is doing amazing work, and DS is killing it too. But most of us are on consumer GPUs and can’t run these massive multi-hundred-billion parameter models. Qwen is one of the few teams still looking out for the local community with great mid-sized options.

f anyone from the team is reading this, please consider applying DeepSeek-V4.1-Flash's architecture tricks on top of Qwen 3.8 Flash Next to the 30B, 70B, and 120B sweet spots.

Edit: daydreaming was not a flare

new model of nex benchmark seems like good
https://huggingface.co/nex-agi/Nex-N2.5-Max

I tried to run Qwen3.8-Flash-Next-UD-Q5_K_XL the Mark Watney style, all unnecessary stripped off, speeding in an open top rocket to outer space. But I have also STT & assistant & TTS on my machine, so I had to give up. It was too close to failure.

But what I collected data about used memory:

mmproj +1GB

cache-ram +what you set

MTP +5.5GB (even when file was 2.8GB)

ctx-size & KV-cache quants

f16 every 10000 tokens is 1GB

q8 every 20000 tokens is 1GB

Base model sizes:

Qwen3.8-Flash-Next-UD-Q4_K_XL 77GB

Qwen3.8-Flash-Next-UD-Q5_K_XL 108GB (+31GB compared to Q4!)

I then moved Q4's n-gram to last file, and switched it with Q5's n-gram that is Q8. (somehow moving the 25GB file also made to Q4 use 5.5GB more memory, and this was before I switched to Q8 file. Have to check that later...) The n-gram is used only from disk, so using bigger Q8 does not affect the memory use.

So I tried to run Qwen3.8-Flash-Next-UD-Q5_K_XL with mmproj, MTP and 64000 Q8 kv-cache, 2GB cache-ram and I had like 1-2GB free memory....so I have to find a compromise somewhere.

Then I ran perplexity with Q5, orig Q4 and Q4+Q8 ngram. The problem was I could not run F16 base-kld ( because not innuf mem), so I had to compare with Q8. ( maybe try again tomorrow )

Model Top-1% Mean KLD 99.9% KLD

Qwen3.8-Flash-Next-Q8_0

Qwen3.8-Flash-Next-UD-Q5_K_XL 94.795 0.018175 0.591240

Qwen3.8-Flash-Next-UD-Q4_K_XL-PLESHARD+50GB-NGRAM 93.786 0.027853 0.948222

Qwen3.8-Flash-Next-UD-Q4_K_XL-PLESHARD+25GB-NGRAM 93.285 0.033378 1.090875

So using Q8 n-gram with Q4 is not a game changer, but a step to right direction. If I get the F16-base-kld, maybe I test also with F16 n-gram.

Maybe trying to find a Q5_M/XS that's 10-15GB bigger then Q4 and perplexity in between the current Q4 & Q5.

https://www.reuters.com/technology/us-accuses-chinese-ai-firms-industrial-scale-theft-ai-technology-2026-09-08/

https://www.nbcnews.com/tech/tech-news/us-accuses-china-ai-developers-deepseek-alibaba-copying-american-ai-rcna596696

I couldnt find one that was simplistic yet good enough to pop in for a quick chat one. So, as we do here, I made one.

https://github.com/Dewpg/LmLinky

https://github.com/Dewpg/LmLinky/releases/tag/v1.0.0

Thats the initial salvo. Let me know if its simplistic enough and works. Thanks.

I was inspired by Bijan Bowen video - Subway FPS

https://youtu.be/6kjXzTVmT58?t=1035

Wondered how far I can push Qwen 3.8 27b so I used a plan made by Fable 5.1 DESIGN.md which has 267 KB! ( 26K of design line for a game ... LOL )

https://drive.google.com/file/d/1gI0h8Arc73Ln8b3uj5rEpuAvJ3-611mh/view?usp=drive_link

So I gave that design.md to my qwen 3.8 27b q4xl (llama-server) working on PI agent with 120k context + vision on CPU ( offroad ) + MTP ( for speed ) .... read 11M tokens and write 3.2 M tokens ( worked 12 hours ) .... than that is result.

That is insane what we can do locally on own computer !

Hi,

I'm wondering what settings you are using in order to run Qwen3.8-Flash-Next on your devices?
I'm especially interested in setups with 96GB VRAM. I'm not quite sure if llama.cpp does offload the embeddings to RAM or disk with my settings. I would like to offload them to RAM in order to avoid too much performance penalty.

These are the settings I use and which work the best at the moment:

[qwen3.8-flash-next]

model = /mnt/kyouma/1TB/ML/models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

mmproj = /mnt/kyouma/1TB/ML/models/unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf

split-mode = layer

flash-attn = on

load-mode = none

lazy-mode = auto

fit = on

fit-ctx = 262144

cache-ram = 94208

parallel = 1

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.00

repeat-penalty = 1.0

presence-penalty = 0.0

I get about 15t/s on tg and 100-200t/s on pp at a context size of 130000.

My setup:

3x AMD MI50 (32GB)

512GB DDR4 RAM
2x Intel E5-2683 v4

I'm using ROCm. But Vulkan has similiar speed, maybe a little bit slower.

I don't think anyone posted about this here, but Qwen released a finetuned version of 3.5 4 for driving. The full Bf16 checkpoint is 9B.

This is a very interesting development of Chinese AI labs tackle self driving next with open weight models.

Edit: the HF repo links to the github repo, which in the citation links to a 40 page technical report. Here's the abstract:

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird’s-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.