Running 35B MoEs on a 6GB Laptop GPU

I pointed four 35B models at an empty folder and told them to build Tinder for horses. That prompt is a shoutout to Fireship, the YouTube channel, not a product I am trying to ship. The laptop has a 6GB GPU, and the folder I would keep came from the base model after I turned its planner off.

A 21GB Model Does Not Fit in 6GB

Mixture-of-experts models like Qwen 3.6 35B A3B are the reason this was worth trying on a laptop at all. The file holds about 35 billion parameters, and a given token only activates about 3 billion of them, which is how you get a large model’s routing while paying a small model’s math. The laptop only cares how much of that file is resident.

The machine is an Intel i7-9750H, six physical cores and twelve threads, a GTX 1660 Ti Mobile with 6GB of VRAM, and 32GB of DDR4, on Pop!_OS. A 4-bit quant of this model is about 21GB, so the GPU cannot hold it, and a 128k context with an ordinary FP16 KV cache wants roughly another 16GB. Ask for both and the session dies with Unterminated string in JSON, a CUDA out-of-memory, or a kernel panic, often in the middle of a tool call the agent was about to trust.

The usual workaround is --n-gpu-layers with a small number, which parks some layers on the CPU and the rest on the card. Every token then pays a PCIe trip. On a 1660 Ti in a laptop chassis, that trip is the session. I wanted the split the architecture already implies: sparse expert weights in system RAM, where 32GB is enough, and the dense tensors on the GPU, where 6GB is almost enough if the cache is not greedy.

For a harness, I decided on Codex with environmental overrides to point at my local llama server. The bar was an autonomous loop that installs packages and leaves a folder passing npm run build, on a machine that never sees the whole model at once.

The Experts Stay in RAM

The flag that matches the architecture is --n-cpu-moe 36. All 36 layers of sparse expert weights stay in system RAM for the session. --n-gpu-layers 36 still runs, but the GPU only receives the dense remainder: attention, embeddings, and normalization. That allocation landed around 3.7GB. The 1660 Ti had space left, which is the result that made the rest of the experiment possible.

Context is the second wall, and coding agents hit it on purpose. The model reads a file, writes a file, and then reads the compiler. A few Vite failures and you are past 16k tokens. A 128k window stored as FP16 keys and values will not sit beside a 21GB quant on this box. --cache-type-k q4_0 and --cache-type-v q4_0 brought that 128,000-token cache down to about 4.2GB. I expected the quant to chew up closing tags and variable names. On this workload the broken syntax came from the model, and the cache quant held up.

--no-mmap --mlock pins the weights in physical RAM so the kernel does not page them out mid-generation. --parallel 1 because this is one agent. The command that held for the scored runs:

llama-server \
  --n-cpu-moe 36 \
  --n-gpu-layers 36 \
  --ctx-size 131072 \
  --parallel 1 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --threads 6 \
  --threads-batch 12 \
  --flash-attn on \
  --ubatch-size 1024 \
  --no-mmap --mlock

Offloading a handful of whole layers would have been easier to explain and worse to wait on. The model already separates experts from attention. These flags stop moving both of them across the bus on every token.

Six Cores, Not Twelve Threads

After the model fit, I tuned throughput without giving up the 128k buffer. The starting point was the config you get when you stop at the defaults: --threads 8, no ubatch, flash attention left alone. Generation sat at 10.13 tokens per second, and a thousand tokens took 98.7 seconds. Hyperthreading was thrashing the cache. Those extra threads shared the physical cores and fought over the same lines.

Matching the flag to the six physical cores and setting --ubatch-size 1024 moved generation to 15.39 tok/s. A 400-token sample finished in 26.0 seconds, about 52% faster than the uncalibrated run. Flash attention with --threads-batch 12 went a step further, to 16.59 tok/s and 24.1 seconds for those 400 tokens, about 64% over the baseline. Prompt processing can use the extra threads while decode sits on the physical cores, and that split was the gain.

Speculative n-gram matching looked like the obvious next knob. ngram-simple needs no extra VRAM, so I tried it. Generation dropped to 15.79 tok/s, roughly 5% under the flash-attention config, and the 400-token sample took 25.3 seconds. The draft table on the CPU cost more than it saved, so I left it off.

Sixteen tokens a second will not impress a datacenter chart. An agent that spends much of its wall clock inside npm can live with it, which is why I stopped tuning once the threads matched the chip.

Build Tinder, but for Horses

Every model got the same prompt and an empty directory. The prompt is a shoutout to Fireship, the YouTube channel. Their go-to fake product is always Tinder for Horses, and so I decided to AI-slop my clone of that. Anyone who watches that channel will recognize the joke, and the spec still has enough surface to grade: swipe, like, and nope, with drag and with the arrow keys (left for nope, right for like, up for a super-like). Horse profiles with usable photo URLs, names, breeds, disciplines, age, height in hands, and a bio. Some audio feedback, even if it is only a synthesized neigh. A matches list that opens a chat of scripted horse banter. Filters for breed, discipline, and age. An equestrian palette and a dark mode toggle. Install whatever you import. npm run build has to pass.

That is a wide surface for a model whose weights are mostly in RAM. It is also an easier test than the gift-card slice I ran on the Strix Halo, and those two results should stay separate. The earlier task had an existing point of sale, a Prisma migration, and a reviewer that did not write the code. This one is a blank folder and a green production build. A model can look finished here and still invent a relation the moment the repo already has a schema. I wanted the blank-folder question answered on hardware that lives in a backpack.

Why These Four

Same server flags, same prompt, four 35B A3B variants. The base run had fewer tools. The weights were the variable.

Qwen/Qwen3.6-35B-A3B is the official release. Qwen’s card sells this size as agentic coding you can actually run: frontend workflows and repository-level reasoning, with SWE-bench Verified at 73.4 and Terminal-Bench 2.0 at 51.5. I included it so the other models had a checkpoint to beat, and because their own writeup is the claim this whole size class is making.

unsloth/Qwen-AgentWorld-35B-A3B is Qwen’s AgentWorld weights. The card describes a language world model. You give it an action and a history, and it predicts the next environment state, across tool calls, a terminal, software engineering tasks, and the web. They also say that training carries over to multi-turn tool use. I wanted the model whose objective is “what does the environment do next,” because that is most of a coding-agent loop.

Jackrong/Qwopus3.6-35B-Coder-MTP reads like a product spec. Thinking stays off. It is tuned to read a file, choose a tool, edit, run the tests, and answer the error without a long reasoning trace on every step. The MTP build adds multi-token prediction for faster local decode. On their thinking-off SWE-bench run, 300 cases at Q5_K_M, they report 62.4%. This is the one that should have been fastest, and it was.

peculiar-ragdoll/Tiel-Coder-35B-A3B is the card that says to pick it for work. It is Ornith-1.5 re-quantized with their own imatrix and the Sharp chat template, and they are clear that the change is the quant and the template. On a 25-problem SWE-bench-Live slice they report 12 fixes, which they line up with Opus 4.6 medium, and they say it holds the best multi-turn conversation of the local models they have measured. They also say it is bad at trivia. I wanted the model that claims it can stay inside a long coding session.

What Came Out of the Folder

All four left a tree that passed npm run build, including the one I expected to die in dependency hell. The differences were in the app, the clock, and how much supervision the loop needed.

Qwopus led raw generation at 14.91 t/s, peaking at 19.2, which is the MTP draft doing the extra work. The base model ingested prompts fastest: 51.95 t/s, with bursts up to 120, and its sustained generation of 12.36 t/s sat just ahead of AgentWorld at 12.03. Tiel averaged 9.07 t/s and spent 50,651 tokens. That session is the D+.

Qwen3.6-35B-A3B

Neighbors is the Tinder-for-horses app, and it is the one I would ship. 33.9 minutes, 12.4 t/s generation, 52.0 t/s prefill. With the planning tools removed, it ran bash in a straight line, caught its own coupling bug while the components were still open, and did not come back for a compiler pass.

How it worked

• It caught the over-coupling between CardStack and SwipeCard mid-build and refactored that hierarchy before moving on.
• Prefill led the four at 52.0 t/s, peaking at 120 t/s, and the shell commands ran without a stall.
• The implementation finished in 33.9 minutes. There was no follow-up pass to debug the compiler.
• Without --disable goals, it gets trapped apologizing for update_goal, a planning status the CLI rejects.

What it built

• 668 lines of CSS: meadow green #2f5d3a, custom slider tracks, parchment cards, and dark mode.
• It expected Unsplash CDN 404s. onError swaps in a gradient and a breed emoji.
• Web Audio API: melodic arpeggios for Super Likes, a match fanfare, and chimes. No audio URL in the page.
• Breed and discipline filters are standard HTML <select> dropdowns. AgentWorld used clickable pills.

Qwen-AgentWorld-35B-A3B

This is the run that needed nothing extra on the harness. 36.1 minutes, 12.0 t/s generation. Tailwind v4, a saddle-leather palette, live Unsplash images, and horse banter in the chat.

How it worked

• It handled the multi-tool schema with no guardrails and no custom prompt.
• Tailwind v4 failed in PostCSS. It read the raw stack trace, installed @tailwindcss/postcss, patched postcss.config.js, and recovered on its own.
• It scaffolded in the workspace root, in an ordinary Vite layout.
• Generation was 12.0 t/s over 36.1 minutes. The base model finished in 33.9, Qwopus in 29.5.

What it built

• Tailwind, with saddle leather, gold trim, and meadow green.
• Hand-selected direct Unsplash equestrian URLs. They loaded in the headless browser, and none were broken.
• Pill-tag multi-selects, a dual-range age slider, and a chat drawer of simulated banter.
• Sound is badges and simple synthesized tones. The base model’s arpeggios, fanfare, and chimes do more.

Qwopus3.6-35B-Coder-MTP

29.5 minutes, 12,104 tokens, 14.91 t/s average, peaks over 19.2. Distilled on Claude Opus reasoning traces, with multi-token prediction. It drafted the app 3.5x faster than Tiel-Coder: 29.5 minutes against 104.3.

How it worked

• Highest sustained generation in the table, 14.91 t/s, from MTP speculative drafting. The session was the short one, 29.5 minutes.
• 12,104 tokens for the whole app, less than a quarter of Tiel’s 50,651.
• It did not fall into circular re-reading. It executed and moved on.
• The app landed in app-qwopus-coder/tinder-horses/, not the workspace root.

What it built

• 15 files. Custom pointer-event physics in useSwiping.js, and audio oscillators.
• Lean React, without a pile of extra dependencies.
• Parameterized Unsplash queries hit CDN rate limits. The first card in the stack was a blank placeholder.

Tiel-Coder-35B-A3B

HorseRnR. 104.3 minutes, 50,651 tokens, 9.07 t/s. That is a D+. About 20 minutes went to validating external APIs and correcting card indexes. The session stayed in that loop long after the fix.

How it worked

• It caught an inverted card-index rendering bug and audited the stack state without being asked.
• It wrote curl scripts against Unsplash, recorded HTTP 503s, then queried the Wikimedia Commons API and kept URLs that resolved.
• It then re-read and re-verified files that already worked. 50,651 tokens, nearly two hours.
• Generation averaged 9.07 t/s. The reasoning context is the overhead on decode.

What it built

• Framer Motion: gesture physics, card velocity flicking, and multi-oscillator audio.
• Wikimedia Commons links, which loaded without Unsplash hotlink throttling.
• The layout is dense and text-heavy next to the cards from the base model and from AgentWorld.

The Base Model Apologized in a Loop

The first attempt with raw Qwen/Qwen3.6-35B-A3B lasted 3.9 minutes. It called update_goal with status "in_progress", which is not in the enum the Rust CLI accepts, so the harness rejected the call. The model apologized, said it would switch to update_plan, and sent the same invalid call again, eight times, until the session dropped.

That failure gets cited as the reason to buy an agentic fine-tune. On this run the base model could still start a React app. It could not respect a planning tool’s schema, and tool-edge training is a real product for exactly that miss. What I had in front of me was one bad enum and a retry loop.

I changed two things and started over. --disable goals removed the planning tools and left exec. An AGENTS.md in the workspace told it to skip meta-planning and go straight at files and npm. I did not add a coding system prompt, a different sampler, or a draft model.

The second run took 33.9 minutes. It scaffolded Neighbors, wrote the image fallback before any URL had 404’d, noticed the card components were tangled, split them, installed dependencies, and produced a clean Vite build. With the planner gone, shell use was as steady as the fine-tunes. The missing piece on this harness was a tool the base weights were going to misuse.

I Would Ship the Base Model’s Folder

If I had to put one of these four apps in front of a person, I would put Neighbors there. The theme lives in the CSS, and the photos have a fallback. AgentWorld is what I would start from if the harness had to stay stock, because it got through a messy tool schema without a flag. Qwopus is what I reach for when I want another pass before dinner. Tiel checked its URLs and then spent 104 minutes and 50,651 tokens on a denser page. That is a D+.

A blank Vite app is still not the gift-card feature, and 16 tokens a second does not retire a Max plan when a cloud model plans and reviews the work. On this laptop the pieces that decided the outcome were --n-cpu-moe 36, a 4-bit KV cache, six physical threads, and a harness that did not offer update_goal. The base Qwen 3.6 35B A3B was the folder I would keep.