A Local LLM for Coding on a 12 GB Card: Strata Is MIT, the Weights Are Not, and It Serves One Request at a Time

Strata puts Qwen3.8-Flash-Next on a gaming PC. The model card lists it at 125B parameters with 6B active per token, plus a 51B n-gram embedding table, and Strata runs it on an NVIDIA or AMD card with 12 GB of VRAM and 32 GB of RAM or more. It exposes the model at 127.0.0.1:8080 as an OpenAI-compatible, Anthropic-compatible and Responses-API endpoint, so Claude Code, Codex CLI and other coding agents can point at it.

The question for a technical lead is different: can this become the team's private coding model? The repository says not yet, for three reasons the README does not lead with.

What it does

It keeps the most-used experts in VRAM, holds all of them in RAM, computes the rest on the CPU, and leaves the n-gram lookup table on the SSD. A small draft head guesses tokens and the large model verifies them. The main weights are ISTA-DASLab's GGUF quantizations, from a 66.4 GB Q2_0 up to IQ3_S, and the installer also offers a Coder variant with half the experts removed that fits 32 GB of RAM.

The README's measured numbers come from two machines. On an RTX 5070 with 12 GB, Q2_0 writes 94 tokens a second and IQ3_S writes 53. On an RX 9070 XT, Q2_0 writes 60. Its RTX 3090 figure is a projection, and other cards point to community reports. The 0.1.39 release notes say plainly that the maintainers have one GPU.

One request at a time

By default Strata answers one request and queues the rest. docs/BATCHING.md documents opt-in slots and measures them on that RTX 5070. With four simultaneous requests and the default, the fourth waits 11.2 seconds for its first token. With four slots it waits 1.8 seconds, but total throughput falls from 70.7 to 63.1 tokens a second, and the 0.1.39 notes put the cost to a request running alone at 11 percent with two slots and 22 with four, because each slot takes 0.56 GiB away from the expert cache. Setup therefore recommends slots only where most experts fit in VRAM, which rules out any single 12 or 16 GB card.

For a solo developer the default is the right setting. For a team, it means one card per concurrent agent, or a multi-GPU build: the same doc shows a four-card, 16 GB-each split serving eight requests at 360 tokens a second in total.

Maturity: forty releases in its first eleven days

The repository was created on 24 September. From v0.1.0 that day to v0.1.39 on 4 October it shipped 40 tagged releases and more than 800 commits, 520 of them from the maintainer's account. On 5 October it had more than 11,500 stars and 1,005 forks against 30 merged pull requests, 213 open ones, and 99 open issues. SECURITY.md supports only the latest release; fixes do not go to older ones.

The tracker shows what that pace hides. Issue 606, opened on 3 October against 0.1.35 on an RTX 5090, reports that after one long request every later reply came back as a single repeated character, while the health endpoint still reported the model as loaded. 0.1.39 clamped the two quantizers the reporter traced the non-finite values to. The same day a contributor found a third quantizer missing the same clamp; the pull request fixing it was still open when we checked, and so is the issue. That is the silent degradation the PoC budget skips: every health check green, every answer wrong.

Security followed the same path. Until 0.1.38 on 3 October, nothing stopped a web page from another site sending requests to a server running without an API key; that release added the Host and Origin checks that now refuse them. Private vulnerability reporting was switched on with 0.1.39, after a user asked for audits in issue 544, and SECURITY.md states that no outside audit has happened. The README's own install path asks the user to paste a setup instruction into their coding agent, which then installs and starts a server on their machine.

The pattern matches Splash, the Mac engine we read at two days old: a real speed result, a young repository, and adoption that outruns the issue tracker.

The license is not MIT

Strata's code is MIT. The weights are not. The Qwen model card declares the Qwen Community License 1.0, and the ISTA-DASLab quantizations are tagged Apache 2.0 on the Hub while their own card says they inherit the base model's license. Even inside Strata's repository, one data file is under the Qwen license, per HOW_IT_WORKS.md.

That license has a clause aimed at this exact use. A licensee whose business includes a model-as-a-service offering or an AI work assistant, which it defines as a product mainly for AI-assisted coding or office work, needs a separate license from Qwen before commercial use. Internal use is exempt only while neither the model, its outputs nor its capabilities reach a third party. An in-house coding agent for your own engineers fits the exemption. An agency exposing the endpoint to clients, or a startup shipping a coding assistant, does not.

Who should not use it

• A team wanting one shared endpoint on one consumer card.
• Anyone who cannot re-test on every release, since only the latest is supported.
• A company that sells hosted inference or a coding product, until it has read clause 2.

The claim that could be wrong: through the end of this quarter, Strata stays a single-developer workstation engine rather than a team server. A supported release line older than latest, a published outside audit, or setup recommending slots on a single 16 GB card would show it wrong. If the goal is a shared box for a team, that is the sizing problem our private LLM deployment work starts from.

Check an install today

On any machine already running Strata without a key, confirm the cross-site check is in place. A 403 is the expected answer from 0.1.38 onward; a 200 means a web page could drive the model.

jq -nc --arg r user --arg c ping '{messages:[{role:$r,content:$c}],max_tokens:1}' | curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Origin: https://example.com' -H 'Content-Type: application/json' --data @- http://127.0.0.1:8080/v1/chat/completions

Then add a canary for issue 606 to whatever monitors the box: after each long job, send one short fixed prompt and alert when the reply is a single character repeated.