Our point of view on shipping AI — page 3 of 10.
- AI Engineering6 min
Multimodal RAG on EmbeddingGemma 2: The 6x Storage Cut Is a Text-Only Number
By Petru Popa ·EmbeddingGemma 2 shipped on 6 October under Apache 2.0, with Matryoshka truncation pitched as up to a 6x storage saving. The model card's own truncation table shows the 128-dimension setting costs 13.4 points on the multimodal benchmark and gives back most of the code-retrieval gain. For a multimodal index the safe floor is 256 dimensions, and the model only loads on a transformers release that came out the same afternoon.
Read - AI Engineering6 min
AI Agent Evaluation by Repetition: ThinkingBox's 20-of-20 Column Is Not What Its Own Tool Prints
By Petru Popa ·Microsoft's ThinkingBox ranks agents by how many of 507 business workflows they pass on all 20 attempts, and counts that literally. The aggregator it ships reports a different number that credits near misses, a pass usually means ten ticket fields matched exactly, and an open issue leaves custom agents able to reach the golden state.
Read - AI Engineering6 min
A Local LLM for Coding on a 12 GB Card: Strata Is MIT, the Weights Are Not, and It Serves One Request at a Time
By Petru Popa ·Strata runs Qwen3.8-Flash-Next, a 125B mixture-of-experts model, on a gaming PC with a 12 GB card, and serves it as an OpenAI- and Anthropic-compatible endpoint for coding agents. The repository is eleven days old, has shipped 40 releases, and supports only the latest. Its issue tracker, its batching docs and the model's license say who should run it and who should not.
Read - AI News5 min
To Run Kolibri Locally, the License Is Apache 2.0 and the Runtime Is a Plugin Pinned Below vLLM 0.30
By Petru Popa ·Aleph Alpha released Kolibri, a 78B-total, 3.46B-active English-German MoE, under Apache 2.0 on 3 October. The model card answers what the announcement left open: the only supported runtime is Aleph Alpha's own vLLM plugin, which refuses vLLM 0.30; the 1M context is extrapolated past a 262,144-token config; and the chat template reasons at high effort unless told otherwise.
Read - AI Engineering6 min
Local LLM on a Mac or a DGX Spark? In ds4's Own Benchmarks, the Reply Length Picks the Winner
By Petru Popa ·ds4, the local LLM engine from the creator of Redis, now has a community site with a clean benchmark table. That table makes the M5 Max look twice as fast as a DGX Spark at generating. Summing the CSVs behind that table gives a different answer for long prompts, and the crossover depends on how many tokens the model writes back.
Read - AI Engineering6 min
Clef as a Private LLM: The Open Weights Read 16,384 Tokens, Not the 65,536 Cloudflare Hosts
By Petru Popa ·Clef and Clef-flash are Cloudflare's new decision models, open-weight on Hugging Face under Apache 2.0. The hosted endpoint documents a 65,536-token context. The release code you would run on your own GPU defaults to 16,384 and silently cuts the end of the state to fit. Here is what the repository says about hardware, maturity, and which of the two to deploy.
Read - AI Engineering6 min
Self-Hosted LLM Serving, Split in Two: vLLM's Multi-Turn Cache Trusts Your Router, and Its Own Router Doesn't Check
By Petru Popa ·On 29 September vLLM published a practical guide to disaggregated serving on v0.30.0: prefill and decode on separate GPUs, tokenization on CPU boxes, and a bidirectional mode that reuses decode's KV cache across chat turns. That mode returns wrong output, silently, whenever a client edits the conversation history between turns. The docs leave detection to the router. The router vLLM ships keys reuse on a conversation ID and nothing else, and the proposed fix has had no maintainer review since June.
Read - AI Engineering6 min
Kumo Tabular Is a Tabular Foundation Model You Can Ship Commercially. Its First Place Is Self-Reported.
By Petru Popa ·NVIDIA released Kumo Tabular on 29 September, an openly licensed tabular foundation model that claims first place on TabArena and BeyondArena. Neither leaderboard lists it yet, its benchmark runs used ensemble and precision settings a plain library call does not apply, and the model card's pip command installs an empty placeholder. The license is the real news.
Read - AI Engineering6 min
Zero-Shot Classification on a Local LLM: Jeff Accepts 255 Options and Picks From 26
By Petru Popa ·Jeff is a day-old MIT repository that fine-tunes Qwen3.5 and Gemma 4 into single-pass zero-shot classifiers you run locally. The server accepts up to 255 options. A third-party issue shows the Qwen models scoring zero whenever the right answer sits past option 26, and no builder in the published training recipe that we read produces a question that wide.
Read - AI News6 min
Ember-1 and LLM Cost Optimization: Fireworks' Own Table Adds Up to 23%, Not Half
By Petru Popa ·Ember-1 is Kimi K3 retrained to emit shorter reasoning, priced identically per token, so every saving is a token saving. The announcement says half the tokens. Summing the dollar column of Fireworks' own five-benchmark table gives roughly 23%, and the model is a serverless research preview with no weights, no fine-tuning and no on-demand option listed.
Read - AI Engineering6 min
AI Watermarking in vLLM Costs No Throughput. Four Settings Decide Whether Your Output Carries It.
By Petru Popa ·vLLM 0.30.0 ships distortion-free text watermarking, and its blog shows no throughput cost on an H100. The shipped code tells a narrower story: greedy requests are never marked, any caller can opt out per request, speculative decoding skips the repetition guard, and the dual-key detector defaults disagree with the generator.
Read - AI Engineering6 min
vLLM vs llama.cpp on a Mac: vllm-metal Wins the First Token, Not the Finish
By Petru Popa ·vllm-metal brings vLLM's scheduler and paged KV cache to Apple Silicon, and its launch post shows it beating llama.cpp and oMLX under concurrent agent load. The benchmark repository behind that post holds a different, footprint-matched run. It agrees on first-token latency and disagrees on how fast requests finish.
Read