Our point of view on shipping AI — page 2 of 8.
- AI Engineering6 min
Local LLM on a Mac or a DGX Spark? In ds4's Own Benchmarks, the Reply Length Picks the Winner
By Petru Popa ·ds4, the local LLM engine from the creator of Redis, now has a community site with a clean benchmark table. That table makes the M5 Max look twice as fast as a DGX Spark at generating. Summing the CSVs behind that table gives a different answer for long prompts, and the crossover depends on how many tokens the model writes back.
Read - AI Engineering6 min
Clef as a Private LLM: The Open Weights Read 16,384 Tokens, Not the 65,536 Cloudflare Hosts
By Petru Popa ·Clef and Clef-flash are Cloudflare's new decision models, open-weight on Hugging Face under Apache 2.0. The hosted endpoint documents a 65,536-token context. The release code you would run on your own GPU defaults to 16,384 and silently cuts the end of the state to fit. Here is what the repository says about hardware, maturity, and which of the two to deploy.
Read - AI Engineering6 min
Self-Hosted LLM Serving, Split in Two: vLLM's Multi-Turn Cache Trusts Your Router, and Its Own Router Doesn't Check
By Petru Popa ·On 29 September vLLM published a practical guide to disaggregated serving on v0.30.0: prefill and decode on separate GPUs, tokenization on CPU boxes, and a bidirectional mode that reuses decode's KV cache across chat turns. That mode returns wrong output, silently, whenever a client edits the conversation history between turns. The docs leave detection to the router. The router vLLM ships keys reuse on a conversation ID and nothing else, and the proposed fix has had no maintainer review since June.
Read - AI Engineering6 min
Kumo Tabular Is a Tabular Foundation Model You Can Ship Commercially. Its First Place Is Self-Reported.
By Petru Popa ·NVIDIA released Kumo Tabular on 29 September, an openly licensed tabular foundation model that claims first place on TabArena and BeyondArena. Neither leaderboard lists it yet, its benchmark runs used ensemble and precision settings a plain library call does not apply, and the model card's pip command installs an empty placeholder. The license is the real news.
Read - AI Engineering6 min
Zero-Shot Classification on a Local LLM: Jeff Accepts 255 Options and Picks From 26
By Petru Popa ·Jeff is a day-old MIT repository that fine-tunes Qwen3.5 and Gemma 4 into single-pass zero-shot classifiers you run locally. The server accepts up to 255 options. A third-party issue shows the Qwen models scoring zero whenever the right answer sits past option 26, and no builder in the published training recipe that we read produces a question that wide.
Read - AI News6 min
Ember-1 and LLM Cost Optimization: Fireworks' Own Table Adds Up to 23%, Not Half
By Petru Popa ·Ember-1 is Kimi K3 retrained to emit shorter reasoning, priced identically per token, so every saving is a token saving. The announcement says half the tokens. Summing the dollar column of Fireworks' own five-benchmark table gives roughly 23%, and the model is a serverless research preview with no weights, no fine-tuning and no on-demand option listed.
Read - AI Engineering6 min
AI Watermarking in vLLM Costs No Throughput. Four Settings Decide Whether Your Output Carries It.
By Petru Popa ·vLLM 0.30.0 ships distortion-free text watermarking, and its blog shows no throughput cost on an H100. The shipped code tells a narrower story: greedy requests are never marked, any caller can opt out per request, speculative decoding skips the repetition guard, and the dual-key detector defaults disagree with the generator.
Read - AI Engineering6 min
vLLM vs llama.cpp on a Mac: vllm-metal Wins the First Token, Not the Finish
By Petru Popa ·vllm-metal brings vLLM's scheduler and paged KV cache to Apple Silicon, and its launch post shows it beating llama.cpp and oMLX under concurrent agent load. The benchmark repository behind that post holds a different, footprint-matched run. It agrees on first-token latency and disagrees on how fast requests finish.
Read - Dev Tools5 min
Your GitHub Copilot Settings Change Meaning on October 22. Unconfigured Will Mean On.
By Petru Popa ·GitHub added a global default policy for Copilot Business and Enterprise features on September 24 and starts enforcing it on October 22. The changelog lists three choices without saying which one is preselected. The docs do: enabled. Every eligible feature you never configured, including MCP servers in Copilot, turns on unless you decide otherwise in the next four weeks.
Read - Dev Tools6 min
Node 20 EOL Reached GitHub Actions on September 23. Your node20 Actions Kept Passing.
By Petru Popa ·Node 20 went end of life on 30 April. GitHub Actions removed it on 23 September and killed the opt-out. Workflows calling node20 actions did not fail: the runner rewrites the declaration to Node 24 and leaves one warning at the end of the job. Over nine thousand public action.yml files still declare node20. The date on a deprecation notice is not the day your pipeline changes.
Read - AI Engineering6 min
To Run an LLM Locally in Transformers, the Metal Kernel Has to Be There
By Petru Popa ·Hugging Face shipped a path that runs llama.cpp quants inside transformers without dequantizing them. The packed path needs a Hub kernel whose only published driver family is metal, it forces float32, it covers Qwen3.5 and nothing else, and the fallback when the kernel is absent costs you the whole quantization.
Read - Dev Tools6 min
Copilot Custom Agents Pinned to Retiring Models: Aim at the LTS, Not the Suggested Alternative
By Petru Popa ·GitHub will retire GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5 mini, Gemini 3.7 Flash and Grok 4.5 from every Copilot surface on 19 October. Over a thousand public custom agent files pin one of them. The docs table that tells you where to migrate still points four retired models at GPT-5.5, and the replacements it names have historically lasted months. The one target with a written commitment is the LTS model.
Read