AI Engineering — page 1 of 4
- AI Engineering5 min
Coding Agent Benchmark: Copilot Fusion Is Two Points Behind Claude Code, and Three Benchmarks Disagree About Why
By Petru Popa ·GitHub says Copilot Fusion Max lands within two points of Claude Code with Opus 5.5 at about half the cost. Artificial Analysis's own table shows the two points are an average of a win of 8 and losses of 5 and 9, so the headline depends on your task mix.
Read - AI Engineering5 min
Multi Agent LLM Teams: Five Qwen Passes Beat Fable 5 on Ten Problems, and Solve 47 of 100 With Thinking Off
By Petru Popa ·Memdoor rebuilt the GVS5H multi-agent loop on Qwen3.8-27B and scored 9.8 of 10 on the authors' first ten LiveCodeBench Hard problems, against 8.0 for Fable 5. Its own larger bench, run with thinking off, solves 47 of 100. The ten-problem table replicates the method. It does not rank the team against a frontier model.
Read - AI Engineering5 min
AI Agent Guardrails: Copilot Assisted Permissions Is a Third Mode Your Allow-All Lock Does Not Cover
By Petru Popa ·GitHub made assisted permissions generally available on 26 October: a language model approves the Copilot requests it judges safe. The managed-settings reference shows that locking down allow-all leaves it switched on, and no page we read publishes how often the model is wrong.
Read - AI Engineering5 min
Prefill-Decode Disaggregation in SGLang Leaked KV Cache Across Requests. No Scanner Will Tell You to Upgrade.
By Petru Popa ·SGLang's own write-up says default prefill-decode disaggregation deployments carried a silent cross-request KV cache race from the v0.4.x line until v0.5.21. The fixes were not backported, the integrity checksum ships off by default, and as of this run neither the GitHub Advisory Database nor OSV lists the issue, so a pinned older SGLang still scans clean.
Read - AI Engineering6 min
A Churn Prediction Model You Never Train: Kumo Relational Wins the RelBench Average and Loses All Three Churn Tasks
By Petru Popa ·NVIDIA announced Kumo Relational on 21 October as the first few-shot foundation model to beat supervised machine learning on relational benchmarks. Stanford's RelBench board confirms it leads every other in-context entry and trails only one trained system on average. It also shows the model losing all three churn tasks to trained models, and Prior Labs' frozen-cutoff re-run of the underlying KumoRFM-2 drops those churn scores by another one to two AUROC points. Where the claim holds, and where it flips.
Read - AI Engineering6 min
Decision Model d1-3B Leads Its Size Class on Public Tests. On the Held-Out Split It Ranks 53rd.
By Petru Popa ·Liquid AI released d1-3B on 7 October as the best decision model under 10B parameters, citing a self-scored 48.57 on Decision Index 0.2.1. The public leaderboard had moved to edition 0.3 the day before, and that edition scores mostly on private tests. On it, d1-3B places 53rd of 117, inside the tie band with Decider 4B, and 76th on tasks from new domains. The public lead is real. It does not travel.
Read - AI Engineering6 min
Multimodal RAG on EmbeddingGemma 2: The 6x Storage Cut Is a Text-Only Number
By Petru Popa ·EmbeddingGemma 2 shipped on 6 October under Apache 2.0, with Matryoshka truncation pitched as up to a 6x storage saving. The model card's own truncation table shows the 128-dimension setting costs 13.4 points on the multimodal benchmark and gives back most of the code-retrieval gain. For a multimodal index the safe floor is 256 dimensions, and the model only loads on a transformers release that came out the same afternoon.
Read - AI Engineering6 min
AI Agent Evaluation by Repetition: ThinkingBox's 20-of-20 Column Is Not What Its Own Tool Prints
By Petru Popa ·Microsoft's ThinkingBox ranks agents by how many of 507 business workflows they pass on all 20 attempts, and counts that literally. The aggregator it ships reports a different number that credits near misses, a pass usually means ten ticket fields matched exactly, and an open issue leaves custom agents able to reach the golden state.
Read - AI Engineering6 min
A Local LLM for Coding on a 12 GB Card: Strata Is MIT, the Weights Are Not, and It Serves One Request at a Time
By Petru Popa ·Strata runs Qwen3.8-Flash-Next, a 125B mixture-of-experts model, on a gaming PC with a 12 GB card, and serves it as an OpenAI- and Anthropic-compatible endpoint for coding agents. The repository is eleven days old, has shipped 40 releases, and supports only the latest. Its issue tracker, its batching docs and the model's license say who should run it and who should not.
Read - AI Engineering6 min
Local LLM on a Mac or a DGX Spark? In ds4's Own Benchmarks, the Reply Length Picks the Winner
By Petru Popa ·ds4, the local LLM engine from the creator of Redis, now has a community site with a clean benchmark table. That table makes the M5 Max look twice as fast as a DGX Spark at generating. Summing the CSVs behind that table gives a different answer for long prompts, and the crossover depends on how many tokens the model writes back.
Read - AI Engineering6 min
Clef as a Private LLM: The Open Weights Read 16,384 Tokens, Not the 65,536 Cloudflare Hosts
By Petru Popa ·Clef and Clef-flash are Cloudflare's new decision models, open-weight on Hugging Face under Apache 2.0. The hosted endpoint documents a 65,536-token context. The release code you would run on your own GPU defaults to 16,384 and silently cuts the end of the state to fit. Here is what the repository says about hardware, maturity, and which of the two to deploy.
Read - AI Engineering6 min
Self-Hosted LLM Serving, Split in Two: vLLM's Multi-Turn Cache Trusts Your Router, and Its Own Router Doesn't Check
By Petru Popa ·On 29 September vLLM published a practical guide to disaggregated serving on v0.30.0: prefill and decode on separate GPUs, tokenization on CPU boxes, and a bidirectional mode that reuses decode's KV cache across chat turns. That mode returns wrong output, silently, whenever a client edits the conversation history between turns. The docs leave detection to the router. The router vLLM ships keys reuse on a conversation ID and nothing else, and the proposed fix has had no maintainer review since June.
Read