AI Engineering — page 1 of 2
- AI Engineering6 min
Ai2 Audited 16 LLM Benchmarks. Nearly Half the Safety Questions Score Reasoning.
By Petru Popa ·Ai2 published a per-question fit of 16 LLM benchmarks and shipped the item table as a dataset. Grouping its 29,574 rows by benchmark shows that BBQ and WMDP, both filed under safety, discriminate on the reasoning dimension instead.
Read - AI Engineering6 min
A Self-Hosted LLM Has One Context Pool and Four Users By Default
By Petru Popa ·llama.cpp v0.4.0 adds a per-slot context limit to its server. Reading the server sources shows the cap is a minimum over three numbers, and that omitting -c and writing -c 0 no longer mean the same thing.
Read - AI Engineering6 min
Hugging Face Shipped 207 WebGPU Kernels. Your Lockfile Pins None of Them.
By Petru Popa ·The npm package is one preview version old. The 207 kernel repositories it loads at runtime carry no git tags, and 63 of them moved on day one. The speed number is the least interesting thing here.
Read - AI Engineering6 min
Confidential AI Moved the Trust Boundary. It Did Not Close It.
By Petru Popa ·Google DeepMind, MLCommons, AVERI, OpenMined and Singapore AISI ran a proprietary model against a private benchmark inside a hardware-encrypted enclave, so neither side saw the other's asset. The announcement page stops there. The technical report discloses that not all inference code was allowlisted, the enclave builds are not independently reproducible, and Google signs and verifies its own attestation.
Read - AI Engineering6 min
MXFP4 Quantization Did Not Improve the Model. A Second Teacher Did.
By Petru Popa ·A Hugging Face write-up reports a 4-bit MXFP4 model beating its own full-precision original on 7 of 9 benchmarks. The paper underneath it says the teacher was already MXFP4, that every number is a single run, and that the baseline which would settle the claim was never run.
Read - AI Engineering6 min
Agentic Memory Helps Until You Cap the Retrieval
By Petru Popa ·IBM Research reports gains of up to 16.1 points from feeding an agent guidelines mined from its own past runs. The technical report underneath contains a configuration of the same system that scores below the no-memory baseline, and the library that ships defaults to a retrieval backend no reported configuration used.
Read - AI Engineering6 min
Word Error Rate Fell 3.5 Points When Someone Fixed the Transcript
By Petru Popa ·Artificial Analysis hand-corrected 628 VoxPopuli clips and average model word error rate dropped 3.5 points with no retraining. A new set of probes explains the other half of that result: the models scoring best had learned to reproduce the errors.
Read - AI Engineering5 min
One NDCG Point for Twelve Times the Index
By Petru Popa ·Late interaction retrieval lands in Sentence Transformers 6.0 with a controlled benchmark behind it: same backbone, same data, one vector per token instead of one per document. The margin is about one NDCG point. The release notes show what that point costs in storage, and the model table shows which knob actually moves it.
Read - AI Engineering6 min
LLM Evaluation Is Three Different Jobs
By Petru Popa ·Picking a model, catching a regression, and watching production are three different measurement problems that share one name. Teams get into trouble when the artifact from one gets used to answer another — and the LLM-as-judge shortcut has failure modes with numbers attached.
Read - AI Engineering6 min
LLM Observability Is Mostly a Schema Problem
By Petru Popa ·Buying a dashboard is the easy half. The hard half is whether your traces carry the fields you will need at two in the morning — and the standard that defines those fields is still marked as in development.
Read - AI Engineering6 min
Pick LLM Evaluation Metrics by What You Can Supply
By Petru Popa ·The usual taxonomy sorts metrics by how they work — n-gram, embedding, model judge. That axis is useless for choosing. Sort them instead by what reference data each one demands from you, because that is the constraint you actually have.
Read - AI Engineering6 min
LLM Testing Already Had a Playbook in 2020
By Petru Popa ·Teams treat testing a language model as a new discipline with no precedent. The methodology that fits it best was published in 2020, and the test type it names as most valuable is the one almost nobody writes.
Read