Testing & Quality

8+ min ago   (564+ words) Disclosure: I maintain Selvedge, an open-source decision-history tool. This article was prepared with AI assistance from Grok and Codex. The example below is hypothetical and uses plain Markdown; it does not require a particular product. A repository can show what…...

26+ min ago   (244+ words) While tech review articles argue about what PR length is safe to let an agent produce, the review problem is set by a simpler number: how often the agent is wrong. Specific Labs' Real-SWE benchmark, published September 2026, runs frontier agents…...

41+ min ago   (415+ words) I was building a tool that detects when data quietly changes meaning — a vendor switching units, a source dropping a field, an undocumented enum appearing. The kind of failure where every test passes and every job is green. Claims about…...

1+ hour, 23+ min ago   (637+ words) I’ve renamed SkillCheck to TraceMantle and expanded its scope. The original project focused mainly on validating AI agent skill files. TraceMantle retains that functionality and adds support for tracking the files in a skill package, comparing changes, and checking whether…...

4+ hour, 9+ min ago   (329+ words) The question I keep getting is how a team can review the growing volume of AI-generated code. Most answers start with throughput: faster review tools, more automated checking, larger batches. That assumes the problem is volume. The review I read…...

4+ hour, 47+ min ago   (181+ words) Originally published on hexisteme notes. Closing this out took more than restoring the CTA. Four changes, each aimed at a different point where the same shape of failure could recur: And I split the downstream metric so "no data yet…...

6+ hour, 11+ min ago   (686+ words) Published on DHSeaDev. A hard-won lesson from shipping a browser-based TCG. When we stress-tested Prismwar, our original gate suite reported everything was fine. Then we ran a mutation campaign, and 31 out of 70 carefully crafted bugs survived without raising a flag....

7+ hour, 15+ min ago   (442+ words) A regular-expression denial-of-service bug is a regex whose backtracking engine can be pushed into super-linear (often exponential) time by a short, hand-crafted input. The textbook shape is a quantifier inside a quantifier: The scary part is that these patterns look…...

6+ hour, 42+ min ago   (528+ words) CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization. For two releases we treated "the pass rate is low" as…...

6+ hour, 52+ min ago   (657+ words) Why this post is different from the last one The last write-up in this series announced four tools. This one is about what happened when I stopped writing tests for my own tools and started checking one of them against…...