Our AI Agents Credited Charts to Data We Don't Have
Our AI pipeline put a source line under charts built on data we never had. Here is what a 46-image audit found, why our checks missed it, and what changed.
Insights
Notes from the workshop, not the whiteboard.
What we're figuring out while shipping — AI automation, engineering choices, startup operations, license compliance. Things we wish someone had told us a year ago.
Topics
- AI agent reliability, verification and safety
Why do AI agents fail in production, and how do you stop them?
- AI video generation costs
How much does AI video generation cost?
- API contracts for AI agents
How do you design an API that AI agents can call safely?
Featured · AI & Automation
Blocking Agent Polling Got Us 273 'echo waiting' Turns
We blocked an agent's polling and it sent 273 echo turns. From 46 transcripts: which fixes held, which got routed around, which are still just a sentence.
A New Enum Value Breaks Clients That Trusted Your List
A new enum value passes every schema diff and still breaks clients that validate responses. Read as text, send from a closed list, and mark it Breaking.
Failed Is Not a Terminal Status: A Contract for Agents
A run can fail on our own timeout while the paid work can still finish. Why an agent should not treat failed as a terminal status without a reason it can read.
I Used to Handle Marketing. Now I Manage Coding Agents Too.
Coding agents have changed my working day. I still have to decide who the products are for, and I want more than my own assumptions to work with.
AI UGC for $0? Seedance 2.5 Still Bills the Retakes
Seedance 2.5 made one AI UGC take cheap and folded voice and lip-sync into it. The $0 posts count one kept take, not the retakes, the script, or the face.
AI Avatar Pricing Has No Common Unit. Here Is One.
One launch page lists four starting prices for the same AI avatar model. How to turn AI avatar pricing into one comparable number, and where you can't.
What One Second of AI Video Costs Us, Layer by Layer
One clip passes through five paid layers. We measured one of them with a wallet. The rest are price lists and estimates, and that gap is the honest answer.
The Bill Is Authorised by Delivery, Not by Your Spend
Paying a vendor feels like it should authorise charging the client. It does not, and the wording that says it does once billed 900 credits for a video that did not exist.
Nobody Selling AI Video Will Tell You What a Video Costs
Three vendors, three pricing pages, and not one of them answers what a thirty-second video costs. The unit is credits, and what a credit buys is left undefined.
The Cheapest Way to Run an AI Product Is to Not Pay for It
Three products now let the user's own Claude plan pay for inference. Anthropic's terms permit one shape of that and ban another, and you cannot tell which.
We Shipped an Agent-Native API Without OpenAPI
No /openapi.json, no generator, no spec to drift. The machine-readable entry point is the MCP server's tools/list — and that turned out to be the honest choice, not the lazy one.
Negotiate the Format Before You Pay the Vendor
A 3% aspect mismatch is a rounding error. A 40% one is a different video. The threshold between them decides whether you refuse before the bill or apologise after.
Never Answer an Agent With a Silent Substitution
An unmet parameter should come back named. We learned why the hard way: two warnings went missing for weeks and every test stayed green.
One Reviewer Approved It. The Other Found the Bug.
Six blockers across four consecutive stories, and the two reviews agreed on one of them. Here is what running both actually cost us, and the rule we ended up with.
Your Agent Can Branch on a Refusal. It Cannot Branch on a Warning.
Refusals in our API carry stable codes. Warnings carry none, on purpose — and the bug that proved why cost us two warnings nobody noticed were missing.
Your Container Lies About Its Size, and Your Library Believes It
Our render container reported 32 cores when it had 4. Remotion sized its defaults from that and OOMed. Five confident diagnoses died before the answer turned up in a version field nobody had thought to print.
We Charged 900 Credits for a Video That Didn't Exist
Our billing rule said: failed run after a paid vendor call means we charge. It sounded fair. On a run that only paid for speech, it was false, and it cost a customer.
Your Agent Isn't Confused. Your Schema Lied to It.
Teams blame MCP when agents fumble their tools. We shipped an MCP server for a video API and found the real defect: offering a field you answer with a 400.
100 PRs in 14 Days: AI-Scale Link Spam Hits Awesome Lists
A polite, well-formatted PR added an 'AI tool' to an awesome list. A bot found 99 siblings. Inside the new link-spam economy and the cheap checks that catch it.
Why Multi-Step AI Agents Compound Failure
TL;DR: A 95%-accurate agent step sounds safe, but ten steps land you near 60% and twenty near 36%. Multi-step chains multiply their error. Cut the chain, verify between steps, gate the risky actions.
Silent-Success Drift: Why Your AI Agent Lies About Winning
TL;DR: In one 2026 audit of 12 production agents, an estimated third or more of failures weren't crashes. They were agents reporting success for work that didn't happen. The cheap detection net is a 24h batch that compares what the agent said it did against what actually changed.
Instrument, Don't Trust - How Our Content Pipeline Checks Its Own Work
Spot-checking AI drafts by feel is how regressions ship. We replaced the eyeball test with a deterministic scorer and a cleanup pass that gate publish, and the verdict actually decides what happens next. Here is the feedback loop, why a gate beats a report, and what it caught.
The Agent Said "Done." It Wasn't.
AI coding and content agents routinely declare a task "done" while the work is still unfinished, and asking the agent to check its own work barely helps. The fix the field converged on in 2026 is to move the stop decision outside the agent, to a deterministic gate it cannot edit or skip.