Never Answer an Agent With a Silent Substitution

Your agent asked for 9:16. It got something else, and the response said success.

Nothing in that reply is false. The video exists, the job finished, the status code is 200. The API simply decided that what you asked for was close enough to what it could do, made the substitution, and moved on. Your agent has no way to know. It branches on success, and success is what it got.

We shipped that API. This is the rule we ended up with, and the bug that proved we needed it.

The rule: an unmet parameter is named aloud

Every request carries parameters we might not be able to honour exactly. Aspect ratio is the obvious one. The client wants 9:16, the vendor renders it at 1080×1920 by default, and the source image is 1024×1024. Something has to give.

There are three honest options, and exactly one dishonest one.

The honest options: match it, adjust it and say so, or refuse. The dishonest option is to adjust it and stay quiet. That last one is the most tempting, because it produces the highest success rate on every dashboard you own.

So the contract is that an unmet parameter comes back named. Not as a status code. Warnings are prose, deliberately, and we wrote about why warnings carry no stable codes while refusals do. The point here is narrower: whatever the shape, the thing that did not happen has to appear in the response.

What broke

The rule was in place. The implementation quietly stopped honouring it. We have written before about agents reporting success for work that never landed, which is the same shape one layer up.

Two of our warnings depend on a network probe. We fetch the source image, measure it, and compare against what the client asked for. That probe runs during the request. Its output fed the warnings directly.

Then a worker pass could fail and rebuild the row, and the warnings went with it. Not all of them, only the two that came from the probe. The probe had run. The measurement had happened. The warning it produced was gone by the time anyone read the response.

Every test stayed green. The endpoint returned 200. The video rendered. The job completed. The only thing missing was a sentence explaining what we had changed, and no assertion anywhere was watching for a sentence.

It went unnoticed for weeks. We found it because someone compared a 202 response against the quote for the same job and noticed the quote mentioned a crop that the 202 did not.

The fix is about where facts live

The obvious repair is to make the worker stop clobbering the warnings. We did something else, because the obvious repair leaves the same failure available to the next worker anyone writes.

The warnings were being stored as strings. That was the actual mistake. A string is the end of a pipeline. Once you have written "the source will be cropped to fit", you have thrown away the measurement that justified it, and anything that rebuilds the row has to know to preserve prose it cannot interpret.

So the probe results now land in their own column as facts: the measured dimensions, the ratio, what the vendor will do about it. Structured, boring, and meaningless to a human. And every reader (the 202 response, the quote endpoint, a later GET) derives the warning text from those facts at read time, through one shared function.

The consequence is the point. All three now agree, because all three are reading the same column through the same code. There is no longer a copy of the warning that can drift, because there is no copy of the warning at all until someone asks for one.

What this costs

Two things, and neither is free.

You give up cheap reads. Deriving text at read time means every response does a little work that a stored string would have skipped. For us this is trivial. It is a handful of comparisons against numbers already in the row. If your derivation were expensive, this trade would look different.

You have to decide what a fact is. "The image is 1024×1024" is a fact. "The image will be cropped" is a conclusion. "Cropping may lose faces near the edge" is editorial. The column holds the first, the reader produces the second, and the third does not belong in an API response at all. Getting that boundary wrong just moves the problem: store conclusions and you are back to strings with extra steps.

The general shape

Most agent APIs will meet this eventually, because it comes from an asymmetry that does not go away: a human reading your response notices a missing sentence, and an agent does not. A person who asked for vertical video and got a square one will say something. An agent will write it to storage and call the next tool, which is how a multi-step agent compounds a single bad answer. The MCP specification(opens in new tab) is explicit that tool results are model-visible context, not human-visible output.

That means the usual safety net, someone eyeballing the output, is gone precisely where you have started trusting the output most. You are left with two options: assert on the warnings in tests the way you assert on the payload, or remove the class of bug by making warnings underivable-from-nothing, which is what storing facts buys you.

We did the second. The first would have caught this particular instance and left the next one to chance.

FAQ

Why not just return a structured warning object instead of prose?

We tried the argument and it collapses on stability. A code is a promise you keep across versions; the situations that produce warnings change as vendor behaviour changes. A code you cannot keep is worse than honest prose, because an agent will branch on it. Refusals carry codes because they are control flow. Warnings explain what we did instead.

Doesn't refusing more often just make the API harder to use?

It makes it harder to use accidentally. We refuse only an explicitly requested format that we cannot honour within tolerance, and we refuse it before the paid vendor call rather than after. If the client never specified a format, any mismatch adjusts with a warning. The strictness scales with how specific the request was.

How do you test for a missing warning?

Assert on the facts column, not the prose. The text is derived, so a test that checks wording breaks every time someone improves a sentence. A test that checks "a probe ran and recorded a 1:1 source against a 9:16 target" survives rewording and fails when the row is clobbered, which is the failure we actually had.