We found duplicate orders in production and had an obvious suspect: the LLM. Over two weeks, we saw 35 duplicate orders across 30 visits and 10 providers.

A common pattern looked like this: “the already scheduled MRI.” The model would see that phrase and generate another order.

At first, this looked like a familiar AI failure. The model misunderstood the sentence. Except it had not. It was following the prompt.

The rule made sense

Our prompt told the model to ignore past-tense history and identify definite orders. That seemed reasonable. Something that already happened belongs in history. Something a provider intends to happen should become an order.

But an appointment that is already booked in the future sits in an awkward middle ground. It is definite. It is future-tense. And it should not be ordered again.

The prompt did not represent that state clearly enough. So when the model saw “the already scheduled MRI,” it had enough evidence to classify it as a definite future action.

According to the prompt, that was reasonable. According to the product, it was wrong.

The model was only one part of the failure

The more interesting problem was what happened after the model made that decision. In this part of the system, every action emitted by the model effectively meant “commit this.”

There was no later stage that could distinguish between “this should happen” and “this is already booked.” Once the model chose the action, the rest of the system had very little room to recover.

That changed how we looked at the bug. This was not just a prompt problem. It was also an architecture problem. We had asked one model output to represent multiple product states.

The obvious fix was too broad

The easiest fix would have been something like: “If something is described as happening in the future, do not order it.” That would have prevented the duplicate MRI. It also would have broken valid cases.

The word “scheduled” can mean different things depending on context. In one case, “this is already scheduled” means do nothing. In another, “schedule this within the lab window” can still mean the action needs to be committed.

A blanket rule would solve the example in front of us and quietly damage something else. That is one of the harder parts of working with large prompts. A line added for one failure can change behaviour somewhere completely different.

We started with the counterexamples

Instead of writing the rule first, we started by defining the cases that still needed to work. That turned out to be more useful.

We needed examples where “scheduled” should mean:

  • Already booked, do not order again
  • Still needs to be scheduled
  • Timing information only
  • Part of a different workflow where the action should still be committed

Once those cases were explicit, the problem became easier to reason about. The eventual change added clearer prompt handling for “already on the books” and separated that from the other scheduling language.

But the more important change was how we approached the prompt. We stopped treating the bad example as the whole problem.

Prompts have side effects

This was something we had already started learning elsewhere in the system. Early on, we relied on large prompts that handled multiple responsibilities. They worked well enough to get the system moving. They were much harder to change safely.

A tweak intended for one behaviour could alter another section that depended on the same instructions. That made evaluation difficult too. If an output changed, it was not always obvious which instruction caused it.

As we split responsibilities into smaller workers, the system became easier to reason about. Accuracy improved. But the bigger improvement was that we could evaluate and change one responsibility without touching everything around it.

That pushed us toward a more useful debugging question.

Is this an input problem, a model problem, or an architecture problem?

“The AI got it wrong” is not a very useful diagnosis. There are several different ways an AI feature can fail.

The input can be incomplete or ambiguous. The prompt can describe the wrong rule. The model can fail to follow a good instruction. Or the architecture can force multiple meanings into one output.

Those failures often look identical when all you see is the final result.

In this case, blaming the model would have sent us in the wrong direction. The model was doing what we asked. The problem was that what we asked did not describe the product precisely enough.

That is a distinction I pay much more attention to now. Before changing the prompt, I want to know which part of the system actually failed.

STATUS ● SHIPPED

END OF FILE