What breaks when you put an LLM in an approval workflow
Audit trails, determinism, and the failure modes nobody demos. Notes from putting a language model inside a regulated process without losing the ability to explain what it did.
August 4, 2026 · 4 min read · Head of AI Engineering
An approval workflow is a sequence of decisions with names attached. Someone submitted, someone checked, someone approved, and the record shows who and when. A language model makes an attractive addition: it can read the request, summarize it, check it against policy, draft the response. The demo is compelling. Then the compliance team asks three questions, and the demo stops being the point.
Can you show me why it decided that? Will it decide the same thing tomorrow? What happens when it is wrong?
We have now built several workflows with a model inside them, in onboarding, vendor verification and expense approval. These are the things that broke, and what we did about them.
Determinism is not optional, and you do not have it
The same request, sent twice, can get two different summaries and two different policy readings. Temperature zero reduces this and does not remove it: model versions change, prompts get edited, retrieved context varies with what else is in the system that day.
An approval process cannot work that way. If a request was approved on Tuesday and an identical one rejected on Thursday, the organization has a fairness problem and an audit problem.
What we did: the model never makes the decision. It produces structured output, a set of extracted facts and a set of policy flags each with a cited clause, and a deterministic rules engine makes the decision from that output. The rules are versioned. Given the same facts and the same rules version, the decision is the same. The model’s variability is confined to the fact-extraction step, where it can be reviewed.
We also pin model versions per workflow and treat a version change as a release: run the regression set, compare outputs, sign off. Silent upgrades are disabled.
The audit trail has to capture what the model saw
A traditional audit trail records who approved and when. With a model in the loop, that is not enough. To explain a decision later you need the exact input the model received, including retrieved context; the prompt version; the model version; the raw output; and the rules version that acted on it.
We store all of it, per decision, as an immutable record. It is large. Storage is cheap and the alternative is a decision nobody can explain fourteen months later when a regulator asks.
One thing we got wrong first time: we stored the prompt template and the variables separately, and later could not reconstruct exactly what was sent because a formatting function had changed. Now we store the rendered prompt as sent.
Confidence is not a probability
Models will give you a confidence score if you ask. It is not calibrated. A high stated confidence on a policy flag does not mean it is right that often. We stopped exposing raw confidence to reviewers after watching them treat it as a probability.
What works better: per-field agreement checks. Extract the same fact two ways, or cross-check it against a structured source, and treat disagreement as low confidence. Route disagreements to a person. This is slower and it is honest.
Failure modes nobody demos
The confidently wrong extraction. The model reads a date and returns a plausible, well-formatted, wrong date. No error, no low score. Cross-checking against the submitted form catches most of these. Not all.
The policy that is not in the context. Retrieval fetches the wrong policy version, or misses the clause that matters, and the model reasons correctly from incomplete rules. The flag is missing rather than wrong, which is harder to notice. We now retrieve by policy version explicitly, tied to the request date, rather than by similarity alone.
Prompt injection through the request itself. A submitted document that contains text addressed to the model. We saw a supplier invoice with instructions embedded in a comment field. The mitigation is structural: the model’s output schema is fixed, the rules engine ignores anything outside it, and any text in the output that is not a schema field is logged and discarded.
Silent scope creep. A model that summarizes well gets asked to also recommend, then to also decide. Each step is small. We wrote down, per workflow, exactly what the model is permitted to produce, and changing that list is a governance decision with sign-off.
What stays with a person
Anything consequential, by which we mean anything the organization would have to defend. The model can flag that an expense exceeds policy. A person approves the exception. The model can extract a vendor’s registration details. A person confirms the vendor.
This is not caution for its own sake. Regulated processes require a named accountable individual, and the system is built so that individual sees what the model found, in plain language, with the source highlighted, and signs with one click. The click is logged with the rules version.
Trade-offs we accepted
Every decision is slower than a pure model call. The structured-output step, the rules engine, the storage of the full record. We accept that because the alternative is fast and unexplainable.
The rules engine means some flexibility is lost. A model could handle a genuinely novel case; the rules engine routes it to a human. We think that is correct for approvals and wrong for, say, a search interface.
What we would do differently
Build the regression set first. We assembled it in the second phase, after a prompt edit changed outputs in a way nobody noticed for two weeks. Now it is the first artifact in any workflow: a few hundred real, anonymized requests with agreed correct outputs, run on every change.
And we would involve the compliance team in writing the permitted-output list from the start. They asked the right questions in the first meeting. We should have written the answers down then.