One of the most useful AI failures I’ve encountered produced a completely blank document. The retrieval layer queried the underlying system with the wrong date-window behavior. It returned no rows. The model received no records.
The visible symptom was an empty AI output. The defect was earlier in the chain. Looking only at the prompt and model would have sent the investigation toward the wrong component.
I am leaving the practice, people, dates, and record contents out of this account. The important fact is the boundary failure: an empty retrieval result reached generation without making its cause useful to the operator.
A successful query can still be the wrong query
An HTTP success response establishes that a request was processed. It does not establish that the request represented the intended workflow. A query can succeed while selecting the wrong date range, applying the wrong status filter, or stopping before the last page.
In this incident, the known problem was date-window behavior. I am not claiming a particular time-zone conversion or pagination bug caused it. Those are separate cases worth testing because they can create the same symptom.
A useful postmortem separates the observed cause from plausible adjacent causes. Otherwise the story becomes memorable while the regression test becomes inaccurate.
“Tomorrow’s patients” needs an executable definition
Consider an illustrative chart-preparation workflow in a pain-management clinic. A service running in UTC interprets “tomorrow” differently from a clinic using local calendar dates. A naive next-24-hours filter can also include parts of two clinic days.
Define the local interval explicitly, translate its boundaries consistently, and test records at both edges. Start-inclusive and end-exclusive intervals are one practical convention. A synthetic appointment exactly at the start should be included; one exactly at the next day’s start should not.
Do not assume every calendar day is exactly 24 hours in UTC. Test daylight-saving transitions where applicable. Those tests belong to the retrieval contract, independently of the model summarizing the returned records.
Without that discipline, the system may prepare the wrong appointment set, omit an expected briefing, or present yesterday’s material as current. A better-written summary cannot repair a schedule the system never selected correctly.
Empty is a state, not an explanation
A chart-preparation pipeline needs to distinguish at least four situations:
- No eligible appointments: the schedule query completed and the selected interval genuinely has no eligible records.
- Evidence incomplete: an appointment exists, but a referenced source could not be retrieved.
- Retrieval failed: a permission, timeout, or dependency error prevented a reliable answer.
- Ready for review: the defined retrieval checks completed and the draft passed evaluation.
Collapsing these into an empty list removes what an operator needs to recover. Retrying generation is pointless when the missing item is a source-system permission. Marking the job complete is misleading when page two of the results never arrived.
A useful interface might say, “Draft held: one referenced report could not be retrieved,” and link an authorized reviewer to the source. This illustrative message names the next action without turning missing evidence into a clinical conclusion.
Available text is not reviewed evidence
A prior pain-management note might mention an outside MRI. If the integration retrieves that note but not the report, the model has evidence that a report was mentioned. It does not have evidence of the report’s contents.
An evaluator should reject a draft that upgrades “MRI referenced in prior note” to “MRI reviewed” or supplies findings absent from the sources. Attach provenance at the claim level when the review requires it. A generic citation to a whole chart may be too broad to help check a specific statement.
This is an illustrative failure mode, not another incident I am claiming occurred. It shows why retrieval completeness and generation faithfulness need separate tests. A system can faithfully summarize incomplete evidence; it can also retrieve the right evidence and generate an unsupported claim.
Approval should expire when the evidence changes
Imagine a staff member reviews a draft, then an updated source arrives before the downstream action. If the system regenerates the draft, the earlier approval must not silently attach to the new version.
Bind approval to the specific proposal and the evidence version supporting it. When material inputs change, re-evaluate and obtain the approval required by the workflow. A clinician’s sign-off is not evergreen permission for future generated text.
Not every read operation needs to wait for a human. Read-only collection, drafting, signing, and external communication have different consequences. Define each boundary, with clinical decisions retained by qualified clinicians.
What I would instrument and test
An authorized operator should be able to inspect the requested local interval, normalized query bounds, filters, pages fetched, record count, completeness status, source references, and the point where a draft was blocked or released. Sensitive clinical content should not be copied wholesale into general-purpose logs.
The regression set should include a boundary-time appointment, a canceled appointment, a missing attachment, a permissions failure, partial pagination, and a source update after approval. Synthetic fixtures can test these semantics without using production patient records.
Then test recovery. Does an incomplete result create an actionable task? Can a retry resume safely? Does the operator know whether a write happened? A warning nobody can resolve is only a more verbose failure.
The lesson from the blank document: before asking whether the model produced the right answer, establish whether the system gave it the right records—and whether the workflow knew what to do when it did not.
That is the engineering around production AI: date semantics, state transitions, evidence checks, versioned approvals, and recovery. These details decide whether an application reduces work or creates another queue of mysteries for the clinic to solve.
Written by Morteza Naraghi, co-founder and CTO of Meros AI and author of Agentic Software Delivery.