Conflicting records are normal, so test for them
Data warehouse modernization repeatedly brought me back to differences between finance, regulatory reporting and the digital channels. The same account or customer could have several valid representations. Documenting why they differed was part of making the platform useful. I want an AI evaluation to include those disagreements, because a system that handles only clean, consistent records has not been tested against the work it will actually receive.
Pick a conflict that occurs in real work: a customer status, a revenue total, a delivery date, a member's coverage effective date. Sanitize it and use it. A deliberately absurd contradiction is less useful than the subtle differences reviewers see every week.
Document what each source is meant to represent. One system holds current operational state while another preserves a reporting snapshot. The values can differ with neither being defective. The test is whether the assistant recognizes that context or simply picks the most recent-looking value.
Define the right answer before you run the test
Ask the business owner to define the expected response first. Sometimes it is the designated authoritative source. Sometimes it is an explanation of the mismatch and a request for clarification. Sometimes the workflow should stop until an owner resolves the issue. All are valid outcomes, and the evaluation should know which one it expects.
Give each record its source identity, timestamp and business definition, so a reviewer can tell stale data from a true disagreement. If the organization has a source precedence policy, make it part of the maintained workflow definition, not a hidden instruction in a prompt. An answer that depends on an undocumented trick cannot be operated or audited when the context changes.
Include a case where the preferred source is unavailable or incomplete. The system should not silently substitute another source when that changes the meaning. A clearly qualified alternative may be acceptable if the business process allows it, but the user has to know.
The failure that matters most: invented explanations
The most dangerous failure is a plausible story with no evidence behind it. An assistant might attribute a discrepancy to currency conversion or a sync delay when the records establish neither. In a bank, that narrative can end up in a variance explanation to the CFO. In a genomics or diagnostics lab, a confident reconciliation of conflicting sample metadata is worse than no answer at all. Reviewers must separate possible explanations from demonstrated ones.
Ask the assistant what evidence would resolve the disagreement. A good answer is concrete: inspect a status history, confirm a reporting cutoff, consult the metric owner. “Check the data” is not operational help.
Check that uncertainty survives to the final summary. Answers often acknowledge a limitation in the middle and then close with an unqualified conclusion. The person acting on the summary needs the qualification at the point of decision.
Turn conflicts into operating model improvements
Classify what you find. Some disagreements reveal source defects, others unclear definitions, missing timestamps or unresolved ownership. Assign an owner and a next step to each category. Do not treat every conflict as a model problem, and do not assume a new model will fix a business policy gap. This is why a governed semantic layer and certified metric definitions matter more than model choice for most enterprises.
Keep the difficult examples in your regression set. A later change can improve routine answers while weakening conflict handling. Preserve the cases where the right outcome is a clarification, a qualified answer or a refusal to reconcile unsupported values.
What examiners and auditors will expect
Where the workflow allows a qualified answer, preserve the unresolved case in the record: which source was used, why it was chosen and what evidence would cause that choice to be revisited. A reviewer who later sees only the selected number will otherwise miss the disagreement entirely.
Keep the precedence rule with the evaluation case, including its owner and the reason it applies. A rule for current customer status may be wrong for a historical regulatory report. Reviewers need that context to judge the response fairly. When the rule changes, preserve the previous version so an investigation can explain why an older answer differed from the result produced today.
In each case the principle holds. A trustworthy assistant does not make every source agree. It explains what the evidence supports, preserves material uncertainty and helps the user reach the right next decision. Conflicting records reveal whether your workflow rewards honesty or merely a smooth narrative.