26.3 Conflict detection in retrieved sources

Goal: don’t blend conflicting sources

When sources conflict, the worst outcome is a confident synthesis that hides the disagreement.

Your goal is to make the system say:

• “I found conflicting sources,”
• show the conflicting quotes,
• apply a consistent resolution policy or ask a clarifying question.

Why conflicts happen in RAG systems

• Version drift: old docs are still indexed.
• Scope mismatch: one doc is general, another is environment-specific.
• Authority mismatch: tickets/chats contradict canonical docs.
• Partial updates: one doc updated, another not.
• Ambiguous language: “should” vs “must,” unstated exceptions.

Detection strategies (cheap to advanced)

1) Simple heuristics

• if retrieved chunks have different versions and disagree, flag conflict.
• if sources contain “deprecated,” “obsolete,” “v1/v2,” treat as potential conflict.
• if top sources come from different doc types (policy vs ticket), flag.

2) Model-assisted conflict detection

Ask the model to compare sources and report conflicts, but require quotes and ids.

This works best when you keep the scope small (top 5–10 chunks) and force structured output.

3) Evaluation-driven detection

When you find a conflict in production, add it as an eval case. Conflicts you’ve seen before should be handled consistently.

Resolution policies (pick one)

Choose a policy and encode it in prompts and code:

• Authority policy: prefer canonical sources; treat others as advisory.
• Recency policy: prefer newer versions when version metadata exists.
• Environment policy: prefer sources scoped to the user’s environment.
• Escalation policy: when conflicts are high impact, require human decision.

Many systems use a hybrid: authority first, then recency, else escalate.

Conflict-aware output schema

{
  "status": "conflict",
  "conflicts": [{
    "topic": string,
    "source_a": { "chunk_id": string, "quote": string, "version": string|null },
    "source_b": { "chunk_id": string, "quote": string, "version": string|null },
    "policy_applied": "authority" | "recency" | "environment" | "escalate",
    "suggested_resolution": string
  }],
  "follow_up_question": string|null
}

Even if you choose to resolve automatically, capturing the conflict in logs is valuable for audits.

UX: how to present conflicts

Good conflict UX:

• shows the conflicting quotes side-by-side,
• labels sources (doc type, version),
• explains what policy was applied (or why it escalated),
• asks one clarifying question if it would resolve the ambiguity.

Copy-paste prompts

Prompt: detect conflicts in retrieved sources

Analyze the SOURCES below and the user question.

Rules:
- Do not answer the question yet.
- Identify any conflicts between sources relevant to the question.
- For each conflict, include chunk_id and a direct quote from both sides.
- Suggest a resolution policy: authority / recency / environment / escalate.

Return JSON:
{ "conflicts": [{ "topic": string, "a": { "chunk_id": string, "quote": string }, "b": { "chunk_id": string, "quote": string }, "policy": string, "suggested_resolution": string }] }

SOURCES:
...

Question: ...

Where to go next

• 26.4 Refusal and escalation flows
• Project 2 evaluation (25.4)