AI & Automation

Stale and Conflicting RAG Sources: Decide Which Document an Answer Can Use

Two documents answer the same question. One says use the blue inspection card. The other says use the green card.

Both are searchable. Both have plausible titles. A fluent answer can hide the disagreement.

Give the retrieval system a source decision before asking the model to write a procedure. Which document has authority? When does it apply? What happened to the version it replaced?

This article proposes a reconciliation method for internal knowledge. Its inspection-card example is fictional. Azure documentation supplies specific indexing behavior. The authority rules and worksheets are proposed editorial methods, not built-in vendor features.

Keep dates and approval status separate

A modification timestamp records a file change. An effective date records when instructions apply. Approval status records whether the responsible owner accepted them. An index update time records when a searchable representation changed.

Write all four in the source register. Do not collapse them into one “updated” field.

A draft can have today's modification date and no approval. An approved procedure can have an older file date and remain in force. A future procedure can be approved while its effective date has not arrived.

The index has its own evidence. Record the source version represented by each indexed document, when ingestion processed it, and what queries showed afterward. A recent ingestion timestamp does not establish that the input was authoritative.

For each question, define the relevant scope too. Product model, location, process stage, and intended audience can separate documents that only appear to conflict. Compare their applicability before declaring one obsolete.

The RAG architecture guide covers the retrieval pipeline. This source register governs which material that pipeline should present as current instructions.

Build a conflict record before choosing a winner

Consider a fictional equipment workshop. Its assistant answers, “Which inspection card applies to the bench assembly?” The following records contain synthetic dates and roles.

Field Document A Document B
Stable document ID BENCH-CHECK BENCH-CHECK
Version 3 4
Instruction Use the blue card Use the green card
Scope Bench assembly, line one Bench assembly, line one
Authority Approved operating procedure Approved replacement procedure
Owner Assembly process owner Assembly process owner
Approval status Approved, pending supersession Approved
Effective date September 1, 2026 November 2, 2026
Supersession Replaced by version 4 when effective Replaces version 3 when effective
Source modification October 27, 2026, formatting edit October 20, 2026, final approval
Index processing October 28, 2026 October 28, 2026

On October 28, this fictional rule keeps version 3 current. Version 4 describes the approved future state. The newer file modification belongs to version 3, but that fact does not decide authority either.

On November 2, the intended current procedure becomes version 4. The owner must confirm the transition and any exceptions. The assistant should not invent that confirmation from the calendar alone if the local release rule requires it.

Now imagine neither record identifies a replacement relationship and the owner has not confirmed one. The review cannot establish which version governs. Ask the owner whether the documents replace one another, cover distinct exceptions, or contain an unresolved mistake.

Keep that uncertainty in the register. Never convert missing authority into a confident instruction because one document appeared first in search.

Define what counts as authoritative input

Create a small set of explicit source states: draft, approved future, current, superseded, withdrawn, and unresolved. These are proposed states for the worksheet. Adapt them to the process owner’s vocabulary.

Give each state a retrieval purpose. Current documents may support current instructions within their scope. Approved future documents may support clearly labeled planning questions. Superseded documents may support a permitted historical question with their version visible.

Exclude drafts and unresolved conflicts from definitive operating instructions. Decide separately whether authorized users can search those materials for review. Eligibility for a review search does not establish authority for an answer.

Put the decision before generation. Give the answer step an approved source set and the question's time frame. Retain the reason each source was admitted or withheld in the private review record.

If the system lacks these controls, narrow the assistant's remit. It can identify candidate documents for a human owner without issuing a procedure. That is a valid operating boundary while the source register is incomplete.

Keep the business knowledge system tied to named document owners. A collection without an accountable owner has no reliable place to send an authority conflict.

Separate updating the index from resolving the source

Once the owner settles authority, reconcile the searchable copies.

Azure's index update documentation distinguishes upload, merge, mergeOrUpload, and delete. A merge updates existing fields and replaces collection values. It does not append new values to an existing collection automatically.

Use that documented distinction when reviewing the ingestion design. Check whether an update replaces the complete authority record or changes selected fields. Ask what happens to an old supersession value, scope list, or approval state when the source omits it.

Azure also documents per-item indexing results and background indexing. A successful request is evidence of accepted work. Confirm the affected version through subsequent queries before treating reconciliation as complete. Review the response semantics.

For this proposed method, keep the source revision, indexing operation, affected chunk identifiers, and verification query together. Mark a partial result partial. Do not close a conflict because unrelated documents indexed successfully.

If the update changes chunk boundaries, compare the old document family with the replacement family. Account for old chunks that no longer have a corresponding paragraph. Their absence from the new source is a reconciliation question.

Treat deletion as a separate path

Deleting the original file is an event to verify through the retrieval system.

Azure Storage indexer documentation describes automatic change detection using storage timestamps. Deletion detection requires a strategy configured from the first indexer run. Adding it later does not recover already orphaned search documents.

Native blob soft delete has specific limits. The documented policy does not support Azure Files or storage accounts with blob versioning enabled. The indexer must process the blob during its soft-delete retention period. A custom metadata soft-delete strategy is also documented. Check the connector requirements.

For this article's proposed review, name the actual connector and deletion mechanism in the worksheet. Ask how a missed indexer run affects detection. Record which component confirms removal and which owner handles an orphaned entry.

Chunking requires another check. Azure states that the two soft-delete strategies described on that page do not apply to its linked one-to-many indexing scenarios. It directs those scenarios to explicit index deletion requests. Do not assume a parent-file deletion removes every derived chunk. Read the documented limitation.

Map the deletion plan to the actual indexing design. A connector label alone cannot establish cleanup behavior for an application’s custom chunk pipeline.

Reconcile chunks, caches, and citations

Use one reconciliation worksheet per source change. The following blank worksheet names evidence to collect. It reports no completed deletion or verified production behavior.

Surface Identity and version to record Evidence to collect Unresolved question and owner
Source document Stable ID, revision, owner, scope Approved state and effective date Who confirms authority?
Supersession record Old revision and replacement revision Owner's explicit replacement decision Do exceptions still require the old text?
Indexed parent Search key and source revision Update result and verification query Does the indexed state match the source?
Derived chunks Parent ID, chunk IDs, revision Expected family compared with query results Which old chunks remain?
Deletion path Connector, mechanism, affected keys Detection record, deletion result, search observation Was any item missed or orphaned?
Retrieval cache Cache scope and source references Invalidation or refreshed lookup evidence Can old candidates still enter context?
Answer cache Cached answer and cited revisions Invalidation or eligibility check Can an obsolete instruction still display?
Conversation Saved answer and reused context Display rule and new-answer context check Is history labeled with its original date?
Citation Source ID, revision, displayed destination Title, excerpt, and destination inspection Does the link open the version used?
Open conflict Documents, question, owner Decision request and response What remains withheld?

Keep “requested,” “processed,” and “verified in retrieval” as separate observations. An accepted cleanup request cannot fill the final column by itself.

For the fictional workshop, inspect the blue-card chunks, the green-card chunks, and a previously cached blue-card answer. Test a current procedure question and an explicit historical question separately.

A citation should identify the revision that supported the answer. If the destination always opens the newest file, decide how the interface preserves that distinction. Otherwise, a reader cannot inspect what the answer used.

Withhold an instruction when authority remains unresolved

Define refusal conditions in the source register before the assistant encounters a conflict.

Withhold a definitive procedure when two applicable approved documents disagree and neither has a confirmed priority. Do the same when a replacement lacks an effective date, an owner has withdrawn the only source, or the index cannot establish its revision.

Use a neutral message. For an authorized reviewer in the fictional workshop: “The bench assembly instructions conflict. The process owner must confirm the current card before this assistant can provide the procedure.”

Show document identifiers only when the reader may access them. A conflict message has its own disclosure boundary.

Send the owner a specific decision request: question asked, applicable scope, conflicting passages, versions, approval states, and missing supersession evidence. Ask which instruction applies and what should happen to the other source.

Never ask the model to settle a process authority dispute by weighing writing quality. A polished draft and a rough approved procedure have different roles. Similarity ranking cannot supply a missing approval.

Give the unresolved record a named owner and a review date. Keep the procedure held until the stated evidence arrives. A quiet queue is not resolution.

Verify the transition with questions people ask

After reconciliation, run a bounded set of questions using harmless fixtures. Include the original wording, a paraphrase, a question naming the retired instruction, and a question about the future procedure.

For each, inspect the retrieved revision, selected context, answer, and citation. Test the old cached question too. Record the expected source before reading the output so fluency cannot redefine success.

Add a question that should remain unanswered. In the fictional workshop, ask about line two when the documents cover only line one. The desired behavior is an explicit scope limit and referral to the owner.

This exercise establishes behavior for the tested documents and paths. Keep untested connectors and interfaces visible as remaining work. The operator knowledge management guide covers the wider record. This worksheet keeps one transition inspectable within it.

Close the record when authority is settled, copies are reconciled, and observed retrieval matches the decision. Preserve the old and new revision identifiers with the verification date.

The answer follows the approved source.

Sources

Sources checked October 28, 2026. The source states, fictional workshop, reconciliation worksheet, and answer-withholding rules are proposed editorial methods.

Topic map: AI knowledge systems and tools

Articles in this cluster, then the 3 clusters that share the most topics with it.

This cluster: AI knowledge systems and tools

Neighbor: APIs, webhooks, and automation

Neighbor: Topical authority and keyword strategy

Neighbor: CRM and pipeline data