A document moves behind a locked door. The answer engine still has a copy.
That is the access question a RAG demo needs to answer. Remove a user from the source system. Then check what that user can retrieve through the assistant.
The test has several destinations: indexed chunks, search responses, conversation history, cached answers, and citation previews. Write the expected result for each destination before running the exercise.
This article proposes a review method for an internal knowledge assistant. The examples are fictional. Azure AI Search documentation supplies the product-specific distinctions. The worksheet supplies questions your application owner must resolve.
Separate identity from document permission
Start with four records.
Authenticated identity establishes who the application believes is asking. Source permission records who may read a document in its original system. Indexed permission metadata describes the access information copied into search. Query-time enforcement decides which indexed material this request can receive.
Give each record an owner and an update path. A successful sign-in answers the first question. It leaves the other three open.
For a proposed review, write down the identity provider, source repository, indexed document identifier, and enforcement mechanism. Add the point where the application obtains the caller's identity. A name typed into a chat box is input to inspect, never sufficient evidence for granting access.
The broader RAG architecture guide describes retrieval and generation. Here, narrow the task to permission changes after ingestion. A useful search result still needs an access decision before it becomes model context.
Treat the source document and every derived chunk as a family. In the worksheet, record the parent identifier beside the chunk identifiers. A permission review needs to find every member of that family.
Choose the enforcement mechanism explicitly
Microsoft's document-level access overview distinguishes application-managed security filters from native identity-based controls. Security filters compare strings. Native ACL and RBAC support, Purview label enforcement, and SharePoint ACL ingestion are documented as preview capabilities.
Preview status matters to the implementation decision. The overview states that preview features lack a service-level agreement and may change. Microsoft does not recommend them for production workloads. Record the selected feature, connector, REST API version, and SDK package. Check their support together before relying on a sample.
In the security filter pattern, the principal identifier is a string used by an OData filter. That string does not authenticate the user. The application must apply the security filter to every query.
This distinction determines the review boundary. With application-managed filters, inspect the trusted identity lookup and every retrieval path that constructs a query. Include alternate search endpoints, follow-up questions, and background summarization. An optional filter parameter is a permission decision waiting to be omitted.
With native enforcement, inspect the documented configuration and the permission metadata supplied to it. In either design, write down how the application prevents ordinary users from bypassing its retrieval path.
Keep administrative diagnostics outside the ordinary assistant workflow. A reviewer using broad administrative access cannot demonstrate what a restricted user receives. Run the user-facing tests with the intended identity and record that identity's role.
Walk one revocation through the whole system
Consider a fictional supplier with a restricted document called Prototype Assembly Notes. It contains harmless test text: “Calibration marker: violet hinge.”
Mara belongs to the prototype group. Ivo does not. Before the exercise, Mara can read the notes and Ivo cannot. The document produces four indexed chunks. Those names, people, and contents are synthetic fixtures.
First, ask both users the same question. Record the search results, context selected for generation, visible answer, and citation behavior. An empty answer alone does not show whether restricted text reached the model.
Next, remove Mara's only grant to the document. Confirm the source application now denies her access. Record when the change happened and which grant was removed. Check for another direct or inherited grant before labeling the expected result “denied.”
Repeat the question in a new conversation. Repeat it in the conversation that previously produced an authorized answer. Try a direct request for the calibration marker and a paraphrase about the assembly procedure.
For this exercise, the desired result is no new disclosure from the restricted notes. That includes the marker, a paraphrase of the restricted instruction, and a revealing citation preview. State this as the application's acceptance condition.
Finally, repeat a permitted question against an unrestricted document. This confirms whether the observed refusal belongs to the restricted document or to a broken assistant that answers nothing.
Keep each observation attached to its request. One successful refusal cannot stand in for every query, session, or interface.
Measure the permission update path
Do not assign one universal revocation delay to RAG systems.
Separate a change to a document's access list from a change to a user's group membership. In the fictional exercise, record whether Mara leaves the group or the group loses access to the document. Test those events separately. For membership changes, inspect identity resolution and any cached membership results as well as indexed permissions.
Azure's query-time enforcement documentation describes different refresh paths. ADLS Gen2 ACL updates require a resync. Custom push ingestion requires reingesting affected documents. Native authorization depends on indexed permission metadata.
For SharePoint, the permission ingestion documentation places incremental ACL updates at 2026-05-01-preview or later. Earlier preview versions need explicit reindexing for later permission changes. Changes inherited from parent scopes require an explicit refresh. Its current prerequisites name 2026-08-01-preview or an equivalent preview SDK.
Translate those documented limits into questions for the selected deployment. What event notices a permission change? What refresh updates the affected documents? What evidence shows completion? What does the application permit while synchronization remains unresolved?
Set the application's acceptable exposure interval before observing results. If the chosen connector cannot support that requirement, change the design or hold the restricted collection. A successful ingestion job is one piece of evidence. Pair it with queries from the revoked identity.
Record source denial time, refresh completion time, and first verified retrieval denial separately. Label an untested interval unknown. A review should expose the gap rather than hide it behind “permissions synchronized.”
Define failure behavior before convenience behavior
For the proposed application, require a deliberate response to missing identity, missing document permissions, failed membership resolution, and incomplete refresh evidence.
Use a fail-closed rule for restricted collections: when authorization cannot be established, withhold their material and return a neutral access error. This is a proposed application requirement. It does not describe the default behavior of every search product.
Azure documents a narrower product behavior: when native ACL evaluation fails, it returns a server error without partially filtered results. Its current query guide says an omitted user token returns public documents only. Those guarantees depend on the documented native configuration. Check the query-time limits.
In an application-managed filter design, test a missing filter directly. Define which trusted component rejects that request. Do the same for a chunk with no permission metadata and a membership record whose freshness cannot be established.
Keep public status explicit. Do not let an empty permission field silently mean “everyone.” Ask the collection owner to approve how public documents are represented and tested.
Record the error shown to the user without copying restricted text into it. The message can explain that access could not be verified. It need not identify a restricted project's contents.
Review caches, history, and citations separately
A search permission test covers the retrieval path you exercised. Extend the proposed review to every place the application can reuse an earlier answer.
Inventory answer caches, retrieval caches, saved conversations, generated summaries, downloadable outputs, and citation previews. For each, name the owner and the access check before display. Mark absent features “not present” after inspection.
For cached answers, ask what ties the entry to the user, source documents, and permission state. Decide whether a permission change invalidates the entry or requires a new authorization decision. Test the chosen behavior with Mara's old question.
For conversation history, specify whether users may reopen previously authorized text after losing access. That is a product decision requiring explicit ownership. Also decide whether the application may reuse that text as context for a new answer.
Revocation cannot make a person forget text already read. The review should distinguish prior disclosure from new disclosure through the application. Do not claim that a new retrieval refusal erased an earlier export.
For citations, inspect the displayed title, snippet, hover preview, and destination. A denied destination does not establish that its preview is safe. Use the harmless marker to check what the interface displays without opening the file.
The business knowledge systems article covers retained context. Add permission ownership to each retained artifact before calling the assistant ready for restricted material.
Use an access review matrix with observable evidence
The following matrix defines a fictional exercise. The evidence column names what to capture. It reports no observed production results.
| User | Document | Action | Expected result for this exercise | Observed evidence to record |
|---|---|---|---|---|
| Mara, authorized | Prototype Assembly Notes | Ask before revocation | Authorized answer with permitted citation | Request ID, selected chunk IDs, visible answer |
| Ivo, unauthorized | Prototype Assembly Notes | Ask for marker | No restricted context or disclosure | Search trace, generation-context record, rendered response |
| Mara, revoked | Prototype Assembly Notes | Ask in new session | No new disclosure after required refresh | Source denial time, refresh evidence, request trace |
| Mara, revoked | Prototype Assembly Notes | Reopen previous conversation | Matches the documented history decision | Conversation ID, display decision, context reuse evidence |
| Mara, revoked | Prototype Assembly Notes | Repeat cached question | No new disclosure through cache | Cache hit or miss, authorization decision, output |
| Mara, revoked | Prototype Assembly Notes | Inspect citation preview | No restricted preview content | Visible title, snippet, destination access result |
| Mara | Public Packing Guide | Ask permitted question | Answer remains available | Request ID, permitted source IDs, visible answer |
| Unknown identity | Restricted collection | Submit query | Restricted retrieval withheld | Identity failure and refusal evidence |
Fill the last column with actual observations during the permitted test. Use restricted review storage for traces. Retain identifiers and decisions without spreading document text into a general log.
Add rows for direct grants, group grants, inherited permissions, and each supported interface. Repeat the matrix when a connector, permission model, or retrieval path changes.
Leave a bounded acceptance record
End the review with the configuration inspected, test identities, document family, timestamps, observed results, and unresolved paths. Give each unresolved path an owner and a release consequence.
The small-team LLM operations guide provides the wider operating frame. This record supplies the narrower evidence for one access boundary.
Accept only the behavior the exercise demonstrates. If cached history remains untested, say so. If inherited permission refresh remains unresolved, keep that collection held. A green result for public documents cannot answer the restricted-document question.
Verify access at every copy.
Sources
- Azure AI Search document-level access overview: enforcement approaches and preview status.
- Azure AI Search security filter pattern: string filters and application responsibility.
- Azure AI Search query-time ACL and RBAC enforcement: refresh paths, query identity, and failure behavior.
- Azure AI Search SharePoint permission ingestion: version requirements and inherited-permission refresh limits.
Sources checked October 28, 2026. Examples, cache requirements, and the review matrix are proposed editorial methods.