A good demonstration ends with an answer on a screen.
A buying decision needs a different record: what enters the system, who controls it, what happens when it fails, and who can stop it.
Start the AI vendor review with one job. Keep the evidence beside the decision. A folder of security documents cannot decide whether a particular workflow belongs in a particular product.
This is a proposed procurement worksheet for a small team. It uses blank questions and a fictional pilot. It makes no claim about any named vendor's controls.
Define the job before the questionnaire
Write a task boundary that another person can enforce.
In a fictional equipment supplier, two staff members want to turn approved product sheets into draft internal answers. The pilot uses synthetic sheets. A supervisor checks every answer. The tool cannot send messages, change records, or promise availability.
That boundary gives the review something concrete to examine. A system that drafts internal answers has a different permission requirement from one that sends them.
Use this opening record:
| Field | What to write |
|---|---|
| Intended task | The specific draft or analysis the tool produces |
| Input types | Documents and fields it may receive, with excluded information named |
| Users | People who can submit, review, administer, and remove access |
| Output destination | Where drafts appear and who can approve further use |
| Unacceptable outcome | A concrete event that stops the pilot |
| Business owner | One person accountable for the decision |
| Fallback | How the task proceeds when the tool is unavailable |
Make exclusions physical. If the pilot excludes customer records, prepare a separate folder containing only permitted files. Do not rely on a reminder above an unrestricted upload button.
For the fictional supplier, an unacceptable outcome includes an answer that invents a product specification and presents it as verified. Uploading a prohibited document also stops the exercise. Those are proposed boundaries for this example, not universal thresholds.
The broader LLM operations framework for small teams covers ongoing use. This worksheet handles the evidence needed before granting a pilot permission.
Keep four kinds of evidence separate
Create four columns in the review notes: contractual commitment, documented control, marketing statement, and pilot observation.
A contractual commitment records what the applicable agreement says. Record the document version and the reviewer who checked its scope. Have the appropriate adviser resolve contractual interpretation when necessary.
A documented control explains how something is supposed to work. Ask which product, plan, region, and configuration the document covers. A control described for one service does not answer a question about a different service.
A marketing statement is a lead for another question. Turn “enterprise ready” into requests about access, logging, incident handling, and support. Keep the original wording beside the evidence request so the answer addresses the claim.
A pilot observation records what happened under stated conditions. “The reviewer could not open the restricted file in this test” is useful. It does not establish that every access path is secure.
Keep disagreements visible. If a demonstration and a policy document appear inconsistent, ask the vendor to explain the difference in writing. Do not average conflicting answers into a passing score.
NIST describes the AI Risk Management Framework as voluntary guidance. Using this checklist, or referring to NIST, is not certification, legal advice, or proof that a vendor is safe.
Turn each question into an evidence request
An unanswered question should have a place to land.
Use the following worksheet as a starting point. Each row needs a named person, a date, and a remaining gap. Replace role labels with names inside the private purchasing record.
| Question | Evidence to request | Evidence owner | Freshness and unresolved gap |
|---|---|---|---|
| What information does this task send? | Data-flow description and actual field list | Internal workflow owner | Checked date; undocumented fields |
| Can our inputs or outputs be used for model improvement? | Applicable terms, settings, and written scope | Vendor contract contact | Version date; exceptions or conflicting terms |
| Who can read or administer our workspace? | Role documentation and a demonstration of assigned permissions | Vendor technical contact | Product version; untested access paths |
| How long is information retained? | Retention policy covering the selected service | Vendor privacy contact | Effective date; logs and backup exclusions |
| How does deletion work? | Procedure, scope, timing, and available confirmation | Vendor privacy contact | Response date; copies outside the procedure |
| Which third parties process information? | Applicable provider list and change process | Vendor contract contact | List date; unclear services or locations |
| What happens during an incident? | Notification route, response process, and responsibility boundaries | Vendor security contact | Review date; unanswered notification questions |
| What support applies to this plan? | Support terms and escalation instructions | Internal purchasing owner | Plan date; unavailable support periods |
| What can we recover on exit? | Export documentation and a permitted sample export | Internal continuity owner | Test date; missing artifacts or metadata |
Do not fill the last column with “current.” Write the date checked and the event that should trigger another check. A changed contract, plan, integration, or data scope can reopen the row.
For documents the vendor cannot share, record the refusal and any alternative evidence offered. Restricted evidence is still an unresolved limitation until the purchasing owner decides how it affects this task.
Follow the data through the whole task
Ask about the input, the generated output, and the surrounding records separately.
For the fictional supplier, the file itself is one artifact. The prompt, saved conversation, application log, support attachment, and exported answer are additional things to ask about. Do not assume they share one retention rule.
Ask who may use each artifact, for which purposes, under which terms. If an opt-out or administrative setting is offered, verify its availability for the selected plan. Record who applies it and how the team checks it before use.
Bring retention and deletion into the same conversation. Ask when the ordinary retention period begins, what deletion covers, what exceptions remain, and what evidence is available afterward. A missing answer stays missing.
Treat legal or regulatory requirements as questions for qualified review in the relevant business context. This article establishes no retention period or legal entitlement to deletion.
Keep third parties in the data map. Ask which service performs the model call, file storage, support, and connected actions. Ask how changes are communicated. The NIST Govern playbook addresses third-party AI risk and contingency planning. The worksheet here translates that concern into purchasing questions.
Test permissions with harmless material
Ask for the smallest permission set that can perform the intended task.
In the fictional pilot, the writer role reads the approved synthetic folder. The reviewer checks drafts. The administrator manages access. No role needs a live customer database connection.
Have the vendor explain what each role can do. Then test permitted actions and expected refusals using harmless material. Record the account role, setting, action, observed result, and any uncertainty.
Check removal as well as invitation. What happens when a user leaves the pilot? Who removes access? What happens to shared links, saved integrations, and that user's artifacts? Verify the available procedure rather than assuming one button covers everything.
Do not add production permissions to make a demonstration easier. A pilot that needs broader access has changed its boundary. Return it to the owner for a new decision.
If the vendor proposes local deployment as the answer, review its operational requirements separately. The local LLM deployment article provides related architecture context. Local deployment still needs a defined access and support model.
Check failure handling before testing convenience
Request the documented path for reporting a problem, escalating it, and suspending use.
Separate support availability from a resolution commitment. Ask what the selected plan provides, which channel applies, and who on your team can use it. Keep any promised response beside the applicable terms.
For reliability, define what an interruption means for this task. Can staff return to approved documents? Are partially completed drafts identifiable? Can the team tell which answers were reviewed before the interruption?
Use a fictional failure exercise. Make one synthetic sheet omit a requested specification. Ask the tool the corresponding question. Record whether it identifies the missing information, invents an answer, or produces an unclear result.
The exercise tests a particular case. It does not estimate a general error rate or prove resistance to every attack. Expand the review when the intended task or its consequences require more evidence.
NIST's Manage playbook addresses deployment decisions and ongoing third-party monitoring. Use that as a reason to keep a decision record after the demonstration ends.
Write a pilot permission with an expiration
Use a short permission record instead of a broad approval email.
For the fictional supplier, the proposed pilot lasts five working days. Two named users work with a fixed synthetic document set. One supervisor reviews every output. No integration writes to another system.
Before starting, the owner records the conditions for continuation. These could include correct handling of missing information, traceable document references, enforced access boundaries, and a successful export exercise. The team chooses the criteria for its task before observing results.
Stop conditions need an action attached. If prohibited information enters the pilot, suspend further submissions and follow the agreed response procedure. If an answer invents a specification, hold that output and record the failure. The owner decides whether remediation justifies another bounded test.
The expiration forces a decision. Continue under the same restrictions, approve a separately defined expansion, request more evidence, or end the pilot. Silence grants no additional access.
Use the operations automation audit to place the task beside other work before expanding it. A promising demonstration does not establish that this is the next workflow the business should fund.
Leave a decision someone else can inspect
Finish with a purchasing record containing the task boundary, evidence worksheet, applicable document versions, pilot observations, unresolved questions, and owner decision.
Describe the limits of that decision. “Permitted for synthetic internal drafts under these conditions” is inspectable. “Approved AI vendor” leaves the next user to invent the scope.
Keep the worksheet open when evidence is incomplete. A vendor may still fit a low-risk task with tight restrictions. Another task may need an answer the vendor cannot provide. Record that judgment without pretending the gap disappeared.
The evidence belongs beside the permission.
Sources
- NIST AI Risk Management Framework: voluntary framework scope.
- NIST AI RMF Playbook: Govern: third-party risks and contingency planning.
- NIST AI RMF Playbook: Manage: deployment decisions and monitoring.
Sources checked October 28, 2026. The worksheets and fictional pilot are proposed editorial methods.