The Assurance Desk
Part of the guide Do you need AI governance software, or is an adviser enough?
Start with the job, then assess the tool
Write down the problem you need to solve. GOV.UK’s AI buying guidance recommends doing this and staying open to other solutions. It was written for public-sector buyers. The questions here use those practical ideas for a UK firm’s buying discussion, without treating them as rules for every private business.
If personal data is involved, the ICO says buyers should assess the trade-offs themselves. Ask what your firm gains, what risk people could face and what evidence would help you decide.
Its AI guidance is under review after the UK’s Data (Use and Access) Act. Keep that in mind when checking how it applies to a particular purchase.
You are also required to specify your requirements at the procurement stage, rather than addressing trade-offs afterwards.
What should you ask about data and human checks?
Ask the supplier to describe the full route through the tool: what goes in, what happens to it and what comes out. The ICO says an AI privacy assessment should cover how data is collected, stored and used, alongside its volume and sensitivity.
The prompts below are questions to adapt to your intended use. A clear answer should describe the product and settings you plan to buy, rather than AI in general.
| Ask the supplier | Evidence to request |
|---|---|
| What data does this use need, and why? | A list of inputs, their purpose and the expected outputs. |
| Where does the data go, and who can use it? | A diagram of data flows and an explanation of access. |
| Can our inputs or outputs be reused to train models? | The relevant terms and settings, including any choices we can make. |
| Which decisions will a person check? | A description of the review step and who carries it out. |
| Can that person change the result? | A demonstration of how review works in this use. |
What would a useful test show?
Ask to test the task your team will use the tool for. GOV.UK’s guidance suggests checking how the model performs under different conditions, what acceptable performance means, how unfair outcomes are addressed and what security is in place.
For example, a firm considering a tool to draft customer replies could ask it to handle a deliberately awkward sample request. The questions are simple: did it answer the right point, what went wrong, and could a reviewer put it right? This is an example test discussion, not proof that a tool is suitable.
Where a person reviews an automated decision, the ICO says that review should be meaningful and decisions should be able to be overturned. Ask to see that process work, rather than relying on the phrase “human review”.
Agree what a useful test needs to show
- Define the task What problem should the tool solve?
- Set the checks What results would be acceptable, and under which conditions?
- Review the evidence What did testing show, and what still needs checking?
- Keep checking How will performance be monitored once the tool is used?
What happens after you start using it?
Ask who will help when staff get stuck and what that support costs. GOV.UK’s guidance says AI may need support throughout its life. It also recommends training so staff understand the tool and can act on its results.
Agree how performance will be checked after launch. The ICO says outsourced services should be reviewed regularly. If the use stops meeting data protection requirements in your circumstances, you should be able to change the service or move to another provider.
| Ask this | Request this |
|---|---|
| Who trains our staff and helps them later? | The training plan, support contact and charges. |
| Who checks whether the tool still works well? | A named owner and an agreed review process. |
| How will we hear about changes to the tool? | The change-notice process and a way to review the effect on our use. |
| What happens when we leave? | Written roles and steps for ending the service and dealing with the data. |
What if the answers are unclear?
Keep a list of open questions. Ask the supplier to show the missing evidence or explain what it cannot yet provide. That gives both sides something concrete to resolve before a decision.
If you cannot assess whether the tool’s use would meet data protection requirements, the ICO recommends choosing a different solution. A trial or a polished demonstration does not answer the questions your assessment still needs to resolve.
What to do
- Describe the task before choosing the tool.
- Ask for evidence about data, tests and human review.
- Agree training, support and reviews after launch.
- Set out who does what when the contract ends.
Common questions
Are these questions only for firms building AI?
No. The ICO’s guidance covers buying an AI solution from a third party as well as outsourcing it. A firm choosing an existing tool still needs to assess the proposed use.
Can we just accept the supplier’s privacy assessment?
Use it to help answer your own questions. The ICO says buyers need an independent evaluation of the trade-offs. The assessment needs to fit how your firm will use the tool.
Where next
Sources
- ICO, What are the accountability and governance implications of AI?, checked 8 October 2026
- GOV.UK, Guidelines for AI procurement, checked 8 October 2026
This is an independent editorial guide, not legal advice.