Governance Insights Series - Paper #05: Governing AI Procurement and Vendor Relationships
Executive Summary
An organisation can inherit AI through a new purchase, an update to existing software or a service delivered by another institution. Each route can affect data, decisions and work long after the initial approval. Procurement is therefore a consequential point at which governance becomes concrete: requirements are specified, evidence is requested, responsibilities are divided and future choices are constrained.
This does not mean every supplier is a strategic partner or that procurement is the first governance decision. The organisation should already know the purpose of the proposed use, the people affected and the limits of acceptable risk. Procurement tests whether a particular supplier can meet those conditions. The depth of that test should reflect the consequences of the use.
For regional organisations, the central question is not simply which product performs best in a demonstration. It is whether the institution will retain enough understanding and control to govern the system as it changes. Leaders need evidence about performance in their setting, clear terms for data use and incident response, visibility of subcontractors and material updates, and a workable route to reduce dependence or exit.
Good procurement leaves capability behind. Staff learn to challenge claims, document decisions and oversee suppliers. Where appropriate, those lessons can strengthen later decisions across a region without requiring institutions to buy the same product or surrender their own accountability.
Before the Market Is Asked
Procurement begins with a defined problem. A team may want to reduce administrative work, improve access to information or support a professional decision. It should describe the desired outcome and examine whether AI is necessary for it. If a workflow problem can be solved by clearer forms or better information access, a sophisticated model may add cost and risk without improving the service.
Leaders also need to identify the people affected and the decisions the system may influence. A drafting assistant for internal material requires different scrutiny from a tool that ranks applicants or recommends access to services. The same platform may support both. A product label cannot determine the appropriate safeguards; the actual use does.
The organisation should therefore set its requirements before a vendor’s features become the agenda. Who owns the use? What information may enter the product? Where must a person be able to question or override an output? What evidence would justify proceeding, and what result would cause the institution to stop? These questions connect procurement to Responsibility, Stewardship and Accountability in the BRAIN Governance Pathway.
Evidence, Not Demonstration
Vendor demonstrations show what a product can do under selected conditions. They rarely establish how it will perform with an institution’s data, exceptions, workforce or obligations. Buyers should ask for evidence of testing relevant to their proposed use, known limitations and the conditions under which performance may change. The National AI Centre advises organisations buying an AI system to ask suppliers for proof of testing and to assess each use in its own context.
The evidence requested should be proportionate. For a low consequence writing aid, a bounded trial and checks on information handling may be enough. For a system used in safety, employment or essential services, leaders may need deeper testing, specialist review and an explanation of how affected people can contest outcomes. An impressive average result cannot excuse a serious failure on the cases that matter most.
Procurement teams should involve the people who understand the work. A service manager can identify unusual cases; frontline staff can test whether review is feasible; privacy and security advisers can examine data flows. Supplier claims should be recorded alongside independent observations and unresolved questions. The outcome might be purchase, a narrower use, further testing or a decision not to proceed.
This is also where the workforce trust discussed in Paper No. 4 becomes operational. If employees are expected to rely on, check or explain a system, they need an opportunity to test it before its purchase makes the operating model difficult to change. Their objections may reveal a design flaw rather than a training need.
Information and Institutional Knowledge
Data residency is one important question, but location alone does not explain what a provider may do with information. AI products may process prompts, uploaded documents, outputs, usage records and metadata in different ways. Buyers need to establish what is collected, how long it is kept, who can access it, whether it can be used to improve a product and what happens at the end of the relationship.
The Office of the Australian Information Commissioner’s guidance on commercially available AI products addresses both privacy obligations and product selection. An institution should assess its own duties rather than assume that a supplier’s general security statement settles the question. Sensitive information may also appear indirectly in prompts or outputs, even when a team believes it is merely testing a tool.
Information has operational value beyond individual records. Prompts, workflows and corrections can reveal how a service works and where staff exercise judgement. A supplier may provide genuine benefits from processing this information, but the institution should know which uses are permitted, what controls are available and whether it can retrieve or delete its material. Contract terms and configuration need to support the intended practice, not merely describe it.
Dividing Responsibility Across the Supply Chain
An AI product may involve a software vendor, a model provider, cloud infrastructure and an implementation partner. The buyer may never contract directly with every participant, yet those arrangements can affect security, data handling, performance and response to incidents. The National AI Centre’s implementation guidance calls for accountability across the AI supply chain to be identified and communicated.
The contract should distinguish what the supplier undertakes to do from what the institution must do itself. Who tests the system before release? Who informs the buyer of an incident or material model change? What assistance is available to investigate an adverse outcome? What information can the institution inspect or share with an auditor? Where a subcontractor changes, what notice or approval is required?
No clause transfers the institution’s responsibility for a decision affecting its people or community. Equally, a buyer cannot govern responsibly if it accepts an arrangement that leaves it unable to obtain important evidence. Leaders should identify gaps before signing and judge whether they are tolerable for the proposed use. The Australian Cyber Security Centre’s guidance on AI supply chain risks reinforces the need to examine dependencies beyond the direct supplier.
Governing Change After Purchase
AI-enabled software can change without a new procurement. Models may be replaced, features enabled by default, pricing revised or data practices altered. A purchase approved for one purpose can gradually support another. These changes make contract management a continuing governance task rather than a final administrative step.
Before committing, an organisation should establish which changes require notice, renewed testing or approval. A new interface colour is different from an altered model used in a consequential decision. Managers need a way to identify material changes, compare them with the approved use and pause deployment where evidence is insufficient. Staff should know whom to tell when outputs behave differently.
Performance measures should reflect the institution’s purpose. Availability and response time may matter, but so do error patterns, effects on workers and complaints from people using a service. Supplier reports can inform review; the organisation also needs its own observations. The right to inspect, challenge and change course has little practical value if nobody is assigned to exercise it.
Exit Is a Governance Capability
A supplier relationship may end because the product disappoints, costs rise, needs change or trust is lost. Leaders should ask at the outset what the institution would need to continue its work. Can it export its data in a usable form? What records of decisions and system outputs will it retain? How will access be removed, information deleted and staff supported through a transition?
The answer may be a different provider, an internal process or a return to human work. None is effortless if knowledge has become embedded in a proprietary workflow. A realistic exit plan limits dependence and strengthens the institution’s bargaining position without assuming that every service must be built locally. Resilience means being able to withdraw from a use without losing the ability to serve people responsibly.
A Ballarat Scenario
Consider a hypothetical Ballarat community organisation assessing a supplier’s AI tool for triaging incoming requests. The demonstration sorts common enquiries quickly. Staff, however, know that a small group of requests contains complex circumstances where a misplaced priority can delay help. The supplier offers overall accuracy figures but little evidence on these exceptions.
A governance-led procurement would define the triage task and test representative difficult cases before purchase. It would establish which requests must be reviewed by a person, what information may enter the tool, how errors are reported and whether staff can override a priority. The contract would address changes to the model, access to records and support for investigation. The organisation might adopt a limited use with human review or decline the product if the evidence remains inadequate.
The case describes no actual Ballarat organisation. It shows why a promising demonstration is the start of a procurement inquiry rather than its conclusion.
Regional Capability
Regional buyers can benefit from sharing appropriate procurement questions and lessons. A council, health service and manufacturer have different requirements, yet each may learn how another tested supplier claims, specified data rights or handled a product update. Such exchanges should respect confidentiality and leave each institution responsible for its own decision.
BRAIN can help make recurring questions visible and develop regional capability to examine them. Better local judgement does not require a preference for a local supplier regardless of quality. It requires institutions to retain expertise, compare evidence and learn from experience so that each subsequent procurement begins from a stronger position.
Conclusion
Procurement turns governance intentions into obligations that can be tested and enforced. It links the purpose of an AI use to the evidence a supplier provides, the information it may handle, the changes it must disclose and the control an institution retains over time. A good decision may be to proceed, narrow the use, seek further evidence or walk away.
For regional leaders, the question is what the relationship will leave behind. Responsible procurement should deliver a useful service while strengthening the institution’s ability to understand, challenge and govern it. That capability matters long after the original product choice.
About BRAIN
The Ballarat Region Artificial Intelligence Network (BRAIN) is a regional institution focused on strengthening the capability required for communities and organisations to prosper in the age of artificial intelligence. Its Governance Insights Series offers practical, regionally grounded thinking for boards, executives and institutional leaders navigating AI-enabled change.
About the Author
Matt Bowd is CEO of BRAIN. His work focuses on AI governance, institutional capability and the implications of artificial intelligence for regional organisations and communities.
Sources
National AI Centre, Guidance for AI Adoption: Foundations.
National AI Centre, Guidance for AI Adoption: Implementation Guidance.
Next in the Series
BRAIN Governance Insights Paper No. 6 — Leading Institutions in the Age of Artificial Intelligence. The next paper examines the leadership capabilities needed to keep institutions adaptable, trusted and accountable through sustained change.
Written by Matt Bowd, CEO of the Ballarat Region Artificial Intelligence Network (BRAIN).
Each study is a step toward a more intelligent and resilient region.
To participate in regional pilots or insights partnerships, in our region or yours, connect via matt@brain.net.au.