Executive Summary

Paper 13 examined the moment an AI recommendation becomes an institutional decision. Scale changes the problem. A team may review an individual case carefully while the organisation lacks any reliable account of hundreds of similar cases, recurring errors, the people who never reached a reviewer, or whether a vendor update has altered outcomes. Each decision can appear defensible while the system of decisions becomes difficult to govern.

The answer is not to demand that a senior executive inspect every outcome. 

Institutions need a defined operating boundary for each AI-supported workflow, records that make patterns visible, sampled review of ordinary cases as well as exceptions, routes for challenge that can correct both an individual outcome and a systemic fault, and authority to pause or narrow use when evidence changes. The volume of decisions should determine the strength of the feedback system, not dilute responsibility.

Australian Government policy assigns accountability to AI use cases and requires impact assessment for in-scope non-corporate Commonwealth entities. Its automated decision-making guide addresses ongoing assurance after deployment. These are government-specific requirements and guidance, but they expose a question for every regional institution: who knows whether a decision process still performs as intended after its thousandth use? [1][2][3]

Scale Changes What Must Be Seen

Imagine a regional service organisation using a model to sort incoming applications. 

Staff review the cases flagged as uncertain, while apparently straightforward cases pass through quickly. The first month brings few complaints. That may indicate good performance. It may also mean that rejected applicants do not know AI shaped the result, that the review channel is inaccessible, or that an input error repeatedly affects a group whose cases never reach the exception queue. Complaint volume alone cannot distinguish these possibilities.

At high volume, no individual sees the whole process. Attention is divided among intake staff, reviewers, managers, technology teams and vendors. The organisation needs records of inputs, system versions, human interventions, outcomes and corrections appropriate to the stakes of the workflow.

The governance unit is the decision process: who sets rules, how cases enter the queue, what evidence is presented, how outputs are acted upon and how challenges change future decisions. Model performance at purchase cannot account for all of this.

Set the Boundary Before Increasing Volume

An institution should describe the exact decisions a system may influence, the population to which it applies, its authorised inputs, permitted outputs and points where a person must intervene. It should record what the system is prohibited from doing. A change to the population, data source, threshold, model version or final action is a change to the governed use case even if a vendor calls it a routine update.

A boundary might permit prioritising incomplete applications for staff contact but prohibit automatic refusal. It might allow a draft summary while requiring inspection of underlying documents before an adverse decision. Defined uses make monitoring possible. Without them, volume quietly expands AI's role through habit.

The Australian Government AI impact assessment tool is designed to assess use-case impacts and risks and embed measures to address them. A regional organisation outside its mandatory scope can borrow the discipline of reassessing a use case when its context changes, while applying its own legal and sector-specific duties. [2]

Keep Three Levels of Accountability Connected

Case accountability belongs to the person or body authorised to make or review a particular decision. Process accountability belongs to the owner who can examine the pattern of outcomes and repair the workflow. Institutional accountability belongs to leaders who set risk tolerance, provide resources and decide whether the use continues. Assigning a name at only one level creates a gap. A frontline worker cannot repair a vendor integration; a board cannot adjudicate each application.

A reviewer who notices repeated missing evidence must flag a process issue. The owner must identify affected decisions, consider re-examination and alert leaders when the operating boundary fails. Leaders must be able to pause use despite operational targets.

The Australian Government Standard for accountability calls for accountable officials and designated accountability for in-scope AI use cases. Its structure is particular to Commonwealth agencies. The wider principle is that responsibility needs an identifiable holder with the power and information to act. [1]

Sample the Ordinary, Not Just the Alarms

Exception queues are useful, but they show only the cases the system or staff recognised as exceptional. If the criteria for flagging are wrong, the most consequential misses may sit among routine approvals and refusals. A monitoring plan should sample ordinary cases, borderline cases, overturned cases and cases from groups or locations where the evidence suggests different effects. The sampling method and review frequency should respond to consequence and volume.

Reviewers should reconstruct selected decisions: Was information correct and complete? Did the decision-maker see it? Was the reason supported independently of the generated explanation? Sampling can expose patterns hidden by aggregate scores; dashboard patterns also need case-level examination before causal claims.

NIST's voluntary AI Risk Management Framework treats risk management as continuous across the lifecycle. Its Manage function includes post-deployment monitoring, user input, appeals, override, incident response and change management. That breadth is useful: monitoring is not merely a technical score, but a way for experience and challenge to alter an operating process. [4]

Measure Consequences and Capacity

A useful dashboard might track volumes, processing times, decisions by category, intervention rates, appeal and reversal outcomes, missing-data rates and the time taken to remedy errors. Measures should be disaggregated where lawful, appropriate and statistically meaningful. Small numbers may reveal private information or produce misleading variation. The owner should record how each measure is defined, its limitations and the action it might trigger.

The capacity of human oversight also matters. If applications double while qualified reviewers remain constant, an unchanged override rate may conceal rushed scrutiny. Track workload, time available per case, requests for additional evidence and reasons given for overrides. Listen to staff when they report that the interface makes a particular decision difficult to question. A nominal review step can remain in the workflow long after it stops functioning as review.

Targets can distort behaviour. Rewarding low appeal rates may discourage appeals; rewarding automation may discourage intervention. No single number establishes fairness. Use measures as prompts for inquiry and check what they miss.

Make Challenge Feed System Repair

An affected person needs a route to ask what happened and present missing or corrected information. The review should be conducted by someone able to reconsider the evidence and change the result. When a challenge succeeds, the institution should ask a second question: could other people have been affected in the same way? That question converts individual redress into organisational learning.

A correction process should distinguish a one-off data error from a rule, interface, training or model problem. It should record the population potentially affected, whether past decisions require review, who authorises remediation and how the organisation will know the repair worked. Where an error may cause ongoing harm, waiting for the next periodic report is inadequate. The process owner needs a route to suspend the affected use immediately.

Australia's AI Ethics Principles identify contestability when an AI system significantly affects a person, community, group or environment, and call for identifiable accountability across the lifecycle. They are principles rather than a universal legal code. The practical lesson is that a challenge channel must be connected to those who can change the system as well as those who can change a single decision. [5]

Govern Vendors and Change

Many organisations do not build their own models. They still choose the context, data, thresholds, staff workflow and action taken on the output. A supplier may be responsible for a defect, but the institution remains the place where a person experiences the decision. Contracts and operating arrangements should make it possible to obtain version histories, relevant logs, performance information, incident notices and support for investigating affected cases.

Before a material update, the owner should test whether previous approval still applies. A change in model, prompt, data feed or interface may alter which cases receive attention despite stable performance figures. Retain the ability to test, roll back and suspend.

The EU AI Act places specific monitoring and logging obligations on deployers of high-risk systems within its scope. Those obligations should not be described as Australian law. They nonetheless demonstrate why a deployer needs access to operational evidence, rather than accepting a supplier's pre-launch assurance as the whole governance arrangement. [6]

What This Means for Ballarat

A small regional organisation may have neither a specialist assurance team nor enough cases to draw confident statistical conclusions. The answer is a proportionate shared discipline, not a thinner duty to affected people. A single workflow owner can maintain a use-case description, a small case sample, a change log, a review route and a brief standing report to leaders. Peer institutions can exchange questions and methods without exchanging sensitive case records or surrendering their own decision rights.

BRAIN can study patterns across regional settings, develop practical governance questions and help institutions build the capability to interpret what they observe. It should not become the hidden approver of another organisation's decisions. The useful regional contribution is to make disciplined oversight attainable where specialist capacity is scarce, while each institution retains ownership of its process and remedies.

Questions for Leaders

What decisions can this system influence, and at what volume? Who owns a case, the process and the decision to continue operating? Which apparently ordinary cases are sampled? Can a challenge correct both an outcome and a recurring fault? What change by a vendor would trigger reassessment? Who can pause the use today, and what evidence would justify restarting it?

Conclusion

Scale turns isolated choices into an institutional pattern. A signature on each case does not reveal whether the pattern is acceptable, and a favourable aggregate score cannot explain a harmful individual result. Governing both requires boundaries, traceable decisions, sampling, challenge, change control and a person with authority to intervene.

The test is whether the institution can notice when its decision process has changed, identify whom that change affected, correct past outcomes where necessary and adapt future practice. That is how accountability survives routine use: the organisation remains capable of learning from the decisions it makes.

About BRAIN

The Ballarat Region Artificial Intelligence Network (BRAIN) is a regional institution focused on understanding the impact of artificial intelligence across Ballarat and surrounding communities and strengthening the capability required to respond. Through research, publications and regional collaboration, BRAIN examines how AI can contribute to regional prosperity and wellbeing while helping institutions navigate its risks and wider consequences.

About the Author

Matt Bowd is Co-Founder and Chief Executive Officer of the Ballarat Region Artificial Intelligence Network (BRAIN). His work focuses on AI governance, institutional capability and the implications of artificial intelligence for regional organisations and communities.

Sources

[1] Australian Government, Standard for accountability.

[2] Australian Government, AI impact assessment tool.

[3] Australian Government, Automated decision-making: better practice guide.

[4] US National Institute of Standards and Technology, AI Risk Management Framework Core, Manage 4.1.

[5] Australian Government, Australia's Artificial Intelligence Ethics Principles.

[6] European Union, Regulation (EU) 2024/1689, Article 26: Obligations of deployers of high-risk AI systems.

Next in the Series

Governing AI Exceptions and Appeals

The next paper will examine what institutions can learn from contested decisions, exceptional cases and appeals, and how to design remedies that remain accessible as AI becomes embedded in ordinary services.

Written by Matt Bowd, Co-Founder and Chief Executive Officer of the Ballarat Region Artificial Intelligence Network (BRAIN).

Each study is a step toward a more intelligent and resilient region.

To participate in regional pilots or research partnerships, in our region or yours, connect via matt@brain.net.au.