Executive Summary

An AI system can recommend an outcome, rank people, summarise evidence or draft the reasons for a decision. An institution may then say that a person made the final call. That description is insufficient. The governance question is whether the person had the authority, information, time and practical freedom to reach a different conclusion.

A recommendation becomes an institutional decision when someone adopts it, acts on it or allows a workflow to act on it. Responsibility cannot be settled by the presence of a human signature. It depends on how judgement was exercised, how the result affected people, whether the organisation can explain its reasons and whether an affected person can challenge the outcome.

Australian Government guidance recognises that an action may be materially influenced by AI even when a human takes the final step. It calls for notification and a timely opportunity to challenge significant administrative actions. The National AI Centre also recommends accessible mechanisms to understand, challenge and appeal AI use and decisions. [1][2]

The Decision Point Is Often Hidden

Consider a regional employer using AI to review applications. 

The system places candidates into three groups and supplies a short rationale for each. A manager selects interviewees from the first group after briefly scanning the list. The employer may accurately say that the manager selected the candidates. Yet the model shaped the field of attention, the order of review and the apparent reasons for selection.

There is no single moment at which influence begins. A system may draft criteria, extract facts, rank options, flag exceptions or suggest an outcome. Earlier outputs can narrow the set of possibilities before the named decision-maker sees them. Governance must therefore trace the entire path from information to action, including what was excluded and who had a genuine opportunity to question the exclusion.

Human Approval Is Not Human Judgement

A reviewer cannot exercise independent judgement merely because a screen has an ‘approve’ button. 

They need to know the task the system was asked to perform, the provenance and limitations of its inputs, what the output means, and which matters still require their own inquiry. They must be permitted to disagree without being penalised for slowing the workflow.

Four conditions make human review credible. 

• First, the person has enough competence to recognise plausible errors and missing context. 
• Second, the interface presents source material and uncertainty rather than only a polished answer.
• Third, there is sufficient time and authority to request more evidence, change the outcome or stop the process. 
• Fourth, the institution can observe whether reviewers actually use those powers.

The EU AI Act addresses human oversight for high-risk systems by requiring arrangements that enable people to understand system capabilities and limitations, remain aware of automation bias, interpret outputs, and disregard, override or reverse them. Its obligations belong to the EU framework, but the design questions are useful to Australian organisations examining whether their own oversight is real. [3]

Design the Human Role for the Consequence

Not every use requires the same intervention. A drafting assistant for a routine internal note may need editorial checking. A tool that affects access to work, housing, finance, care or public services demands a stronger account of who decides, what evidence they inspect and how errors are corrected. Human involvement should be proportionate to the stakes and to the system’s ability to influence the result.

One pattern is ‘human before action’: no consequential outcome proceeds until a qualified person examines the case. Another is ‘human over the process’: automated handling occurs within defined limits, with monitoring and intervention for exceptions. A third is a deliberately automated decision under a lawful and properly governed arrangement. Organisations should name which pattern they are using. Calling all three ‘human oversight’ conceals important differences.

For each use, a decision-rights map should identify the system’s permitted contribution, the authorised decision-maker, the evidence available to that person, the conditions requiring escalation, the circumstances in which action pauses and the route for review. The map should also show who can change criteria or thresholds; upstream design choices may shape far more outcomes than any individual approval.

Reasons Must Belong to the Institution

A generated explanation is not automatically the reason an institution reached a decision. It may describe the model’s output fluently while omitting facts the decision-maker should have considered. If a person signs that explanation without checking it against the evidence and their own judgement, the record can obscure rather than reveal how the outcome arose.

For consequential decisions, the record should distinguish the AI contribution from the human assessment. It should identify the evidence accepted, material evidence rejected, uncertainty, relevant policy or criteria, the person with authority and the reason for the final result. A reviewer need not reproduce every intermediate prompt. They do need enough of the path to test whether the institution applied the right criteria to the right facts.

In public administration, reasons and review have established significance. The Administrative Review Council’s 2025 guide addresses best practice in preparing statements of reasons. AI makes that discipline more demanding because a plausible generated narrative can create an appearance of explanation without exposing the decisive reasoning. [4]

Make Contestability Work at the Point of Harm

A person affected by a decision should be able to find out that AI materially contributed where disclosure is appropriate, understand the basis of the action and reach a review pathway that can change the outcome. A complaint address alone is not enough if the reviewer sees only the original score or treats the first decision as presumptively correct.

Australian Government impact assessment guidance advises notification when an AI-influenced administrative action has a legal or significant effect, including information about available review rights. It also says ordinary review rights should not be limited because AI was used. National AI Centre guidance calls for accessible, understandable channels through which stakeholders can challenge AI use and decisions. [1][2]

For organisations outside that government setting, the same design question remains even where the applicable duty differs: could a person bring missing context, challenge an incorrect fact, and obtain a fresh judgement by someone with authority to remedy the result? A review process should record what was changed and whether a recurring system problem is indicated.

Australia’s forthcoming Privacy Act provisions add a distinct transparency requirement. From 10 December 2026, covered APP entities will need specified information in their privacy policies where a computer program uses personal information in making, or doing something substantially and directly related to making, a decision reasonably expected to significantly affect an individual’s rights or interests. This is a privacy-policy disclosure duty, not by itself a general right to human review. [5]

Test the Oversight, Not Only the Model

Pre-deployment model testing asks whether outputs are accurate or fair enough for an intended use. Governance must also test the combined system of people, interfaces, incentives and procedures. A highly accurate model can still produce poor decisions if staff overlook unusual cases, cannot see omitted evidence or treat the suggested answer as the default.

An organisation can sample decisions and ask:

• How often do reviewers overturn recommendations? 
• When they do, what information made the difference? 
• Are certain groups disproportionately moved to exception queues? How long does a review take? 
• Can staff explain the basis of a decision without copying the generated rationale? Do challenges reveal errors that routine monitoring missed? 
• No single metric proves meaningful oversight. Together, these observations can reveal where the human role has weakened.

Testing also needs a baseline. NIST’s AI Risk Management Framework notes that systems intended to augment or replace human activity may require human baseline measures for comparison. Institutions should assess the combined workflow against a credible alternative, including its speed, error patterns and consequences for affected people, rather than assuming that adding AI improves a process. [6]

What This Means for Ballarat

A regional organisation may have fewer specialist reviewers and a stronger reliance on vendor products. This can make a polished recommendation attractive and independent scrutiny difficult, particularly when decisions must be made quickly. The pressure is real in a small employer, service provider or community institution, but the absence of a large risk team does not make an affected person’s interests less important.

A workable local starting point is to select one consequential workflow and follow a case through it. Ask who defined the criteria, what the system did, what the person saw, what they could change and how an affected individual would contest the result. Write the answer in plain language and test it with staff who were not involved in procurement. Where the answers are unclear, narrow the AI use until the organisation can govern it.

BRAIN can help regional institutions develop shared questions, research the patterns that appear across sectors and build capability to distinguish recommendations from decisions. Individual organisations must retain responsibility for their own choices, legal obligations and remedies. Regional learning is valuable when it improves judgement within institutions, rather than replacing it.

Questions for Leaders

• Which outcomes does AI materially influence, including through ranking or omission?
• Who has authority to disagree, and can they access the underlying evidence? 
• What makes review independent in a busy working day? 
• Can the institution give a reason that is its own? 
• Can a person challenge an outcome in time for correction to matter? 
• What do actual case records reveal about how often human intervention changes the result?

Conclusion

An AI recommendation is a contribution to a process. An institutional decision is an act for which a person or body can explain, take responsibility and provide a remedy. The distinction survives only when the workflow makes independent judgement possible and the institution checks that it occurs.

As AI enters ordinary decisions, governance should follow influence rather than labels. The most telling question is not whether a human was somewhere in the loop. It is whether the institution could have reached a different decision for good reasons, knows when it did, and can revisit the outcome when someone shows it was wrong.

About BRAIN

The Ballarat Region Artificial Intelligence Network (BRAIN) is a regional institution focused on understanding the impact of artificial intelligence across Ballarat and surrounding communities and strengthening the capability required to respond. Through research, publications and regional collaboration, BRAIN examines how AI can contribute to regional prosperity and wellbeing while helping institutions navigate its risks and wider consequences.

About the Author

Matt Bowd is Co-Founder and Chief Executive Officer of the Ballarat Region Artificial Intelligence Network (BRAIN). His work focuses on AI governance, institutional capability and the implications of artificial intelligence for regional organisations and communities.

Sources

[1] Australian Government Digital Transformation Agency, “Guidance for the artificial intelligence impact assessment tool: Contestability.”

[2] Australian Government National AI Centre, “Guidance for AI adoption: implementation guidance.”

[3] European Union, Regulation (EU) 2024/1689, Article 14, “Human oversight.”

[4] Administrative Review Council, “Best Practice Guide: Statements of Reasons” (2025).

[5] Office of the Australian Information Commissioner, “Chapter 1: APP 1 Open and transparent management of personal information,” automated decision provisions.

[6] US National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework (AI RMF 1.0)” (2023).

Next in the Series

Governing AI Decisions at Scale

The next paper will examine how institutions preserve accountability, review and avenues for correction as AI-influenced decisions become routine, distributed across teams and embedded in high-volume workflows.

Written by Matt Bowd, Co-Founder and Chief Executive Officer of the Ballarat Region Artificial Intelligence Network (BRAIN).

Each study is a step toward a more intelligent and resilient region.

To participate in regional pilots or research partnerships, in our region or yours, connect via matt@brain.net.au.