Effective Human-in-the-Loop AI Governance

“Human in the loop” is rapidly becoming one of the most overused phrases in AI governance. It is often offered as proof that an organisation remains in control: the model makes a recommendation, a person clicks approve, and accountability is supposedly preserved.

But a human approval step is not the same thing as human judgement. If people cannot see the evidence, question the recommendation, slow the process down, choose another option or safely disagree, they are not supervising the AI. They are providing a human signature for a machine-led decision.

That matters now, while organisations are rushing to deploy generative AI, automate workflows and demonstrate productivity gains. Too much of the discussion assumes that adding a person at the end of a process is enough to make the system safe, fair and accountable.

It is not.

The real question is whether the human has the authority, context, time, competence and organisational support to exercise meaningful judgement.

This is consistent with the OECD AI Principles, which call for human agency and oversight appropriate to the context and impact of an AI system. It also aligns with the NIST AI Risk Management Framework, which treats governance as an ongoing organisational function rather than a one-off technical check.

The rubber-stamp problem

The recent debate about AI-enabled military targeting makes this point in stark terms. Dan Summers argues that AI can compress decision-making to a pace at which traditional models of human oversight become impossible. A person asked to approve large volumes of AI-generated targeting recommendations under severe time pressure is not necessarily providing meaningful oversight. They may be legitimising a decision they cannot properly interrogate. The AI-Assisted Strategic Own Goal in the Kill Chain makes the case that oversight must be redesigned rather than merely preserved.

This is not just a military problem. It is already relevant to government services, banking, insurance, healthcare, recruitment, policing, cybersecurity, critical infrastructure and any organisation using AI to prioritise people, risks or resources.

The governance trap is simple: an organisation retains nominal human involvement while practical decision-making power shifts to the machine.

A human reviewer becomes a rubber stamp when:

• The system determines which cases, options or risks they are allowed to see
• They receive an answer without the underlying evidence, uncertainty or alternatives
• Decision volumes make proper review impossible
• Performance measures reward speed, throughput or agreement with the system
• Escalation pathways are unclear, slow or professionally risky
• They do not have the authority to reject, pause or amend the recommendation
• The organisation treats human review as a compliance control rather than a substantive decision point

This is not meaningful human oversight. It is accountability theatre.

The difference between can and should

One useful distinction is between decisions that can be assessed against defined rules and decisions that require judgement about competing values.

As Summers notes:

"The 'should we' question can be a more nuanced proposition. There is no fixed rule that resolves it. It is a dynamic assessment of the strategic and political implications of action."

Some decisions are clearer and rule-based. A system might check whether a form is complete, whether an application meets published eligibility criteria, whether a transaction breaches a known threshold or whether a cybersecurity alert matches a documented pattern. 

Even these uses require testing, monitoring, auditability and a route for human intervention. The NIST AI RMF Playbook is particularly useful on this point because it treats AI risk management as a lifecycle activity involving governance, mapping context, measurement and ongoing management.

But many important decisions are not just rule-matching exercises. They involve fairness, legitimacy, individual circumstances, public confidence, political context and downstream effects.

An AI system may estimate the probability that someone will default on a loan. It cannot decide whether using that estimate is appropriate as a matter of public policy, ethics or accountability.

An AI system may identify people who are statistically more likely to need government support. It cannot settle whether the resulting intervention is fair, proportionate, respectful or likely to undermine trust in public institutions.

These are not technical questions alone. They are decisions about values, power and responsibility. The UNESCO Recommendation on the Ethics of Artificial Intelligence similarly places human oversight and determination at the centre of AI governance, particularly where systems may affect people’s rights and dignity.

The most important role for people is not merely to correct the occasional model error. It is to remain accountable for decisions where society has not, and should not, delegate the balancing of values to a model.

Design for real intervention

Effective human oversight needs to be designed into the operating model from the beginning. It cannot be bolted onto the final screen of an automated workflow.

A practical model has six elements.

1) Define non-delegable decisions

Organisations should identify decisions that must remain with accountable people. These will often include decisions that:

• Affect rights, entitlements, liberty, safety or access to essential services
• Require competing values to be weighed rather than a rule to be applied
• Create significant legal, reputational or political consequences
• Affect people or communities already subject to disadvantage or discrimination
• Establish or change rules, thresholds, risk tolerances or decision criteria
• Involve novel, uncertain or high-impact circumstances

This is not a call to ban AI from high-risk settings. It is a call to be explicit about where AI can support judgement and where it must not replace it.

For government agencies, this should be a design and procurement requirement, not an afterthought once a system is in production. Australia’s National Framework for the Assurance of Artificial Intelligence in Government is useful here because it frames assurance as a way to identify and manage risks before, during and after AI systems are deployed.

If a vendor’s human oversight model consists of an approval screen and a promise of explainability, that is not enough.

2) Give people authority

A human reviewer must be able to say no.

They need the authority to reject a recommendation, request more information, select an alternative option, override the system or escalate the case. These actions need to be genuinely usable in practice.

If overriding an AI recommendation requires multiple approvals, creates extra work, damages performance metrics or invites management scrutiny, people will quickly learn not to do it. A right to intervene that cannot be exercised safely is not a real control.

Organisations should monitor override rates, escalation patterns and the reasons people disagree with the system. Very low override rates do not automatically mean that a system is excellent. They may show that staff do not feel able to challenge it.

There should also be clear accountability for the operating model. Someone needs to be responsible for whether the system is being used appropriately, not merely for the decisions made by individual staff members.

3) Show evidence and uncertainty

A recommendation without context is not decision support. It is an instruction.

People need to understand what information the system relied on, how current that information is, what important information may be missing and how confident the system is in its output. They should also be able to see plausible alternatives, not merely the system’s preferred answer.

For generative AI, that means more than a fluent response. It means access to sources, clear provenance and a distinction between verified evidence, inference and speculation.

For predictive systems, it means showing the factors that materially influenced an output, the model’s limitations and whether a case sits outside the conditions in which the system was tested.

Explainability is not about forcing every AI system into a neat visualisation. It is about ensuring that a competent reviewer can investigate the basis for a decision and explain it to the affected person, a regulator, a court, a board or a parliamentary committee.

The Australian Government’s AI Ethics Framework is still worth reading for its practical articulation of transparency, explainability, contestability and accountability. Those principles are relevant even where a system is technically lawful and performing as intended.

4) Protect time for judgement

Humans cannot provide meaningful oversight at a pace designed for machines.

This is difficult because the promise of AI is often faster processing, lower costs and greater scale. But if AI creates more decisions, faster, while reducing the time available to review them, it may improve throughput at the expense of accountability.

That is not efficiency. It is a transfer of risk from the system to the person required to approve it.

Not every case requires the same degree of review. Risk-based triage can help, provided the triage process itself is governed and tested. Higher-impact, lower-confidence, novel or contested cases should trigger slower and deeper review.

A useful test is straightforward: could the reviewer reasonably explain why they accepted the AI’s recommendation? If the answer is no because they had 30 seconds and 200 cases waiting, the control has failed.

5) Build in challenge

Good governance does not only ask, “Why should we do this?” It also asks, “What could we be missing?”

Summers proposes using AI in an adversarial role within military targeting processes: a live red team that tests recommendations against legal, ethical and strategic criteria, surfacing what human decision-makers should consider rather than making the judgement itself.

There is a useful civilian version of this idea. Alongside an AI recommendation, systems should prompt structured challenge:

• What evidence contradicts this recommendation?
• Which assumptions are doing the most work?
• What information may be missing or out of date?
• What is the likely impact on different groups?
• Is the model being used outside the context in which it was tested?
• What would change the decision?
• Is there a less harmful or more proportionate option?
• What happens if the system is wrong?

This is one of the more promising uses of AI in governance. Rather than simply making decisions faster, AI can help decision-makers notice uncertainty, bias, missing context and second-order effects.

6) Keep a decision record

If an organisation cannot reconstruct what happened, it cannot learn from its mistakes, provide assurance or account for its use of AI.

For significant decisions, records should capture:

• The system and model version used
• The relevant data inputs and their provenance
• The recommendation or output generated
• The uncertainty, limitations or warnings identified
• The person who made the decision and their role
• The questions asked, evidence considered and alternatives available
• Any override, escalation or exception
• The final decision and its rationale

This does not mean documenting every low-risk use of a generative AI tool. Records should be proportionate to risk. But when AI affects people’s rights, opportunities, access to services or safety, auditability is not optional.

This is also where Australian privacy reform matters. The Privacy Act Review Report recommended stronger protections relating to automated decision-making, including greater transparency where personal information is used in substantially automated decisions that significantly affect individuals.

When something goes wrong, organisations need to determine whether the issue was poor data, flawed model design, weak implementation, inadequate training, bad incentives or a decision made without enough information. Without that learning loop, AI allows the same mistakes to be repeated at scale.

A better starting point

The technology is moving faster than most organisations’ ability to govern it. At this early stage, the sensible default for consequential decisions is to use AI to augment human judgement rather than substitute for it.

That means using AI to identify patterns, organise evidence, test assumptions, simulate options, detect anomalies and surface risks. It does not mean asking AI to decide what is fair, legitimate or acceptable on behalf of an organisation or a community.

This will require investment in people as much as technology. Decision-makers need AI literacy, domain expertise, confidence to challenge systems and protection from incentives that prioritise speed over judgement. Senior leaders and boards need to understand that human oversight is not a single control in a risk register. It is an organisational capability.

At the start of the AI age, the challenge is not simply to keep a person somewhere in the workflow. It is to design systems in which people retain the practical ability to exercise judgement, challenge the system and take responsibility for the outcome.

That means authority, evidence, time, escalation and records. Without those things, “human in the loop” is not governance. It is theatre.