Five takeaways from industry discussions on agentic AI and risk

Throughout September the National AI Centre held information sessions and workshops to help shape new resources on agentic AI and AI risk assessment.

Creating resources to help Australian organisations better understand and engage with AI is a big part of our work here at the National AI Centre. 

Right now we’re working on new resources that cover what you need to know about two important topics:

• Agentic AI: This refers to the use of AI agents – a type of AI system that can plan and carry out multi-step tasks on your behalf, rather than just responding to individual prompts. Agentic AI is being used more widely and can act with more independence. This brings unique challenges for accountability, oversight and safety.
• AI risk assessment: Understanding the risks is a key concern for organisations that are using or planning to use AI. We want to find out how they are identifying and managing risks in their use of AI, so that we can share practical approaches to understanding impact, defining risk levels and applying safeguards.

Holding information sessions and workshops was the first step in this piece of work. We connected with businesses, researchers, AI developers, government representatives and community organisations from across Australia to hear about their experience.

Here are 5 key takeaways that are helping to shape our work.

Many organisations we spoke to are citing the benefits of agentic AI – reducing repetitive work, improving productivity and supporting more complex workflows. Examples ranged from customer service, research and administration to compliance activities, information management and operational support. 

Smaller organisations are using agentic AI to access skills that used to require larger teams or more resources. This is especially the case when multiple, specialised AI agents can deliver tasks while working at the same time.

Not-for-profit organisations and those working with vulnerable communities are approaching agentic AI with caution due to data privacy concerns. But they see its benefits in increasing the reach of services and improving financial sustainability. 

However, participants underscored that benefits should not be assumed. Organisations need to build capability and implement effective accountability and oversight to get the benefits of agentic AI.

Agentic features are often contained with AI systems these days. Some products and services are also being marketed as agentic when they are actually not using agentic AI, just other kinds of automated systems. This can make it challenging to identify when AI systems are operating with agentic capability. 

When organisations are considering using an agentic system, they should ask:

• What can it access?
• What can it do?
• What authority has it been given?
• Who remains accountable?

Participants highlighted that agentic AI risk is shaped less by the technology itself and more by the permissions, autonomy and authority that organisations choose to give it. 

“It’s a delegation problem, more than a technology problem,” one participant said.

This reflects a broader shift towards thinking about agentic AI as a delegation challenge. Defining boundaries, understanding what systems can access, and determining which actions continue to require human approval are key.

Human oversight is a core part of controlling agentic AI systems. Our participants consistently emphasised the importance of maintaining human accountability, particularly where systems influence consequential decisions or can take actions on behalf of an organisation. 

At the same time, many challenged the assumption that simply placing a human ‘in the loop’ is enough. 

Effective human oversight depends on:

• appropriate training
• knowledge of the business and AI system
• ability to intervene
• adequate resourcing.

Combining human judgement with other safeguards such as monitoring, logging, escalation pathways, access controls and clear approval thresholds is a growing focus.

AI systems evolve. Models are updated, workflows change, permissions expand and organisations discover new use cases over time. Our participants highlighted the importance of reviewing and reassessing risks as these changes occur. 

Monitoring, logging and auditability featured prominently in the discussions. Participants emphasised the need to understand:

• what a system did
• what information it accessed
• what changed over time
• when human intervention may be required. 

This aligns with a broader shift towards continuous monitoring and ongoing governance capabilities. 

The most consistent message across our engagement sessions was the demand for practical, accessible guidance.

Participants repeatedly asked for:

• worked examples
• templates and checklists
• vendor assessment questions
• practical examples of accountability and oversight. 

The challenge is not simply identifying good practice. It is making it practical, proportionate and usable, especially for organisations without mature governance systems and teams in place.

These conversations are helping to shape future resources on agentic AI and AI risk assessment.

Over the coming weeks, we’ll refine and test our resources on these topics. Thanks to everyone who has contributed their knowledge, experience and feedback. 

Keep an eye on ai.gov.au for upcoming resources and information to support Australian organisations to realise the benefits of AI with confidence.