As AI agents move from answering questions to executing tasks, organisations face growing challenges around governance, accountability and regulatory compliance. In healthcare, where trust and patient safety are paramount, Safe AI is becoming a strategic necessity.
In many organisations, experimentation with AI agents is progressing faster than the governance mechanisms needed to manage them, and healthcare may be where that gap matters most.
AI is entering a new phase. Rather than simply generating content or answering questions, emerging AI agents can access applications, retrieve information, trigger workflows and carry out tasks with limited human intervention. This shift promises significant gains in productivity, but it also introduces governance challenges that many organisations are only beginning to recognise.
For years, AI was a tool you queried: you asked, it answered. That model, however imperfect, preserved a clear line of human accountability. Someone reviewed the output. Someone made the decision. The line is becoming less clear, and many organisations are still adapting their governance frameworks to this shift.
Agentic AI refers to systems that can:
- plan actions,
- chain tasks,
- interact with external tools,
- access data
- and execute workflows on behalf of users or organisations.
These capabilities are already appearing in platforms such as Microsoft Copilot, Salesforce Agentforce, ServiceNow AI Agents and OpenAI Operators, moving AI from information retrieval towards task execution. Crucially, they can autonomously determine how to use their assigned permissions to achieve a given objective. In many systems, the distance between recommendation and action is shrinking considerably. The question organisations now face is not whether this is happening, but whether they are ready for it.
In many organisations, experimentation with AI agents is progressing faster than the governance mechanisms needed to manage them effectively.

Amplification, Not Creation
The most important thing to understand about agentic AI risk is that it does not create new categories of threat. It amplifies the weaknesses that were already there. Overexposed data, inherited permissions that were never reviewed, weak information classification: these were always vulnerabilities. An agent can identify and process that exposed data far faster than any human could, and it can do so across dozens of systems simultaneously, often without anyone noticing until something goes wrong.
Three pressure points are emerging as particularly acute.
- The first is identity: AI agents do not map cleanly onto existing access models. They are neither human users nor conventional service accounts, and they can autonomously determine how to use their assigned permissions to achieve a given objective. They can also be manipulated through content embedded in documents and emails they process, a technique known as prompt injection and identified among the most significant emerging risks in the OWASP Top 10 for LLM Applications.
- The second pressure point is permissions: organisations tend to grant broad access to agents, creating a wide attack surface.
- The third is traceability. When an agent causes an incident, can you reconstruct who activated it, what data it accessed, and what instructions it was following? In many deployments today, providing a complete and auditable answer remains challenging.
The Healthcare Dimension
Healthcare providers are increasingly deploying AI for clinical documentation, diagnostic support, clinical decision support and AI-assisted triage systems, creating new governance challenges around accountability, transparency and patient safety.
The same governance gaps appearing in enterprise deployments are showing up in healthcare too, compounded by the demands of GDPR, the EU AI Act and clinical liability frameworks. Tools supporting radiology decision support, hospital workflow orchestration and clinical triage are already in use, and each introduces questions about explainability, auditability and human oversight that the sector is still working through.
In healthcare, the question of accountability is not abstract. Under the EU AI Act, certain AI systems used for clinical decision support or influencing medical decisions may fall within the high-risk category, triggering requirements for:
- risk management,
- documentation,
- human oversight
- and post-market monitoring.
These requirements highlight capabilities that many current agentic AI deployments are still working to mature, particularly around auditability, documentation and continuous monitoring.

The growing consensus among security researchers and AI governance professionals is that the answer is not to slow down deployment, but to treat each agent the way a responsible organisation treats any member of its workforce:
- with a verifiable identity,
- a named human owner,
- access limited to what it genuinely needs,
- and a full record of its activity.
Governance as an Accelerator
Experience from early enterprise deployments suggests that AI governance does not slow down adoption. In fact, surveys such as McKinsey’s State of AI consistently show that organisations achieving the greatest value from AI are also those investing most heavily in governance, oversight and operational maturity. Experience from early enterprise deployments suggests that AI governance does not slow down adoption. It makes adoption possible at scale. Organisations with clear rules about who can deploy agents, which data they can access and which actions require human sign-off are the ones successfully moving from isolated pilots to production systems. Those without those rules find themselves rebuilding every pilot before it can ship.
The practical foundations of AI compliance are becoming clearer: directory-based identity for agents, conditional access with immediately revocable permissions, data classification and confidentiality labelling that restrict an agent’s access to only the information required for its intended function, runtime monitoring and detection mechanisms for prompt injection and other emerging AI-specific attack vectors, and a central inventory that tracks all active agents, flags orphaned ones and feeds into security monitoring.
Increasingly, these capabilities are being discussed under the broader concept of Safe AI:
An approach that combines technical safeguards, human oversight, transparency, security and regulatory compliance to ensure AI systems remain trustworthy throughout their lifecycle.
This perspective aligns closely with frameworks such as the AI Risk Management Framework (AI RMF) and the OECD AI Principles.
For organisations operating in regulated domains such as healthcare, AI risk management and governance increasingly depend on being able to trace how data, decisions and AI services interact across complex digital ecosystems. This is one of the reasons why scalable data infrastructures, interoperability frameworks and robust data governance capabilities are becoming strategic assets rather than purely technical concerns, and why, for healthcare AI in particular, this infrastructure is not a compliance burden, but the basic precondition for any solution that aspires to move from a research environment into routine clinical practice.
The organisations that benefit most from agentic AI are unlikely to be those that adopt it fastest, but those that establish the governance foundations needed to deploy it safely at scale. As AI agents become part of everyday operations, success will depend not only on model performance, but also on the ability to govern identity, traceability and accountability at scale.
→ Check our Trustworthy AI solution


























