The rise of Shadow AI: A new mental health crisis for the NHS

A paper published this week in Nature Medicine tested nine frontier AI chatbots across 810 simulated mental health conversations. The researchers found that every chatbot amplified the psychological vulnerabilities of simulated patients through feedback loops they call vulnerability-amplifying interaction loops. Risk accumulated over multiple turns. It was worst with psychosis and mania profiles. And it was highest when the chatbot's apparently supportive responses inadvertently reinforced the patterns that maintain mental illness: validating delusional thinking, encouraging emotional dependence, or normalising symptoms that needed clinical attention.

Separately, ECRI, the international patient safety organisation, named the misuse of AI chatbots in healthcare as the number one health technology hazard for 2026, citing growing use by clinicians, patients, and healthcare staff despite the tools not being regulated as medical devices or validated for clinical use. It is the first time a widely available consumer technology has topped their annual list.

These findings show a clear and present danger. A survey of 1,000 NHS healthcare professionals published this summer found that 90% use AI in their clinical work, with 65% doing so ahead of any formal workplace guidance. Only 10% have never used AI in a clinical setting. The AMA's 2026 physician survey found 81% of US physicians now use AI in clinical practice, more than double the 38% reported in 2023. In the UK, a Nuffield Trust and RCGP study found 28% of GPs using AI at work, with 11% using tools they sourced themselves. Polling by Mental Health UK found 37% of UK adults have used an AI chatbot for mental health or wellbeing, rising to 64% among 25 to 34 year olds, with 11% reporting they received harmful information about suicide. OpenAI reports that more than 40 million people use ChatGPT for health queries every day. If anyone is in any doubt about the danger posed by this technology for people in mental health crisis, there are documented cases across the world of patients coming to harm or in some tragic cases, death. In the UK, a coroner's inquest heard in March that a 16-year-old boy from Hampshire asked ChatGPT for advice on ending his life, sidestepping its safety features by claiming the question was for research. He died the following day. In the US, multiple teenagers have died by suicide after extended conversations with AI chatbots, prompting lawsuits, state-level legal action, and a warning letter from 42 attorneys general.

The picture is clear: clinicians and patients are using AI for mental health, and this is dangerous. The question is whether anyone is going to give them something safe.

Two different problems

The risks to patients and clinicians are different and need different responses.

For patients, particularly those with psychotic symptoms, mania, or severe depression, the Nature Medicine findings show that consumer chatbots can actively make things worse. The chatbot validates a delusional belief because that is what supportive conversation looks like to a language model trained on helpfulness. The patient has no clinical framework to recognise when the chatbot is reinforcing a symptom rather than offering support. They are on a waiting list, it is 2am, and ChatGPT is the only thing responding. Over multiple turns, the feedback loop tightens and the risk of harm increases.

For clinicians, the risk is different. A psychiatrist reasoning through a complex case with ChatGPT is not going to introduce any delusional beliefs. But they are sharing patient data with an ungoverned tool that has no audit trail, no data governance, and no accountability. They may receive confidently wrong guidance on areas outside their expertise, and the less they know about the topic, the less able they are to catch the error. ECRI's concern is precisely this: when a chatbot's output feels helpful and definitive, clinicians start relying on it without questioning it. Standard ChatGPT also defaults to US clinical guidelines rather than NICE or BNF, so a UK clinician gets US-weighted guidance unless they know to specify otherwise. The chatbot does not have access to any data about the patient in question outside that given to it by the clinician who initiated the chat.

Both groups need governed alternatives. But those alternatives look different. Patient-facing tools need the safety guardrails the Nature Medicine paper calls for: mechanisms to detect and interrupt the vulnerability-amplifying loops before they compound.

Clinician-facing tools need data governance, clinical context, and integration with the trust's own protocols and patient records.

What the NHS has done and what it has not

The NHS has responded to the AI revolution very slowly. NHS England published guidance on AI-enabled ambient scribing in April 2025, updated in January 2026. Over a year later and scribes are only now gaining momentum.

CQC published guidance on what its inspectors will look for when assessing formal AI tools, not the informal use in question here. That's with 65% of NHS clinical staff using AI ahead of any formal workplace guidance.

Microsoft Copilot is being rolled out to 500,000 NHS staff for administrative tasks like drafting letters, meeting minutes, and discharge paperwork.

So the response to the AI revolution does not address the ways people are actually using the technology, which is to solve problems through reasoned discussion. A clinician who wants to discuss a complex patient with an AI tool that understands NICE guidelines, the Trust's own formulary, the patient's care plan, and the local escalation pathway has no governed option.

They have ChatGPT, or nothing.

The Nuffield Trust and RCGP study described a wild west of unregulated AI tools across the NHS, with some ICBs discouraging AI use altogether while others actively encourage piloting. GPs called for clear national guidance. What they got was a patchwork. Individual trusts have published their own AI policies. Rotherham Doncaster and South Humber NHS Foundation Trust's policy, published in February 2026, states that ChatGPT "must not be used to process patient-identifiable information, confidential or business sensitive data, or make clinical decisions." It then directs staff to Copilot Chat, a less capable model than the frontier chatbots clinicians are already using. The Nature Medicine paper found that less capable models showed measurably higher rates of harmful behaviour in mental health conversations. Swapping a better model for a worse one and calling it safe because it is on the approved list does not solve the problem. It may make it worse.

This is shadow AI, and the pattern is identical to the shadow IT I documented in my FOI research. When I asked 192 NHS trusts about their locally developed information tools, the ones used outside any formal system, the overwhelming majority had no audit trail, no risk register, no ownership, and no policy governing their use. Staff built spreadsheets and databases because the organisation did not give them what they needed. The same dynamic is playing out with AI. Staff are using ChatGPT because nobody has given them a governed clinical reasoning tool that understands their patients, their caseload, and their trust's protocols. The tools have changed, but the governance gap has not.

The mental health workforce equation

This week the government announced 100 new community mental health centres and 59 dedicated mental health emergency departments, £343m of capital funding, the biggest redesign of mental health services in a generation. The same week, East London Foundation Trust confirmed it is cutting £25m and 330 jobs from its mental health services because NHS England has required every Trust to balance its books. Fothergill ward has already stopped taking referrals and is due to close in September. Unite is balloting members on strike action.

2.36 million people in England had open referrals for mental health services at the end of May 2026, up from 2.10 million a year earlier. The new centres have capital funding but no one has said where the staff to run 159 new facilities come from when existing Trusts are cutting the workforce they already have.

Mental health trusts cannot recruit their way to larger capacity. The waiting list for treatment stretches to years in some areas. A governed clinical AI tool that helps existing staff manage larger caseloads safely, that supports triage, that provides supervised self-help for patients on waiting lists, that assists clinicians in reasoning through complex cases inside a framework with clinical safety assurance and audit trails, is not a future ambition. It is the only way the numbers work.

What governed clinical AI could look like

Products like Medwise AI already integrate Trust-level policies and formularies with national NICE guidelines at an institutional level. But each is standalone, each requires separate procurement, and none is integrated with the platform that holds the Trust's clinical data. A clinician asking about a specific patient gets generic guidance rather than guidance grounded in that patient's care plan, medication history, risk assessment, and local protocols.

A governed clinical AI service would run inside the Trust's data boundary, grounded in the Trust's own clinical data and protocols. When a clinician asks about a patient, the tool draws on that patient's care plan, risk history, current medication, and the Trust's own guidelines. When a patient on a waiting list engages with a supervised self-help tool, the system knows their referral pathway, their risk profile, and when to escalate to a human. Every interaction is logged and auditable.

Crucially, these interactions become a learning resource. A central AI support team within the Trust, or shared across a region, reviews the accumulated data on how clinicians are using AI: what questions they ask, where the responses fall short, which clinical areas generate the most queries. That team feeds back to clinicians, updates the Trust's AI guidelines, and builds documentation that continuously improves the quality and safety of AI responses through retrieval-augmented generation.

Scale this learning across a shared platform and the economics transform. A single Trust funding its own AI governance team is expensive. Ten trusts sharing a regional service, or 220 trusts sharing a national one, becomes viable when the underlying data layer, audit infrastructure, and governance framework are shared. The learning from one Trust's clinicians improves the service for every trust on the platform. A prompt that surfaced a dangerous response at one trust generates a guardrail update that protects clinicians at every other Trust. That network effect only works on shared infrastructure.

A learning organisation supports its staff in using new tools safely. What most NHS Trusts are doing instead is issuing policies that prohibit AI use while offering no alternative, placing clinicians in an impossible position: use AI and risk your career, or don't and provide a worse service to patients you cannot manage without it. 65% are choosing the first option and using AI without any formal guidance from their employer. In healthcare, being forced to act against your professional values has a name: moral injury. It is already one of the leading drivers of NHS sickness absence, with mental health accounting for 28.6% of all days lost. Adding another source of it, by forcing well-intentioned clinicians to choose between breaking the rules and letting patients down, is the last thing the workforce needs.

The cost of waiting

Staff and patients are already using AI for mental health, every day, ungoverned. Every month spent debating the politics of data platforms and the guardrails of AI without building the governed alternatives that would make those guardrails operational is a month where patients suffer the consequences of the NHS's slow response to the AI revolution.

Let's fix this. Everything we need is already in place: the platforms, the clinical data, the AI models, the governance frameworks. What is missing is the decision and focus needed to build and implement something governed before something ungoverned causes another harm that makes the front page of every newspaper in the country. That has already happened in the US, has already happened in Hampshire. The question for the NHS is whether it builds the alternative before or after it happens again.