Helpline spunout.ie reports rise in AI chatbot referrals of young people 'at a higher rate of distress'

Children's Ombudsman Niall Muldoon told the committee that AI chatbots are not currently considered high risk under the EU AI Act, 'so they don't get monitored in the way they should be'. File picture

A free 24/7 helpline for young people, which is funded by the HSE, has seen an increase in referrals of distressed people to its services coming from AI chatbots.

Sinead Keane of spunout.ie said its 'Text About It' messaging service has seen such referrals "coming to us at a higher rate of distress than if they came to us directly". Ms Keane was speaking at the Oireachtas education committee on Thursday morning, which was meeting to discuss mental health supports available to schools.

Sinn Féin senator Pauline Tully asked about suicide prevention, AI, and online safety, referencing an RTÉ Prime Time interview with Maria Raine, whose teenage son Adam died by suicide last year. Based in California, his family have filed a wrongful death lawsuit against OpenAI alleging ChatGPT fostered a relationship of dependency with their son and failed to protect him.

In response to Ms Tully, Ms Keane said she had seen the interview.  "It was extremely disturbing to see what happened to that family," she said. "I know that's not a singular case or a singular experience."

The 'Text About It' service has seen an increase in recent months of referrals coming in from chatbots. "It's now at a 14% referral rate," she said.

"What we are noticing at this early stage is that these people are coming to us at a higher rate of distress than if they came to us directly."

"That then takes longer to de-escalate them, and bring them to that cool, calm place. When we can't do that, we bring in, and we connect with, the emergency services."

It emphasises the need for regulation in this area, she said. "To protect the whole population, not just young people in a rapidly changing environment."

The text messaging service is staffed by trained volunteers. In September, approximately one in eight conversations initiated on the service were about school. 

Stress and anxiety are among the most common topics discussed, closely followed by loneliness, suicide, and depression, Ms Keane told the committee. 

"School conversations focus on bullying, not belonging, exam stress, friendship breakdowns, and conflict in peer groups, intimate image abuse and exploitation, self-harming or suicidal thoughts linked to school," she said. 

"Some of these issues lead young people to avoid school." 

EU Kids Act

Children's Ombudsman Niall Muldoon told the committee that AI chatbots are not currently considered high risk under the EU AI Act. "They are considered a medium risk so they don't get monitored in the way they should be." 

That will change with the EU Kids Act, he said. We also need to recognise that platforms are selling something that's "unsafe", he added. 

There needs to be a move towards 'safe before sale', he said. "So that you cannot buy it, you cannot have your chatbot unless its safe. 

What's your view on this issue?

You can tell us here

"Children that we talk to are very clear: They don't want a chatbot helping them with health or mental health; they know that's dangerous." 

"Platforms can say 'listen, we will not talk to you about anything in the mental health range'. We need to regulate in a way that puts the platforms back on notice." 

• Jess Casey, education correspondent